Re: [Flink-SQL] FLOORing OR CEILing a DATE or TIMESTAMP to WEEK uses Thursdays as week start
Thanks a lot for the added context and pointers Julian and Leonard, I've fixed it by going down to the arithmetics as suggested in one of the Calcite discussions. The changes proposed by FLIP-126 definitely look good. I'll check its details further. Best Regards, On Thu, 4 Mar 2021 at 04:18, Leonard Xu wrote: > Hi, Sebastián Ramírez Magrí > (Sorry for wrong name in above mail) > > Flink follows old version calcite’s behaviour which lead to the wrong > behavior. snd Julian is right that calcite has corrected FLOOR and CEIL > functions in CALCITE-3412, Flink has upgraded calcite to 1.26 version > which contains the patch, what we need is only to adapt it in Flink code, > I’d like to make this as a part of FLIP-162 and fix it soon. > > Thanks Julian and Timo for the reminder. > > > Best, > Leonard > > 在 2021年3月4日，12:14，Leonard Xu 写道： > > Hi, Jaffe > > Flink follows old version calcite’s behaviour which lead to the wrong > behavior. snd Julian is right that calcite has corrected FLOOR and CEIL > functions in CALCITE-3412, Flink has upgraded calcite to 1.26 version > which contains the patch, what we need is only to adapt it in Flink code, > I’d like to make this as a part of FLIP-162 and fix it soon. > > Thanks Julian and Timo for the reminder. > > > Best, > Leonard > > > -- Sebastián Ramírez Magrí
[Flink-SQL] FLOORing OR CEILing a DATE or TIMESTAMP to WEEK uses Thursdays as week start
While using a simple query such as this SELECT `ts`, FLOOR(`ts` TO WEEK) as `week_start`, CEIL(`ts` TO WEEK) as `week_end` FROM some_table I get some weird results like these: 2021-03-01T00:00|2021-02-25T00:00|2021-03-04T00:00 Which is obviously wrong since March 1st is on Monday, February 25th is Thursday as well as March 04th. I've tried different combinations of timezone configurations and with both timestamps and dates, with the same results. Is there anything obviously wrong in that query? Is there any configuration to keep in mind for the start of week day? -- Sebastián Ramírez Magrí
Re: Best way to handle BIGING to TIMESTAMP conversions
Thanks a lot Jark, On Mon, 1 Mar 2021 at 02:38, Jark Wu wrote: > Hi Sebastián, > > You can use `TO_TIMESTAMP(FROM_UNIXTIME(e))` to get a timestamp value. > The BIGINT should be in seconds. Please note to declare the computed > column > in DDL schema and declare a watermark strategy on this computed field to > make > the field to be a rowtime attribute. Because streaming over window > requires to > order by a time attribute. > > Best, > Jark > > On Sun, 21 Feb 2021 at 07:32, Sebastián Magrí > wrote: > >> I have a table with two BIGINT fields for start and end of an event as >> UNIX time in milliseconds. I want to be able to have a resulting column >> with the delta in milliseconds and group by that difference. Also, I want >> to be able to have aggregations with window functions based upon the `end` >> field. >> >> The table definition looks like this: >> |CREATE TABLE sessions ( >> | `ats` STRING, >> | `e` BIGINT, >> | `s` BIGINT, >> | `proc_time` AS PROCTIME(), >> | PRIMARY KEY (`ats`, `s`, `e`) NOT ENFORCED >> |) >> >> Then I have a few views like this: >> >> CREATE VIEW second_sessions AS >> SELECT * FROM sessions >> WHERE `e` - `s` = 1000 >> >> And some windows using these views like this: >> >> WINDOW w3m AS ( >> PARTITION BY `t` >> ORDER BY `proc_time` >> RANGE BETWEEN INTERVAL '3' MINUTE PRECEDING AND CURRENT ROW >> ) >> >> I'd like to use the `e` field for windowing instead of `proc_time`. But I >> keep running into errors with the `TO_TIMESTAMP(BIGINT)` function now >> missing or with unsupported timestamp arithmetics. >> >> What is the best practice for a case such as this? >> >> Best Regards, >> -- >> Sebastián Ramírez Magrí >> > -- Sebastián Ramírez Magrí
Re: Community chat?
I agree with Yuval. If we wanted to keep chats in the open source world, there's also Matrix nowadays which works quite well. On Wed, 24 Feb 2021 at 09:58, Yuval Itzchakov wrote: > Both have their place IMO. > > There's a lot of value in synchronous communication for which I'd prefer > Slack or Discord. > For async communication, I think moving away from mailing lists into > something like a Discourse forum would be good. > > On Wed, Feb 24, 2021 at 11:36 AM Marta Paes Moreira > wrote: > >> Ah! That freenode channel dates back to...2014? The community is not >> maintaining any channels other than the Mailing List (and Stack Overflow), >> currently. >> >> But this is something we're looking into, as it's coming up more and more >> frequently. Would Slack be your first pick? Or would something async but >> easier to interact with also work, like a Discourse forum? >> >> Thanks for bringing this up! >> >> Marta >> >> >> >> On Mon, Feb 22, 2021 at 10:03 PM Yuval Itzchakov >> wrote: >> >>> A dedicated Slack would be awesome. >>> >>> On Mon, Feb 22, 2021, 22:57 Sebastián Magrí >>> wrote: >>> >>>> Is there any chat from the community? >>>> >>>> I saw the freenode channel but it's pretty dead. >>>> >>>> A lot of the time a more chat alike venue where to discuss stuff >>>> synchronously or just share ideas turns out very useful and estimulates the >>>> community. >>>> >>>> -- >>>> Sebastián Ramírez Magrí >>>> >>> > > -- > Best Regards, > Yuval Itzchakov. > -- Sebastián Ramírez Magrí
Community chat?
Is there any chat from the community? I saw the freenode channel but it's pretty dead. A lot of the time a more chat alike venue where to discuss stuff synchronously or just share ideas turns out very useful and estimulates the community. -- Sebastián Ramírez Magrí
Re: Compile time checking of SQL
Thanks a lot Timo!
On Mon, 22 Feb 2021 at 08:19, Timo Walther wrote:
> Yes, this is possible. You can run `tableEnv.sqlQuery("...").explain()`
> or `tableEnv.toRetractStream(table)` which would trigger the complete
> translation of the SQL query without executing it.
>
> Regards,
> Timo
>
> On 20.02.21 18:46, Sebastián Magrí wrote:
> > I mean the SQL queries being validated when I do `mvn compile` or any
> > target that runs that so that basic syntax checking is performed without
> > having to submit the job to the cluster.
> >
> > On Thu, 18 Feb 2021 at 16:17, Timo Walther > <mailto:twal...@apache.org>> wrote:
> >
> > Hi Sebastián,
> >
> > what do you consider as compile time? If you mean some kind of SQL
> > editor, you could take a look at Ververica platform (the community
> > edition is free):
> >
> >
> https://www.ververica.com/blog/data-pipelines-with-flink-sql-on-ververica-platform
> > <
> https://www.ververica.com/blog/data-pipelines-with-flink-sql-on-ververica-platform
> >
> >
> > Otherwise Flink SQL is always validated at (what we call) "pre-flight
> > phase". A cluster is not required but it is already JVM runtime of
> the
> > client.
> >
> > Regards,
> > Timo
> >
> > On 18.02.21 14:55, Sebastián Magrí wrote:
> > > Is there any way to check SQL strings in compile time?
> > >
> > > --
> > > Sebastián Ramírez Magrí
> >
> >
> >
> > --
> > Sebastián Ramírez Magrí
>
>
--
Sebastián Ramírez Magrí
Union fields with time attributes have different types
I'm using a query like this WITH aggs_1m AS ( SELECT `evt`, `startts` `endts`, SUM(`value`) AS `value` FROM aggregates_per_minute ), aggs_3m AS ( SELECT `evt`, TUMBLE_START(`endts`, INTERVAL '3' MINUTE) AS `startts`, TUMBLE_END(`endts`, INTERVAL '3' MINUTE) AS `endts`, SUM(`c`) AS `value` FROM aggregates_per_minute GROUP BY t, TUMBLE(`endts`, INTERVAL '3' MINUTE) ) SELECT `evt`, `value`, `startts`, `endts` FROM aggs_1m UNION SELECT `evt`, `value`, `startts`, `endts` FROM aggs_3m But it's throwing this exception org.apache.flink.table.api.ValidationException: Union fields with time attributes have different types. Doesn't TUMBLE_START(somets, ...) return a TIMESTAMP of the same type? -- Sebastián Ramírez Magrí
Best way to handle BIGING to TIMESTAMP conversions
I have a table with two BIGINT fields for start and end of an event as UNIX time in milliseconds. I want to be able to have a resulting column with the delta in milliseconds and group by that difference. Also, I want to be able to have aggregations with window functions based upon the `end` field. The table definition looks like this: |CREATE TABLE sessions ( | `ats` STRING, | `e` BIGINT, | `s` BIGINT, | `proc_time` AS PROCTIME(), | PRIMARY KEY (`ats`, `s`, `e`) NOT ENFORCED |) Then I have a few views like this: CREATE VIEW second_sessions AS SELECT * FROM sessions WHERE `e` - `s` = 1000 And some windows using these views like this: WINDOW w3m AS ( PARTITION BY `t` ORDER BY `proc_time` RANGE BETWEEN INTERVAL '3' MINUTE PRECEDING AND CURRENT ROW ) I'd like to use the `e` field for windowing instead of `proc_time`. But I keep running into errors with the `TO_TIMESTAMP(BIGINT)` function now missing or with unsupported timestamp arithmetics. What is the best practice for a case such as this? Best Regards, -- Sebastián Ramírez Magrí
Re: [Flink SQL] FLOOR(timestamp TO WEEK) not working
I'm using the latest Flink 1.12 and the timestamps precision is coming from Debezium, which I think is a standard ISO-8601 timestamp. On Thu, 18 Feb 2021 at 16:19, Timo Walther wrote: > Hi Sebastián, > > which Flink version are you using? And which precision do the timestamps > have? > > This looks clearly like a bug to me. We should open an issue in JIRA. > > Regards, > Timo > > On 18.02.21 16:17, Sebastián Magrí wrote: > > While using said function in a query I'm getting a query compilation > > error saying that there's no applicable method for the given arguments. > > The parameter types displayed in the error are > > > > org.apache.flink.table.data.TimestampData, > > org.apache.flink.table.data.TimestampData > > > > And there's no overload listed for > > `org.apache.calcite.runtime.SqlFunctions.floor` that matches those > > parameters. > > > > Is this supported at all? > > Is there anything in particular which should be imported or added to the > > build to make this work? > > > > -- > > Sebastián Ramírez Magrí > > -- Sebastián Ramírez Magrí
Re: Compile time checking of SQL
I mean the SQL queries being validated when I do `mvn compile` or any target that runs that so that basic syntax checking is performed without having to submit the job to the cluster. On Thu, 18 Feb 2021 at 16:17, Timo Walther wrote: > Hi Sebastián, > > what do you consider as compile time? If you mean some kind of SQL > editor, you could take a look at Ververica platform (the community > edition is free): > > > https://www.ververica.com/blog/data-pipelines-with-flink-sql-on-ververica-platform > > Otherwise Flink SQL is always validated at (what we call) "pre-flight > phase". A cluster is not required but it is already JVM runtime of the > client. > > Regards, > Timo > > On 18.02.21 14:55, Sebastián Magrí wrote: > > Is there any way to check SQL strings in compile time? > > > > -- > > Sebastián Ramírez Magrí > > -- Sebastián Ramírez Magrí
[Flink SQL] FLOOR(timestamp TO WEEK) not working
While using said function in a query I'm getting a query compilation error saying that there's no applicable method for the given arguments. The parameter types displayed in the error are org.apache.flink.table.data.TimestampData, org.apache.flink.table.data.TimestampData And there's no overload listed for `org.apache.calcite.runtime.SqlFunctions.floor` that matches those parameters. Is this supported at all? Is there anything in particular which should be imported or added to the build to make this work? -- Sebastián Ramírez Magrí
Compile time checking of SQL
Is there any way to check SQL strings in compile time? -- Sebastián Ramírez Magrí
Re: Conflicts between the JDBC and postgresql-cdc SQL connectors
The root of the previous error seemed to be the flink version the connector was compiled for. I've tried compiling my own postgresql-cdc connector, but still have some issues with dependencies. On Thu, 28 Jan 2021 at 11:24, Sebastián Magrí wrote: > Applied that parameter and that seems to get me some progress here. > > I still get the shade overlapping classes warning, but I get the > PostgreSQLTableFactory in the merged table.factories.Factory service file. > > However, now on runtime the application fails to find the debezium source > function class coming down to this error: > > Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException: > Cannot load user class: > com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction > ClassLoader info: URL ClassLoader: > Class not resolvable through given classloader. > > The class is indeed in jar, though. > > Any thougths? > > On Thu, 28 Jan 2021 at 09:57, Jark Wu wrote: > >> Hi Sebastián, >> >> Could you try to add combine.children="append" attribute to the >> transformers configuration? >> You can also see the full shade plugin configuration here [1]. >> >> Best, >> Jark >> >> [1]: >> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/connectors/#transform-table-connectorformat-resources >> >> On Thu, 28 Jan 2021 at 17:28, Sebastián Magrí >> wrote: >> >>> Hi Jark! >>> >>> Please find the full pom file attached. >>> >>> Best Regards, >>> >>> On Thu, 28 Jan 2021 at 03:21, Jark Wu wrote: >>> >>>> Hi Sebastián, >>>> >>>> I think Dawid is right. >>>> >>>> Could you share the pom file? I also tried to >>>> package flink-connector-postgres-cdc with ServicesResourceTransformer, and >>>> the Factory file contains >>>> >>>> com.alibaba.ververica.cdc.connectors.postgres.table.PostgreSQLTableFactory >>>> >>>> >>>> Best, >>>> Jark >>>> >>>> >>>> On Tue, 26 Jan 2021 at 21:17, Sebastián Magrí >>>> wrote: >>>> >>>>> Thanks a lot for looking into it Dawid, >>>>> >>>>> In the >>>>> src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory >>>>> file I only see >>>>> >>>>> org.apache.flink.connector.jdbc.table.JdbcDynamicTableFactory >>>>> >>>>> Even after applying the ServicesResourceTransformer. >>>>> >>>>> >>>>> On Tue, 26 Jan 2021 at 11:58, Dawid Wysakowicz >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Unfortunately I am not familiar with the packaging of >>>>>> flink-connector-postgres-cdc. Maybe @Jark could help here? >>>>>> >>>>>> However, I think the problem that you cannot find the connector is >>>>>> caused because of lack of entry in the resulting Manifest file. If there >>>>>> are overlapping classes maven does not exclude whole dependencies, but >>>>>> rather picks the overlapping class from one of the two. Could you check >>>>>> if >>>>>> you see entries for all tables in >>>>>> src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory. >>>>>> >>>>>> If not, you could try applying the ServicesResourceTransformer[1] >>>>>> >>>>>> Best, >>>>>> >>>>>> Dawid >>>>>> >>>>>> [1] >>>>>> https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer >>>>>> On 26/01/2021 12:29, Sebastián Magrí wrote: >>>>>> >>>>>> Hi! >>>>>> >>>>>> I've reported an issue with the postgresql-cdc connector apparently >>>>>> caused by the maven shade plugin excluding either the JDBC connector or >>>>>> the >>>>>> cdc connector due to overlapping classes. The issue for reference is >>>>>> here: >>>>>> >>>>>> https://github.com/ververica/flink-cdc-connectors/issues/90 >>>>>> >>>>>> In the meantime, however, I've been trying to figure out if I can set >>>>>> up an exclusion rule to fix this in my pom.xml file, without success. >>>>>> >>>>>> The `org.postgresql:postgresql` dependency is being added manually by >>>>>> me to have a sink on a postgresql table and injected by the cdc connector >>>>>> seemingly via its debezium connector dependency. >>>>>> >>>>>> Any guidance or hints I could follow would be really appreciated. >>>>>> >>>>>> -- >>>>>> Sebastián Ramírez Magrí >>>>>> >>>>>> >>>>> >>>>> -- >>>>> Sebastián Ramírez Magrí >>>>> >>>> >>> >>> -- >>> Sebastián Ramírez Magrí >>> >> > > -- > Sebastián Ramírez Magrí > -- Sebastián Ramírez Magrí
Re: [Flink SQL] CompilerFactory cannot be cast error when executing statement
Hi Timo! I've been building my jobs instead of using the binaries to avoid this issue, hence I've not looked at this again. But I'd say it's still an issue since nothing from the set up have changed in the meantime. Thanks! On Tue, 2 Feb 2021 at 08:51, Timo Walther wrote: > Hi Sebastian, > > sorry for the late reply. Could you solve the problem in the meantime? > It definitely looks like a dependency conflict. > > Regards, > Timo > > > On 22.01.21 18:18, Sebastián Magrí wrote: > > Thanks a lot Matthias! > > > > In the meantime I'm trying out something with the scala quickstart. > > > > > > On Fri, 22 Jan 2021 at 17:12, Matthias Pohl > <mailto:matth...@ververica.com>> wrote: > > > > Ok, to be fair, I just did some research on the error message and > > didn't realize that you're working with binaries only. > > > > I tried to set it up on my machine to be able to reproduce your > > error. Unfortunately, I wasn't able to establish the connection > > between Flink and Postgres using your docker-compose.yml. > > I'm going to cc Timo. Maybe, he has a guess what's causing this > error. > > > > Best, > > Matthias > > > > On Fri, Jan 22, 2021 at 4:35 PM Sebastián Magrí > > mailto:sebasma...@gmail.com>> wrote: > > > > Hi Matthias! > > > > I went through that thread but as I'm just using the > > `apache/flink` docker image for testing I honestly couldn't > > figure out how I would do that since I don't have a pom file to > > edit. If it's possible to do it through the configuration I'd be > > glad if you could point me out in the right direction. > > > > Pretty evident I don't have a lot of experience with mvn or > > "modern" Java in general. > > > > :-) > > > > Thanks! > > > > On Fri, 22 Jan 2021 at 15:19, Matthias Pohl > > mailto:matth...@ververica.com>> wrote: > > > > Hi Sebastián, > > have you tried changing the dependency scope to provided > > for flink-table-planner-blink as it is suggested in [1]? > > > > Best, > > Matthias > > > > [1] > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Flink-1-10-exception-Unable-to-instantiate-java-compiler-td38221.html > > < > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Flink-1-10-exception-Unable-to-instantiate-java-compiler-td38221.html > > > > > > On Fri, Jan 22, 2021 at 4:04 PM Sebastián Magrí > > mailto:sebasma...@gmail.com>> wrote: > > > > Hi! > > > > I'm trying out Flink SQL with the attached > > docker-compose file. > > > > It starts up and then I create a table with the > > following statement: > > > > CREATE TABLE mytable_simple ( > >`customer_id` INT > > ) WITH ( > >'connector' = 'jdbc', > >'url' = 'jdbc:postgresql://pgusr:pgpwd@postgres > /pdgb', > >'table-name' = 'mytable' > > ); > > > > However when I try to run this: > > > > select * from mytable_simple; > > > > I get the following error in the client: > > > > [ERROR] Could not execute SQL statement. Reason: > > java.lang.ClassCastException: > > org.codehaus.janino.CompilerFactory cannot be cast to > > org.codehaus.commons.compiler.ICompilerFactory > > > > At first I thought it could be an incompatibility issue > > with the libraries I was putting in, like the > > postgres-cdc library version, but even after leaving > > only the JDBC libraries in I still get the same error. > > > > It'd be great if you could give me some pointers here. > > > > Thanks! > > > > -- > > Sebastián Ramírez Magrí > > > > > > > > -- > > Sebastián Ramírez Magrí > > > > > > Managing Directors: Yip Park Tung Jason, Jinwei (Kevin) Zhang, Karl > > Anton Wehner > > > > > > > > -- > > Sebastián Ramírez Magrí > > -- Sebastián Ramírez Magrí
Re: Conflicts between the JDBC and postgresql-cdc SQL connectors
Applied that parameter and that seems to get me some progress here. I still get the shade overlapping classes warning, but I get the PostgreSQLTableFactory in the merged table.factories.Factory service file. However, now on runtime the application fails to find the debezium source function class coming down to this error: Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot load user class: com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction ClassLoader info: URL ClassLoader: Class not resolvable through given classloader. The class is indeed in jar, though. Any thougths? On Thu, 28 Jan 2021 at 09:57, Jark Wu wrote: > Hi Sebastián, > > Could you try to add combine.children="append" attribute to the > transformers configuration? > You can also see the full shade plugin configuration here [1]. > > Best, > Jark > > [1]: > https://ci.apache.org/projects/flink/flink-docs-master/dev/table/connectors/#transform-table-connectorformat-resources > > On Thu, 28 Jan 2021 at 17:28, Sebastián Magrí > wrote: > >> Hi Jark! >> >> Please find the full pom file attached. >> >> Best Regards, >> >> On Thu, 28 Jan 2021 at 03:21, Jark Wu wrote: >> >>> Hi Sebastián, >>> >>> I think Dawid is right. >>> >>> Could you share the pom file? I also tried to >>> package flink-connector-postgres-cdc with ServicesResourceTransformer, and >>> the Factory file contains >>> >>> com.alibaba.ververica.cdc.connectors.postgres.table.PostgreSQLTableFactory >>> >>> >>> Best, >>> Jark >>> >>> >>> On Tue, 26 Jan 2021 at 21:17, Sebastián Magrí >>> wrote: >>> >>>> Thanks a lot for looking into it Dawid, >>>> >>>> In the >>>> src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory >>>> file I only see >>>> >>>> org.apache.flink.connector.jdbc.table.JdbcDynamicTableFactory >>>> >>>> Even after applying the ServicesResourceTransformer. >>>> >>>> >>>> On Tue, 26 Jan 2021 at 11:58, Dawid Wysakowicz >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> Unfortunately I am not familiar with the packaging of >>>>> flink-connector-postgres-cdc. Maybe @Jark could help here? >>>>> >>>>> However, I think the problem that you cannot find the connector is >>>>> caused because of lack of entry in the resulting Manifest file. If there >>>>> are overlapping classes maven does not exclude whole dependencies, but >>>>> rather picks the overlapping class from one of the two. Could you check if >>>>> you see entries for all tables in >>>>> src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory. >>>>> >>>>> If not, you could try applying the ServicesResourceTransformer[1] >>>>> >>>>> Best, >>>>> >>>>> Dawid >>>>> >>>>> [1] >>>>> https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer >>>>> On 26/01/2021 12:29, Sebastián Magrí wrote: >>>>> >>>>> Hi! >>>>> >>>>> I've reported an issue with the postgresql-cdc connector apparently >>>>> caused by the maven shade plugin excluding either the JDBC connector or >>>>> the >>>>> cdc connector due to overlapping classes. The issue for reference is here: >>>>> >>>>> https://github.com/ververica/flink-cdc-connectors/issues/90 >>>>> >>>>> In the meantime, however, I've been trying to figure out if I can set >>>>> up an exclusion rule to fix this in my pom.xml file, without success. >>>>> >>>>> The `org.postgresql:postgresql` dependency is being added manually by >>>>> me to have a sink on a postgresql table and injected by the cdc connector >>>>> seemingly via its debezium connector dependency. >>>>> >>>>> Any guidance or hints I could follow would be really appreciated. >>>>> >>>>> -- >>>>> Sebastián Ramírez Magrí >>>>> >>>>> >>>> >>>> -- >>>> Sebastián Ramírez Magrí >>>> >>> >> >> -- >> Sebastián Ramírez Magrí >> > -- Sebastián Ramírez Magrí
Re: Conflicts between the JDBC and postgresql-cdc SQL connectors
Hi Jark!
Please find the full pom file attached.
Best Regards,
On Thu, 28 Jan 2021 at 03:21, Jark Wu wrote:
> Hi Sebastián,
>
> I think Dawid is right.
>
> Could you share the pom file? I also tried to
> package flink-connector-postgres-cdc with ServicesResourceTransformer, and
> the Factory file contains
>
> com.alibaba.ververica.cdc.connectors.postgres.table.PostgreSQLTableFactory
>
>
> Best,
> Jark
>
>
> On Tue, 26 Jan 2021 at 21:17, Sebastián Magrí
> wrote:
>
>> Thanks a lot for looking into it Dawid,
>>
>> In the
>> src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory
>> file I only see
>>
>> org.apache.flink.connector.jdbc.table.JdbcDynamicTableFactory
>>
>> Even after applying the ServicesResourceTransformer.
>>
>>
>> On Tue, 26 Jan 2021 at 11:58, Dawid Wysakowicz
>> wrote:
>>
>>> Hi,
>>>
>>> Unfortunately I am not familiar with the packaging of
>>> flink-connector-postgres-cdc. Maybe @Jark could help here?
>>>
>>> However, I think the problem that you cannot find the connector is
>>> caused because of lack of entry in the resulting Manifest file. If there
>>> are overlapping classes maven does not exclude whole dependencies, but
>>> rather picks the overlapping class from one of the two. Could you check if
>>> you see entries for all tables in
>>> src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory.
>>>
>>> If not, you could try applying the ServicesResourceTransformer[1]
>>>
>>> Best,
>>>
>>> Dawid
>>>
>>> [1]
>>> https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer
>>> On 26/01/2021 12:29, Sebastián Magrí wrote:
>>>
>>> Hi!
>>>
>>> I've reported an issue with the postgresql-cdc connector apparently
>>> caused by the maven shade plugin excluding either the JDBC connector or the
>>> cdc connector due to overlapping classes. The issue for reference is here:
>>>
>>> https://github.com/ververica/flink-cdc-connectors/issues/90
>>>
>>> In the meantime, however, I've been trying to figure out if I can set up
>>> an exclusion rule to fix this in my pom.xml file, without success.
>>>
>>> The `org.postgresql:postgresql` dependency is being added manually by me
>>> to have a sink on a postgresql table and injected by the cdc connector
>>> seemingly via its debezium connector dependency.
>>>
>>> Any guidance or hints I could follow would be really appreciated.
>>>
>>> --
>>> Sebastián Ramírez Magrí
>>>
>>>
>>
>> --
>> Sebastián Ramírez Magrí
>>
>
--
Sebastián Ramírez Magrí
http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;>
4.0.0
org.sebasmagri
stats-aggregator
1.0-SNAPSHOT
jar
Stats Aggregator
UTF-8
1.12.1
2.11
2.11.12
2.12.1
42.2.18.jre7
org.apache.flink
flink-scala_${scala.binary.version}
${flink.version}
provided
org.apache.flink
flink-streaming-scala_${scala.binary.version}
${flink.version}
provided
org.apache.flink
flink-clients_${scala.binary.version}
${flink.version}
provided
org.scala-lang
scala-library
${scala.version}
provided
com.alibaba.ververica
flink-connector-postgres-cdc
1.1.0
org.apache.flink
flink-connector-jdbc_${scala.binary.version}
${flink.version}
org.postgresql
postgresql
${postgresql.version}
org.apache.logging.log4j
log4j-slf4j-impl
${log4j.version}
runtime
org.apache.logging.log4j
log4j-api
${log4j.version}
runtime
org.apache.logging.log4j
log4j-core
${log4j.version}
runtime
org.apache.flink
flink-table-api-scala-bridge_${scala.binary.version}
${flink.version}
provided
org.apache.flink
flink-table-planner-blink_${scala.binary.version}
${flink.version}
provided
org.apache.flink
flink-streaming-scala_${scala.binary.version}
${flink.version}
provided
org.apache.maven.plugins
maven-shade-plugin
3.1.1
package
shade
org.apache.flink:force-shading
com.google.code.findbugs:jsr305
org.slf4j:*
org.apache.logging.log4j:*
Re: Conflicts between the JDBC and postgresql-cdc SQL connectors
Thanks a lot for looking into it Dawid, In the src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory file I only see org.apache.flink.connector.jdbc.table.JdbcDynamicTableFactory Even after applying the ServicesResourceTransformer. On Tue, 26 Jan 2021 at 11:58, Dawid Wysakowicz wrote: > Hi, > > Unfortunately I am not familiar with the packaging of > flink-connector-postgres-cdc. Maybe @Jark could help here? > > However, I think the problem that you cannot find the connector is caused > because of lack of entry in the resulting Manifest file. If there are > overlapping classes maven does not exclude whole dependencies, but rather > picks the overlapping class from one of the two. Could you check if you see > entries for all tables in > src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory. > > If not, you could try applying the ServicesResourceTransformer[1] > > Best, > > Dawid > > [1] > https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer > On 26/01/2021 12:29, Sebastián Magrí wrote: > > Hi! > > I've reported an issue with the postgresql-cdc connector apparently caused > by the maven shade plugin excluding either the JDBC connector or the cdc > connector due to overlapping classes. The issue for reference is here: > > https://github.com/ververica/flink-cdc-connectors/issues/90 > > In the meantime, however, I've been trying to figure out if I can set up > an exclusion rule to fix this in my pom.xml file, without success. > > The `org.postgresql:postgresql` dependency is being added manually by me > to have a sink on a postgresql table and injected by the cdc connector > seemingly via its debezium connector dependency. > > Any guidance or hints I could follow would be really appreciated. > > -- > Sebastián Ramírez Magrí > > -- Sebastián Ramírez Magrí
Conflicts between the JDBC and postgresql-cdc SQL connectors
Hi! I've reported an issue with the postgresql-cdc connector apparently caused by the maven shade plugin excluding either the JDBC connector or the cdc connector due to overlapping classes. The issue for reference is here: https://github.com/ververica/flink-cdc-connectors/issues/90 In the meantime, however, I've been trying to figure out if I can set up an exclusion rule to fix this in my pom.xml file, without success. The `org.postgresql:postgresql` dependency is being added manually by me to have a sink on a postgresql table and injected by the cdc connector seemingly via its debezium connector dependency. Any guidance or hints I could follow would be really appreciated. -- Sebastián Ramírez Magrí
Re: [Flink SQL] CompilerFactory cannot be cast error when executing statement
Thanks a lot Matthias! In the meantime I'm trying out something with the scala quickstart. On Fri, 22 Jan 2021 at 17:12, Matthias Pohl wrote: > Ok, to be fair, I just did some research on the error message and didn't > realize that you're working with binaries only. > > I tried to set it up on my machine to be able to reproduce your error. > Unfortunately, I wasn't able to establish the connection between Flink and > Postgres using your docker-compose.yml. > I'm going to cc Timo. Maybe, he has a guess what's causing this error. > > Best, > Matthias > > On Fri, Jan 22, 2021 at 4:35 PM Sebastián Magrí > wrote: > >> Hi Matthias! >> >> I went through that thread but as I'm just using the `apache/flink` >> docker image for testing I honestly couldn't figure out how I would do that >> since I don't have a pom file to edit. If it's possible to do it through >> the configuration I'd be glad if you could point me out in the right >> direction. >> >> Pretty evident I don't have a lot of experience with mvn or "modern" Java >> in general. >> >> :-) >> >> Thanks! >> >> On Fri, 22 Jan 2021 at 15:19, Matthias Pohl >> wrote: >> >>> Hi Sebastián, >>> have you tried changing the dependency scope to provided >>> for flink-table-planner-blink as it is suggested in [1]? >>> >>> Best, >>> Matthias >>> >>> [1] >>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Flink-1-10-exception-Unable-to-instantiate-java-compiler-td38221.html >>> >>> On Fri, Jan 22, 2021 at 4:04 PM Sebastián Magrí >>> wrote: >>> >>>> Hi! >>>> >>>> I'm trying out Flink SQL with the attached docker-compose file. >>>> >>>> It starts up and then I create a table with the following statement: >>>> >>>> CREATE TABLE mytable_simple ( >>>> `customer_id` INT >>>> ) WITH ( >>>> 'connector' = 'jdbc', >>>> 'url' = 'jdbc:postgresql://pgusr:pgpwd@postgres/pdgb', >>>> 'table-name' = 'mytable' >>>> ); >>>> >>>> However when I try to run this: >>>> >>>> select * from mytable_simple; >>>> >>>> I get the following error in the client: >>>> >>>> [ERROR] Could not execute SQL statement. Reason: >>>> java.lang.ClassCastException: org.codehaus.janino.CompilerFactory >>>> cannot be cast to org.codehaus.commons.compiler.ICompilerFactory >>>> >>>> At first I thought it could be an incompatibility issue with the >>>> libraries I was putting in, like the postgres-cdc library version, but even >>>> after leaving only the JDBC libraries in I still get the same error. >>>> >>>> It'd be great if you could give me some pointers here. >>>> >>>> Thanks! >>>> >>>> -- >>>> Sebastián Ramírez Magrí >>>> >>> >> >> -- >> Sebastián Ramírez Magrí >> > > Managing Directors: Yip Park Tung Jason, Jinwei (Kevin) Zhang, Karl Anton > Wehner > -- Sebastián Ramírez Magrí
Re: [Flink SQL] CompilerFactory cannot be cast error when executing statement
Hi Matthias! I went through that thread but as I'm just using the `apache/flink` docker image for testing I honestly couldn't figure out how I would do that since I don't have a pom file to edit. If it's possible to do it through the configuration I'd be glad if you could point me out in the right direction. Pretty evident I don't have a lot of experience with mvn or "modern" Java in general. :-) Thanks! On Fri, 22 Jan 2021 at 15:19, Matthias Pohl wrote: > Hi Sebastián, > have you tried changing the dependency scope to provided > for flink-table-planner-blink as it is suggested in [1]? > > Best, > Matthias > > [1] > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Flink-1-10-exception-Unable-to-instantiate-java-compiler-td38221.html > > On Fri, Jan 22, 2021 at 4:04 PM Sebastián Magrí > wrote: > >> Hi! >> >> I'm trying out Flink SQL with the attached docker-compose file. >> >> It starts up and then I create a table with the following statement: >> >> CREATE TABLE mytable_simple ( >> `customer_id` INT >> ) WITH ( >> 'connector' = 'jdbc', >> 'url' = 'jdbc:postgresql://pgusr:pgpwd@postgres/pdgb', >> 'table-name' = 'mytable' >> ); >> >> However when I try to run this: >> >> select * from mytable_simple; >> >> I get the following error in the client: >> >> [ERROR] Could not execute SQL statement. Reason: >> java.lang.ClassCastException: org.codehaus.janino.CompilerFactory cannot >> be cast to org.codehaus.commons.compiler.ICompilerFactory >> >> At first I thought it could be an incompatibility issue with the >> libraries I was putting in, like the postgres-cdc library version, but even >> after leaving only the JDBC libraries in I still get the same error. >> >> It'd be great if you could give me some pointers here. >> >> Thanks! >> >> -- >> Sebastián Ramírez Magrí >> > -- Sebastián Ramírez Magrí
[Flink SQL] CompilerFactory cannot be cast error when executing statement
Hi! I'm trying out Flink SQL with the attached docker-compose file. It starts up and then I create a table with the following statement: CREATE TABLE mytable_simple ( `customer_id` INT ) WITH ( 'connector' = 'jdbc', 'url' = 'jdbc:postgresql://pgusr:pgpwd@postgres/pdgb', 'table-name' = 'mytable' ); However when I try to run this: select * from mytable_simple; I get the following error in the client: [ERROR] Could not execute SQL statement. Reason: java.lang.ClassCastException: org.codehaus.janino.CompilerFactory cannot be cast to org.codehaus.commons.compiler.ICompilerFactory At first I thought it could be an incompatibility issue with the libraries I was putting in, like the postgres-cdc library version, but even after leaving only the JDBC libraries in I still get the same error. It'd be great if you could give me some pointers here. Thanks! -- Sebastián Ramírez Magrí docker-compose.yml Description: application/yaml
