Re: SQL Engine Type inference when extending AsyncTableFunction class twice.

2022-09-16 Thread Jonathan Weaver
ltiple extends in the future, instead of just allowing a direct single subclass? On Thu, Sep 15, 2022 at 4:42 PM Jonathan Weaver wrote: > I am having an issue with the automatic type inference with SQL engine in > an AsyncTableFunction class. > > I am extending AsyncTableFunction in a Bas

SQL Engine Type inference when extending AsyncTableFunction class twice.

2022-09-15 Thread Jonathan Weaver
I am having an issue with the automatic type inference with SQL engine in an AsyncTableFunction class. I am extending AsyncTableFunction in a BaseClass (common code). Then extending again for some specific implementations. FinalClass extends BaseClass If I use BaseClass it correctly infers the

Re: Flink task lifecycle listener/hook/SPI

2022-08-04 Thread Jonathan Weaver
I think the piece you are missing is you cannot guarantee where the function will run in general. It may get sent to several different task executors, and each executor may not be on the same machine or JVM so the code has to init once distributed at least once. You have to think that every functi

Source API question around idle (expensive) SplitFetchers being shutdown.

2022-03-04 Thread Jonathan Weaver
I am working on developing a custom source with the new Source api. What I'm noticing is that during periods of low incoming data it repeatedly will shutdown and restart the fetchers when the split assignments are empty and periodically added. I get log message such as org.apache.flink.connector

Looking for advice on moving Datastream job to Table SQL

2022-03-01 Thread Jonathan Weaver
I'm doing a POC on moving an existing Datastream API job to use Table SQL to make it more accessible for some of my teammates. However I'm at a loss on how to handle watermarking in a similar way to how it was handled in the Datastream API. In the existing job a CDC stream is read, and 3 SQL tabl

Possible BUG in 1.15 SQL JSON_OBJECT()

2022-02-24 Thread Jonathan Weaver
Using the latest SNAPSHOT BUILD. If I have a column definition as .column( "events", DataTypes.ARRAY( DataTypes.ROW( DataTypes.FIELD("status", DataTypes.STRING().notNull()), DataTypes.FIELD("times

Re: Exception Help

2022-02-16 Thread Jonathan Weaver
ot a correct > type, the serializer just fails hard. > > Please correct me if I misunderstood what you're doing, and in case > provide more info about what your goal and how you've implemented the job. > > FG > > On Wed, Feb 16, 2022 at 4:02 AM Jonathan Weaver &

Re: Exception Help

2022-02-15 Thread Jonathan Weaver
or the community to help. > > On Tue, 15 Feb 2022, 23:33 Jonathan Weaver, > wrote: > >> I'm getting the following exception running locally from my IDE >> (IntelliJ) but seems to not occur >> when running on a cluster. I'm assuming it may be related to memory &g

Exception Help

2022-02-15 Thread Jonathan Weaver
I'm getting the following exception running locally from my IDE (IntelliJ) but seems to not occur when running on a cluster. I'm assuming it may be related to memory settings on the runtime (machine has 64GB of ram avail) but not sure what setting to try and change. Caused by: java.lang.IndexOutOf

Parallelism of Flink SQL LookupTableSource in 1.14 ..

2022-01-10 Thread Jonathan Weaver
I'm attempting to do a proof of concept conversion of a DataStream based Flink program over to using almost entirely Table SQL. I have a primary CDC stream (an unbounded scan table source) that does two joins to LookupTableSource tables and then on to a sink. In the datastream program the only wa