Re: Schema Evolution of POJOs fails on Field Removal

2022-05-18 Thread David Jost
> On 18. May 2022, at 15:34, Alexis Sarda-Espinosa > wrote: > > Hi David, > > Please refer to https://issues.apache.org/jira/browse/FLINK-21752 > > Regards, > Alexis. Hi Alexis and Hangxiang, thank you both for you quick responses. Following Alexis' link, I noticed, that we were still on

Schema Evolution of POJOs fails on Field Removal

2022-05-18 Thread David Jost
Hi, we currently have an issue, where our job fails to restart from a savepoint, after we removed a field from a serialised (POJO) class. According to [0], this kind of evolution is supported, but it sadly only works when adding, but not removing fields. I was hoping, someone here might be

How to mock new DataSource/Sink

2022-07-04 Thread David Jost
Hi, we are currently looking at replacing our sinks and sources with the respective counterparts using the 'new' data source/sink API (mainly Kafka). What holds us back is that we are not sure how to test the pipeline with mocked sources/sinks. Up till now, we somewhat followed the 'Testing

Re: How to mock new DataSource/Sink

2022-07-06 Thread David Jost
> On 5. Jul 2022, at 01:48, Alexander Fedulov wrote: > > Hi David, > > I started working on FLIP-238 exactly with the concerns you've mentioned in > mind. It is currently in development, feel free to join the discussion [1]. > If you need something ASAP and are not interested in rate-limiting

Re: Slow Tests in Flink 1.15

2022-09-09 Thread David Jost
nt in Kafka record > deserializer it is hit but very slow, roughly 3 records per 5 minute (the > topic was pre-populated) > > No table/sql API, only stream API > From: Chesnay Schepler > Sent: Wednesday, September 7, 2022 5:20 AM > To: Alexey Trenikhun ; David Jost ; >

Re: Slow Tests in Flink 1.15

2022-09-09 Thread David Jost
al in some places and disabled checkpoints altogether in > some other tests > > maciek > > On 09.09.2022 10:44, David Jost wrote: >> Hey, sorry for not coming back to this earlier, but I was hoping to better >> isolate the problem for analysis. >> >> Ma

Slow Tests in Flink 1.15

2022-09-05 Thread David Jost
Hi, we were going to upgrade our application from Flink 1.14.4 to Flink 1.15.2, when we noticed, that all our job tests, using a MiniClusterWithClientResource, are multiple times slower in 1.15 than before in 1.14. I, unfortunately, have not found mentions in that regard in the changelog or

PyFlink Performance

2024-04-24 Thread David Jost
Hi, I am currently evaluating PyFlink in comparison to Java and did some various tests, mainly comparing identical pipelines with focus on throughput. For me it seems, that PyFlink is generally worse for wear and seems to reach its limits in throughput at a point where Java still has resources