Reading Iceberg Tables in Batch mode performance

2024-03-06 Thread Charles Tan
Hi all, I have been looking into using Flink in batch mode to process Iceberg tables. I noticed that the performance for queries in Flink's batch mode is quite slow, especially when compared to Spark. I'm wondering if there are any configurations that I'm missing to get better performance out of

Possible bug when converting table to upsert changelog stream

2023-09-18 Thread Charles Tan
|Order_1 | ++--++ Output: 6> +I[Order_1, 1] 6> -U[Order_1, 1] Here is the link to a github repository that contains an example. ( https://github.com/charles-tan/flink-upsert-changelog-bug) Thanks, Charles

Reading KafkaSource state from a savepoint using the State Processor API

2023-05-23 Thread Charles Tan
nt. (https://github.com/charles-tan/flink-state-processor-example) Thanks, Charles

FlinkKinesisConsumer cannot consume events from Kinesalite with EFO enabled

2023-04-25 Thread Charles Tan
Hi all, I’ve tried a simple Flink application which uses FlinkKinesisConsumer. I noticed that when trying to consume from Kinesalite using the FlinkKinesisConsumer with EFO enabled, I run into SSL handshake errors. This is despite disabling certificate validation. Has anybody successfully tested

Re: LEFT and FULL interval joins in Flink SQL leads to very out of order outputs

2023-03-20 Thread Charles Tan
on interval joins and can explain cleanUpTime and minCleanUpInterval. Thanks, Charles On Tue, Mar 14, 2023 at 1:44 PM Charles Tan wrote: > Hi everyone, > > I have been playing around with Flink SQL’s interval joins and noticed > that some outputs from unmatched LEFT or FULL joins are arrivin

LEFT and FULL interval joins in Flink SQL leads to very out of order outputs

2023-03-14 Thread Charles Tan
Hi everyone, I have been playing around with Flink SQL’s interval joins and noticed that some outputs from unmatched LEFT or FULL joins are arriving much later than I expected. Take the following query for example: SELECT * FROM orders o LEFT JOIN shipments s ON (o.orderID = s.orderID) AND

Re: UDFs classloading changes in 1.16

2023-01-20 Thread Charles Tan
updated both tickets with comments and new code snippets. Is there another way to load UDFs in Flink 1.16? Code examples for reference: https://github.com/charles-tan/udfs-flink-1.16/blob/main/src/test/java/com/example/UDFTest.java Thanks, Charles On Fri, Nov 4, 2022 at 10:51 AM Alexander Fedulov