You can also check the apache paimon project https://paimon.apache.org/ (previously known as Flink Table Store). Might help in some scenarios
On Mon, Aug 28, 2023 at 5:05 AM liu ron <ron9....@gmail.com> wrote: > Hi, Nirmal > > Flink SQL is standard ANSI SQL and extends upon it. Flink SQL provides > rich Join and Aggregate syntax including Regular Streaming Join, Interval > Join, Temporal Join, Lookup Join[2], Window Join[3], unbounded group > aggregate[4] and window aggregate[5], and so on. Theoretically, it can > support your complex query needs. You need to spend some time learning > these syntaxes so that you can see which ones meet your needs from a > business point of view! > > [1] > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/tableapi/ > [2] > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/joins/ > [3] > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/window-join/ > [4] > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/group-agg/ > [5] > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/window-agg/ > > Best, > Ron > > Nirmal Chhatrala <nirmal.chhatr...@gmail.com> 于2023年8月28日周一 04:36写道： > >> Hello! >> >> We have a use case requirement to implement complex joins and aggregation >> on multiple sql tables. Today, it is happening at SQLServer level which is >> degrading the performance of SQLServer Database. >> >> Is it a good idea to implement it through Apache Flink Table API for >> real-time data joins? >> >> ie: there are 10 different sql tables with complex join queries with 10 >> different conditions. (Approx. - The accumulative 10 tables size is ~100GB >> and could grow in future). We don't have experience with Flink Table API >> and never measured the performance on such a huge dataset. >> >> Looking for your advice and best alternative if it is not suitable with >> Apache Flink Table API. >> >> Thanks! >> >
