回复： Flink DataStream 作业如何获取到作业血缘？

获取到Source 或者 DorisSink信息之后, 如何知道来自那个flink任务，好像不能获取到flinkJobId | | 阿华田 | | a15733178...@163.com | 签名由网易邮箱大师定制 在2024年02月26日 20:04，Feng Jin 写道： 通过 JobGraph 可以获得 transformation 信息，可以获得具体的 Source 或者 Doris Sink，之后再通过反射获取里面的 properties 信息进行提取。 可以参考 OpenLineage[1] 的实现. 1.

Re: Flink Checkpoint & Offset Commit

Hi Jacob, > I have multiple upstream sources to connect to depending on the business > model which are not Kafka. Based on criticality of the system and publisher > dependencies, we cannot switch to Kafka for these. Sounds like you want to implement some custom connectors, [1][2] may be

Re:Re:flink sql关联维表在lookup执行计划中的关联条件问题

好的，已经贴了sql片段 在 2024-03-08 11:02:34，"Xuyang" 写道： >Hi, 你的图挂了，可以用图床或者直接贴SQL > > > > >-- > >Best！ >Xuyang > > > > >在 2024-03-08 10:54:19，"iasiuide" 写道： > > > > > >下面的sql片段中 >ods_ymfz_prod_sys_divide_order 为kafka source表 >dim_ymfz_prod_sys_trans_log 为mysql为表

Re:Re: flink sql关联维表在lookup执行计划中的关联条件问题

你好，我们用的是1.13.2和1.15.4版本的，看了下flink ui，这两种版本针对下面sql片段的lookup执行计划中的关联维表条件是一样的 在 2024-03-08 11:08:51，"Yu Chen" 写道： >Hi iasiuide, >方便share一下你使用的flink版本与jdbc connector的版本吗？据我所了解，jdbc >connector在FLINK-33365[1]解决了lookup join条件丢失的相关问题。 > >[1] https://issues.apache.org/jira/browse/FLINK-33365 > >祝好~ >

Re: flink sql关联维表在lookup执行计划中的关联条件问题

Hi iasiuide, 方便share一下你使用的flink版本与jdbc connector的版本吗？据我所了解，jdbc connector在FLINK-33365[1]解决了lookup join条件丢失的相关问题。 [1] https://issues.apache.org/jira/browse/FLINK-33365 祝好~ > 2024年3月8日 11:02，iasiuide 写道： > > > > > 图片可能加载不出来，下面是图片中的sql片段 > .. > END AS trans_type, > >

Re:flink sql关联维表在lookup执行计划中的关联条件问题

Hi, 你的图挂了，可以用图床或者直接贴SQL -- Best！ Xuyang 在 2024-03-08 10:54:19，"iasiuide" 写道： 下面的sql片段中 ods_ymfz_prod_sys_divide_order 为kafka source表 dim_ymfz_prod_sys_trans_log 为mysql为表 dim_ptfz_ymfz_merchant_info 为mysql为表 flink web ui界面的执行计划片段如下：

Re:flink sql关联维表在lookup执行计划中的关联条件问题

图片可能加载不出来，下面是图片中的sql片段 .. END AS trans_type, a.div_fee_amt, a.ts FROM ods_ymfz_prod_sys_divide_order a LEFT JOIN dim_ymfz_prod_sys_trans_log FOR SYSTEM_TIME AS OF a.proc_time AS b ON a.bg_rel_trans_id = b.bg_rel_trans_id AND

flink sql关联维表在lookup执行计划中的关联条件问题

下面的sql片段中 ods_ymfz_prod_sys_divide_order 为kafka source表 dim_ymfz_prod_sys_trans_log 为mysql为表 dim_ptfz_ymfz_merchant_info 为mysql为表 flink web ui界面的执行计划片段如下： [1]:TableSourceScan(table=[[default_catalog, default_database, ods_ymfz_prod_sys_divide_order, watermark=[-(CASE(IS

Re:Re: Using User-defined Table Aggregates Functions in SQL

Hi, Jad. IIUC, TableAggregateFunfunction has not been supported in SQL. The original Flip[1] only implements it in Table API. You can send an email to dev maillist for more detail and create an improvement jira[2] for it. [1]

Re: Flink Checkpoint & Offset Commit

Hi Jacob. Flink uses "notification" to let an operator callback the completion of a checkpoint. After gathering all checkpoint done messages from TMs, JM sends a "notify checkpoint completed" RPC to all TMs. Operators will handle this notification, where checkpoint success callbacks are invoked.

Re:Re: Handling late events with Table API / SQL

Hi, Sunny. A watermark always comes from one subtask of this window operator's input(s), and this window operator will retain all watermarks about multi input subtasks. The `currentWatermark` in the window operator is the min value of these watermarks. -- Best！ Xuyang At

Re: Re:RE: RE: flink cdc动态加表不生效

Hi， casel chan, 社区已经对增量框架实现动态加表（https://github.com/apache/flink-cdc/pull/3024 ），预计3.1对mongodb和postgres暴露出来，但是Oracle和Sqlserver目前并没暴露，你可以去社区参照这两个框架，将参数打开，并且测试和适配。 Best, Hongshun

Re:Window properties can only be used on windowed tables

public static void main(String[] args) throws Exception { StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); StreamTableEnvironment tEnv = StreamTableEnvironment.create(env); ListPerson list = new ArrayList();list.add(new Person("Fred",35));

Re:Window properties can only be used on windowed tables

在非窗口化的表上使用窗口属性 At 2024-03-08 09:28:10, "ha.fen...@aisino.com" wrote: >public static void main(String[] args) { >StreamExecutionEnvironment env = > StreamExecutionEnvironment.getExecutionEnvironment(); >StreamTableEnvironment tEnv = StreamTableEnvironment.create(env); >

Re:Window properties can only be used on windowed tables

Hi, fengqi. 这看起来像是select语句中，不能直接使用非来源于window agg的proctime或者event函数。目前不确定这是不是预期行为，方便的话可以在社区jira[1]上提一个bug看看。 快速绕过的话，可以试试下面的代码： DataStream flintstones = env.fromCollection(list); // Table select = table.select($("name"), $("age"), $("addtime").proctime()); Table table = tEnv.fromDataStream(

Re: Re: Running Flink SQL in production

Hi, If you need to use Flink SQL in a production environment, I think it would be better to use the Table API [1] and package it into a jar. Then submit the jar to the cluster environment. [1] https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/common/#sql Best, Feng On

Re: Using User-defined Table Aggregates Functions in SQL

Hi Junrui, Thank you for the pointer. I had read that page, and I can use the function with the Java Table API ok, but I'm trying to use the Top2 accumulator with a SQL function. I can't use a left lateral join on it since the planner fails with "not a table function". I don't think a join is the

Fwd: Flink Checkpoint & Offset Commit

Hello, I am implementing proof of concepts based Flink realtime streaming solutions. I came across below lines in out-of-the-box Flink Kafka connector documents. *https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/kafka/*

Re: Using User-defined Table Aggregates Functions in SQL

Hi Jad, You can refer to the CREATE FUNCTION section ( https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/create/#create-function) and the Table Aggregate Functions section (

Re: Handling late events with Table API / SQL

Thanks for the response! Sad that that side output for late data is not supported in Table API and SQL. I will start the discussions regarding this. In the meanwhile, I am trying to use the built-in function CURRENT_WATERMARK(rowtime) to be able to collect late data. The scenario I have is : I

Using User-defined Table Aggregates Functions in SQL

Hi, The docs don't mention the correct syntax for using UDTAGGs in SQL. Is it possible to use them with SQL? Thanks, Jad Naous Grepr, CEO/Founder ᐧ

Re: TTL in pyflink does not seem to work

Note, that in Java code, it prints `State: Null`, `State: Null`, as I was expecting in, unlike pyflink code On Mar 7, 2024 at 15:59 +0400, Ivan Petrarka , wrote: > Hi! I’ve created a basic pyflink pipeline with ttl and it does not seem to > work. I have reproduced the exact same code in Java and

TTL in pyflink does not seem to work

Hi! I’ve created a basic pyflink pipeline with ttl and it does not seem to work. I have reproduced the exact same code in Java and it works! Is this a pyflink bug? If so - how can I report it? If not - what can I try to do? Flink: 1.18.0 image: flink:1.18.0-scala_2.12-java11 Code to

使用avro schema注册confluent schema registry失败

我使用注册kafka topic对应的schema到confluent schema registry时报错，想知道问题的原因是什么？如何fix? io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Schema being registered is incompatible with an earlier schema for subject "rtdp_test-test_schema-value", details:

Re:Re: Running Flink SQL in production

Hi. Hmm, if I'm mistaken, please correct me. Using a SQL client might not be very convenient for those who need to verify the results of submissions, such as checking for exceptions related to submission failures, and so on. -- Best！ Xuyang 在 2024-03-07 17:32:07，"Robin