RE: DI in flink

Hi Yashoda, I use Spring-Boot to setup my job networks and DI-compose streaming components like operators/functions etc. Important part is that all components need to be serializable in order for this to work. Specific task implementations are a little more difficult (little experience) to set

[SUMMARY] Flink 1.17 Release Sync 2/14/2023

Hi devs and users, I'd like to share some highlights from Flink 1.17 release sync on 2/14/2023. Release testing: - The original deadline of cross-team testing is Feb 21, 2023 (next Tuesday). We will monitor the status throughout the week and hopefully conclude everything before the deadline. -

Re: Reusing the same OutputTag in multiple ProcessFunctions

I can't respond to the python-specific aspects of this situation, but I don't believe you need to use the same OutputTag instance. It should be enough that the various tag instances involved all have the same String id. (That's why the id exists.) David On Tue, Feb 14, 2023 at 11:51 AM Andrew

Re: regular join每条流单独设置ttl

你好, 目前 Flink SQL 还不支持为每条流单独设置 state TTL, 不过社区计划支持这个功能, 最近就会有 FLIP 提出, 也欢迎参与讨论. Best regards, Jane On Wed, Feb 15, 2023 at 11:13 AM Jason_H wrote: > 大家好， > 我遇到一个问题，在使用flinksql做双流join时，我用的是常规的regular > join，分别是一条数据流，一条维表流，现在我想单独给数据流设置ttl，请教一下可以怎么做，flink版本是1.15.2 > > > | | > Jason_H > | > | >

Re: DI in flink

Thanks Austin. I can make use of Rich functions to solve my problem. Thanks Yashoda On Wed, Feb 15, 2023 at 12:42 AM Austin Cawley-Edwards < austin.caw...@gmail.com> wrote: > (note: please keep user@flink.apache.org included in replies) > > Ah, I see. Then no, this is not provided by Flink.

regular join每条流单独设置ttl

大家好， 我遇到一个问题，在使用flinksql做双流join时，我用的是常规的regular join，分别是一条数据流，一条维表流，现在我想单独给数据流设置ttl，请教一下可以怎么做，flink版本是1.15.2 | | Jason_H | | hyb_he...@163.com |

Reusing the same OutputTag in multiple ProcessFunctions

Hi, I'm attempting to implement a generic error handling ProcessFunction in pyflink. Given a user provided function, I want to invoke that function for each element in the DataStream, catch any errors thrown by the function, convert those errors into events, and then emit those event errors to a

Re: DI in flink

(note: please keep user@flink.apache.org included in replies) Ah, I see. Then no, this is not provided by Flink. When I've used dependency inject with Flink in the past, I instantiated everything in the `open()` method of the Flink Rich* classes. Could you solve this by having a common base Sink

Re: DI in flink

What would be the benefits and features over what can be done in user land? On Tue, Feb 14, 2023 at 10:41 Yashoda Krishna T wrote: > Hi Austin > > Yes this can be done in Usrr land. > Can we do it in flink land too? > > Thanks > Yashoda > > On Tue, 14 Feb 2023, 9:05 pm Austin Cawley-Edwards, <

Re: DI in flink

Hey Yashoda, This can be done in userland (eg with Dagger ) unless you're wanting Flink to do something in addition? Best, Austin On Tue, Feb 14, 2023 at 10:01 AM Yashoda Krishna T < yashoda.kris...@unbxd.com> wrote: > Does flink support dependency injection in flink task

DI in flink

Does flink support dependency injection in flink task functions in java? If not is there an alternative?

Re: Watermark in global commit

Hi, I'm not expert on Flink specifially, but your approach might be easier solve when broken down into two steps - create a "stable" input to downstream processing, this might include a specific watermark. In Flink, the "stability" of input for downstream processing is ensured by a

Metrics or runtimeContext in global commit

Dear flink team, I would like to use metrics (which are then written to an influxdb) in the method org.apache.flink.api.connector.sink2.Committer::commit(Collection> committables) that I use for global commit. I use the helper method StandardSinkTopologies.addGlobalCommitter(...) to define

Watermark in global commit

Dear flink team, I am facing the following problem: I would need to write events to parquet files using the FileSink. Subsequently, I want to do something else in a global commit where I need the corresponding watermark. However, the org.apache.flink.connector.file.sink.FileSink forces the

Re: 运行中的作业状态清除操作

Hi 这个是无法办到的，建议还是重启 On Mon, Feb 13, 2023 at 4:33 PM Jason_H wrote: > 遇到的问题：在flink中，使用状态的地方已经设置了ttl， 但是上下游把数据清空了，然后想重新测试发现， > flink计算的结果不是从0开始累计的，是基于之前的结果继续累计的，这种情况在不重启作业的情况下 有什么办法处理吗？ > > > 具体来说就是，在不停止flink作业的情况下，怎么清楚运行中的flink作业的状态数据，以达到计算结果归0开始累计。 > > >