Re:如何在Flink Connector Source退出时清理资源

[Xuyang]

Hi, 
看一下你的DynamicTableSource实现的类，如果你用的是InputFormat的旧source（用的是类似InputFormatProvider.of），可以使用InputFormat里的close方法；
如果用的是flip-27的source（用的是类似SourceProvider.of），SplitReader里也有一个close方法










--

Best！
Xuyang





在 2023-10-24 11:54:36，"jinzhuguang"  写道：
>版本：Flink 1.16.0
>
>需求：在某个source结束退出时清理相关的资源。
>
>问题：目前没有找到Source退出时相关的hook函数，不知道在哪里编写清理资源的代码。
>
>恳请大佬们指教。

如何在Flink Connector Source退出时清理资源

[jinzhuguang]

版本：Flink 1.16.0

需求：在某个source结束退出时清理相关的资源。

问题：目前没有找到Source退出时相关的hook函数，不知道在哪里编写清理资源的代码。

恳请大佬们指教。

Re: Problems with the state.backend.fs.memory-threshold parameter

[rui chen]

Hi，Zakelly
Thank you for your answer.

Best，
rui


Zakelly Lan  于2023年10月13日周五 19:12写道：

> Hi rui,
>
> The 'state.backend.fs.memory-threshold' configures the threshold below
> which state is stored as part of the metadata, rather than in separate
> files. So as a result the JM will use its memory to merge small
> checkpoint files and write them into one file. Currently the
> FLIP-306[1][2] is proposed to merge small checkpoint files without
> consuming JM memory. This feature is currently being worked on and is
> targeted for the next minor release (1.19).
>
>
> Best,
> Zakelly
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-306%3A+Unified+File+Merging+Mechanism+for+Checkpoints
> [2] https://issues.apache.org/jira/browse/FLINK-32070
>
> On Fri, Oct 13, 2023 at 6:28 PM rui chen  wrote:
> >
> > We found that for some tasks, the JM memory continued to increase. I set
> > the parameter of state.backend.fs.memory-threshold to 0, and the JM
> memory
> > would no longer increase, but many small files might be written in this
> > way. Does the community have any optimization plan for this area?
>