Unsubscribe

[Pritam Agarwala]

Unsubscribe

[Nikolaos Paraskakis]

Unsubscribe

[Swagat Mishra]

unsubscribe

[Vishnu I P]

unsubscribe

Unsubscribe

[Hou Ying]

Unsubscribe

Unsubscribe

[HyungHo Kim]

Unsubscribe


This communication contains Aeris Confidential Information. If it has been sent 
to you in error, please reply to advise the sender of the error and then 
immediately delete this message.

Unsubscribe

[Bryan Cantos]

Unsubscribe

Unsubscribe

[Pritam Agarwala]

Please Unsubscribe

Unsubscribe

[SAURABH VERMA]

Unsubscribe

Thanks & Regards,
Saurabh Verma,
India

Unsubscribe

[naga sudhakar]

Unsubscribe

Unsubscribe

[李佳宸]

Unsubscribe

Unsubscribe

[lin wang]

Unsubscribe

Unsubscribe

[Andra Lungu]

Unsubscribe

Unsubscribe

[Titus Rakkesh]

Unsubscribe

Unsubscribe

[Andra Lungu]

Unsubscribe

Unsubscribe

[Sanchay Sharma via user]

Unsubscribe



If you are not the intended recipient or have received this message in error, 
please notify the sender and permanently delete this message and any 
attachments.

Unsubscribe.

[lin wang]

Unsubscribe.

Unsubscribe

[localnohost8...@163.com]

Unsubscribe

Unsubscribe

[localnohost8...@163.com]

Unsubscribe

Unsubscribe

[HyungHo Kim]

Unsubscribe



This communication contains Aeris Confidential Information. If it has been sent 
to you in error, please reply to advise the sender of the error and then 
immediately delete this message.

Unsubscribe.

[Luyi Wang]

-Luyi.

Unsubscribe

[jasvendra kumar]

Unsubscribe

Thanks & Regards
Jasvendra

Unsubscribe

[Olga Gorun]

Unsubscribe,
Regards,
Olga Gorun

Unsubscribe

[Selvaraj chennappan]

Unsubscribe

Regards,
Selvaraj C

Unsubscribe

[Indigo Blue]

Unsubscribe

Unsubscribe

[Rohan Thimmappa]

Unsubscribe

-- 
Thanks
Rohan

Unsubscribe

[Indigo Blue]

Unsubscribe

Unsubscribe

[naga sudhakar]

Unsubscribe

Unsubscribe

[Nadia Mujeeb]

Thanks

Unsubscribe

[Phil Stavridis]

Unsubscribe

Unsubscribe

[jasvendra kumar]

Thanks & Regards
Jasvendra

Unsubscribe

[Vince Castello via user]

Unsubscribe

[Nadia Mujeeb]

bye

Unsubscribe

[Ecaterina-Sonia Lazarescu]

Unsubscribe

unsubscribe

[Phil Stavridis]

unsubscribe

unsubscribe

[Xavier]

unsubscribe

unsubscribe

[Xavier]

unsubscribe

-- 

Best Regards,
*Xavier*

unsubscribe

[Phil Stavridis]

unsubscribe

Unsubscribe

[3pang zhu]

Unsubscribe

Unsubscribe

[Sijun Yang]

Unsubscribe

Unsubscribe

[王业楼]

Unsubscribe
发自我的 iPhone

Re: Unsubscribe

[Zhanghao Chen]

Please send email to user-unsubscr...@flink.apache.org if you want to
unsubscribe the mail from user@flink.apache.org.

Best,
Zhanghao Chen

From: Mujahid Niaz 
Sent: Wednesday, February 5, 2025 9:29
Cc: user@flink.apache.org 
Subject: Unsubscribe

Unsubscribe

Unsubscribe

[Mujahid Niaz]

Unsubscribe

Unsubscribe

[Lin Hou]

Unsubscribe

[Muazim Wani]

Unsubscribe

Unsubscribe

[iñigo san jose]

Unsubscribe

Unsubscribe

[Hatem Mostafa]

Unsubscribe

Unsubscribe

[Samuel S]

Unsubscribe

[Samuel S]

Unsubscribe

Unsubscribe

[Shubham Badkur]

Unsubscribe

Unsubscribe

[Srinivas V]

Unsubscribe

Unsubscribe

[Songyc5]

Unsubscribe

Unsubscribe

[Nitin Chauhan]

Unsubscribe

Request to unsubscribe

[Niki Thomas]

Please unsubscribe.

-- 
Thanks and Regards
Nikita Thomas

Unsubscribe

[fan xiao]

Unsubscribe

Unsubscribe

[Sweta Kalakuntla via user]

Unsubscribe

unsubscribe

[simon money via user]

 
unsubscribe  

Unsubscribe

[Luyi Wang]

Unsubscribe

Unsubscribe

[Lei Chen]

Unsubscribe

Unsubscribe

[HyungHo Kim]

Unsubscribe


This communication contains Aeris Confidential Information. If it has been sent 
to you in error, please reply to advise the sender of the error and then 
immediately delete this message.

Unsubscribe

[simon money via user]

Unsubscribe 


Sent from Yahoo Mail for iPhone

unsubscribe

[Akos Barabas]

Unsubscribe

unsubscribe

[Wai Chee Yau]

Unsubscribe 

> On 18 Nov 2024, at 6:22 PM, Xavier  wrote:
> 
> 
> unsubscribe
> 
> --
> 
> Best Regards,
> Xavier

unsubscribe

[Xavier]

unsubscribe

-- 

Best Regards,
*Xavier*

unsubscribe

[高岩]

unsubscribe

Unsubscribe

[Anita Kumari]

Unsubscribe

unsubscribe

[95chenjz]

unsubscribe
- 原始邮件 -
发件人：Simon Frei 
收件人：Alexis Sarda-Espinosa , user 

主题：Re: unexpected high mem. usage / potential config misinterpretation
日期：2024年09月19日 04点11分


Hi Alexis,
 
Thanks for chiming in. To my knowledge and experience that’s not how it
works – we can consistently succeed to run this job in batch mode on less data, 
but still considerably more data than would fit in memory. Also even
 on the full data, the initial source tasks always succeed. To my understanding 
flink in batch mode writes all the data produced by a source/operator to disk. 
It even can run on less memory than the equivalent streaming job, as it can 
process a small part of
 the data in a partition at once from/to disk, without having to keep the state 
of all the data at once in memory. At least that’s my observation and 
understanding of e.g. this documentation:

https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/execution_mode/#batch-execution-mode
 
Best,
Simon
 

From:
Alexis Sarda-Espinosa 

Date: Wednesday, 18 September 2024 at 19:45

To: Simon Frei 

Cc: user 

Subject: Re: unexpected high mem. usage / potential config misinterpretation


 


Hi Simon, 

 


I hope someone corrects me if I'm wrong, but just based on "batch mode 
processing terabytes of data", I feel batch mode may be the issue. I am under 
the impression that batch mode forces everything emitted by the sources to RAM 
before any
 downstream operators do anything, so even if each parallel task of your source 
runs in a different task manager and loads a subset of your data, they might 
each end up trying to load terabytes into RAM.


 


Hopefully someone more knowledgeable about batch mode can comment. 


 


Regards, 


Alexis. 

 


On Wed, 18 Sept 2024, 18:04 Simon Frei,  wrote:




Hi,
 
tl;dr:
Flink batch streaming API job resident memory usage grows far beyond 
expectations, resulting in system OOM kill/JVM native memory allocation failure 
- would appreciate
 a look over our config/assumptions to potentially spot any obvious mistakes.
 
Longer form:
 
My colleague and I are troubleshooting a large batch job for a long time, and 
still experience behaviour around flinks memory usage we cannot explain. My hope
 is that by explaining our configuration and observations, someone can spot a 
misconception. And in the ideal case I can then send a PR for the documentation 
to hopefully make that misconception less likely for other users.
 
I'll start with an overview/"story-like" form, and then below that are some 
numbers/configs.
 
This is a streaming job run in batch mode, processing terabytes of data 
sourcing and sinking to compressed files in S3. In between there are a few 
simple decoding
 and filter operations, then two processors with our main business logic and 
finally a few simple transformations and reduce steps. While reduce and sink 
writer tasks run, we encounter much more resident memory usage of the flink TM 
java process than expected
 from configuration, i.e. higher than the configured process memory. And that 
leads to failures, either the system OOM killer intervening or the JVM not 
being able to mmap. I know that the writers do use native memory, e.g. for avro 
deflate compression, which
 is a native method. Also the IO likely uses some native memory. We now 
configure 5g of task off-heap memory to compensate for any such native memory 
usage, but still encounter higher memory usage. Even 5g seems way too much for 
some compression buffers and
 IO, let alone more than that. So currently my main theory is that I 
misunderstand something about the memory related config. E.g. that task slots 
factor into used/allocated memory.
 
We during the late stages of the job, i.e. during reduce and sink operations, 
we observe much higher memory usage than expected. The increase in memory usage
 isn't happening slowly, gradually over time, but quickly when those tasks 
start. This is an example of ps output for one TM:
 

PID %CPU %MEM    VSZ   RSS TTY  STAT START   TIME COMMAND

    14961  130 94.8 32213536 30957104 ?   Sl   Sep12 7453:03 
/usr/lib/jvm/java-11-amazon-corretto.aarch64/bin/java -Xmx7435661840 
-Xms7435661840 -XX:MaxDirectMemorySize=9663676416 
-XX:MaxMetaspaceSize=268435456 
-Dlog.file=/var/log/hadoop-yarn/containers/application_1724829444792_0007/container_1724829444792_0007_01_000330/taskmanager.log
 -Dlog4j.configuration=file:./log4j.properties 
-Dlog4j.configurationFile=file:./log4j.properties 
org.apache.flink.yarn.YarnTaskExecutorRunner -D 
taskmanager.memory.network.min=2147483648b -D taskmanager.cpu.cores=4.0 -D 
taskmanager.memory.task.off-heap.size=5368709120b
 -D taskmanager.memory.jvm-metaspace.size=268435456b -D external-resources=none 
-D taskmanager.memory.jvm-overhead.min=1073741824b -D 
taskmanager.memory.framework.off-heap.size=2147483648b -D 
taskmanager.memory.network.max=2147483648b -D 
taskmanager.memory

Unsubscribe

[95chenjz]

Unsubscribe
- 原始邮件 -
发件人：Ganesh Walse 
收件人：user@flink.apache.org
主题：Hikari data source and jdbc connection issue
日期：2024年09月04日 09点05分

Hi All,
I am using Hikari data source with ojdbc 8 to connect with database in flink 
application.
I have series of jobs which other java application submit to my jobmanager.And 
on each job submission I will connect to database after that my job manager and 
task manager metaspace increases by 10mb and it will never get released.
Due to above scenario my application gets oom error after certain number of 
jobs.

Please help me here to get out of this situation.
Thanks and regards,Ganesh walse

Please Unsubscribe anita.kum...@insight.com

[Anita Kumari]

Unsubscribe

Unsubscribe

[amenreet sodhi]

Unsubscribe

Unsubscribe

[smallwong]

Unsubscribe

Unsubscribe

[amenreet sodhi]

Unsubscribe

Unsubscribe

[Angel Francisco Orta]

Unsubscribe

Unsubscribe

[yadong shi]

Unsubscribe

Unsubscribe

[Phil Stavridis]

Unsubscribe

unsubscribe

[Phil Stavridis]

kirankumarkathe...@gmail.com-unsubscribe

[Kiran Kumar Kathe]

Kindly un subscribe for this gmail account kirankumarkathe...@gmail.com

Unsubscribe

[Mark Petronic]

Re: unsubscribe

[Yuxin Tan]

Hi, Ralph,

Please send an email to user-unsubscr...@flink.apache.org to unsubscribe
the user mailing list. [1]

[1] https://flink.apache.org/what-is-flink/community/

Best,
Yuxin


Ralph Matthias Debusmann  于2023年11月15日周三
06:18写道：

>
>

Re: unsubscribe

[Junrui Lee]

Hi,

Please send email to user-unsubscr...@flink.apache.org if you want to
unsubscribe the mail from user@flink.apache.org.

Ralph Matthias Debusmann  于2023年11月15日周三
07:29写道：

>
>

unsubscribe

[Ralph Matthias Debusmann]

Re: Unsubscribe from user list.

[Hang Ruan]

Hi,
Please send email to user-unsubscr...@flink.apache.org if you want to
unsubscribe the mail from user@flink.apache.org, and you can refer [1][2]
for more details.

Best,
Hang

[1]
https://flink.apache.org/zh/community/#%e9%82%ae%e4%bb%b6%e5%88%97%e8%a1%a8
[2] https://flink.apache.org/community.html#mailing-lists


bharghavi vajrala  于2023年10月21日周六 19:03写道：

> Team,
>
> Please unsubscribe my email id.
>
> On Thu, Oct 19, 2023 at 6:25 AM jihe18717838093 <18717838...@126.com>
> wrote:
>
>> Hi team,
>>
>>
>>
>> Could you please remove this email from the subscription list?
>>
>>
>>
>> Thank you!
>>
>>
>>
>> Best,
>>
>> Minglei
>>
>

Re: Unsubscribe from user list.

[bharghavi vajrala]

Team,

Please unsubscribe my email id.

On Thu, Oct 19, 2023 at 6:25 AM jihe18717838093 <18717838...@126.com> wrote:

> Hi team,
>
>
>
> Could you please remove this email from the subscription list?
>
>
>
> Thank you!
>
>
>
> Best,
>
> Minglei
>

Re: Unsubscribe from user list.

[Zakelly Lan]

Hi Lijuan,

Please send email to user-unsubscr...@flink.apache.org if you want to
unsubscribe the mail from user@flink.apache.org.


Best,
Zakelly

On Thu, Oct 19, 2023 at 6:23 AM Hou, Lijuan via user
 wrote:
>
> Hi team,
>
>
>
> Could you please remove this email from the subscription list? I have another 
> email (juliehou...@gmail.com) subscribed as well. I can use that email to 
> receive flink emails.
>
>
>
> Thank you!
>
>
>
> Best,
>
> Lijuan

Unsubscribe from user list.

[jihe18717838093]

Hi team, 

 

Could you please remove this email from the subscription list? 

 

Thank you!

 

Best,

Minglei

Unsubscribe from user list.

[Hou, Lijuan via user]

Hi team,

Could you please remove this email from the subscription list? I have another 
email (juliehou...@gmail.com) subscribed as well. I can use that email to 
receive flink emails.

Thank you!

Best,
Lijuan

Re: Unsubscribe

[liu ron]

Hi,

Please send an email to user-unsubscr...@flink.apache.org in order to be
removed from the User mailing list.

Best,
Ron

yunchao li  于2023年7月17日周一 09:14写道：

> Unsubscribe --
>
>>

Re: Unsubscribe

[liu ron]

Hi, wang

Please send an email to user-unsubscr...@flink.apache.org in order to be
removed from the User mailing list.

Best,
Ron


wang <24248...@163.com> 于2023年7月17日周一 23:10写道：

> Unsubscribe
>

Re: Unsubscribe

[liu ron]

Hi, William

Please send an email to user-unsubscr...@flink.apache.org in order to be
removed from the User mailing list.

Best,
Ron


William Wang  于2023年7月17日周一 14:59写道：

>
>

Re: Unsubscribe

[Martijn Visser]

Hi,

Please send an email to user-unsubscr...@flink.apache.org in order to be
removed from the User mailing list.

Best regards,

Martijn

On Wed, Jul 26, 2023 at 3:44 AM Lu Weizheng 
wrote:

> Unsubscribe
>

Unsubscribe

[Lu Weizheng]

Unsubscribe

Unsubscribe

[wang]

Unsubscribe

Unsubscribe

[William Wang]

Re: Unsubscribe

[yunchao li]

Unsubscribe --

>

Unsubscribe

[Boot]

Unsubscribe --

Unsubscribe

[wang]

Unsubscribe

Re: Unsubscribe

[liu ron]

Hi,

As Hang say, please send an email to user-unsubscr...@flink.apache.org if
you want to unsubscribe the mail from user@flink.apache.org,

Best,
Ron

Ragini Manjaiah  于2023年7月5日周三 13:36写道：

> Unsubscribe
>
> On Tue, Jul 4, 2023 at 1:33 PM Bauddhik Anand  wrote:
>
>> Unsubscribe
>>
>

Re: Unsubscribe

[Ragini Manjaiah]

Unsubscribe

On Tue, Jul 4, 2023 at 1:33 PM Bauddhik Anand  wrote:

> Unsubscribe
>

Re: Unsubscribe

[Hang Ruan]

Please send email to user-unsubscr...@flink.apache.org if you want to
unsubscribe the mail from user@flink.apache.org, and you can refer [1][2]
for more details.
请发送任意内容的邮件到 user-unsubscr...@flink.apache.org 地址来取消订阅来自
user@flink.apache.org 邮件组的邮件，你可以参考[1][2] 管理你的邮件订阅。

Best,
Hang

[1]
https://flink.apache.org/zh/community/#%e9%82%ae%e4%bb%b6%e5%88%97%e8%a1%a8
[2] https://flink.apache.org/community.html#mailing-lists

Ajinkya Pathrudkar  于2023年7月4日周二 20:22写道：

> Unsubscribe --
> Thanks & Regards,
> Ajinkya Pathrudkar
>

Re: Unsubscribe

[Hang Ruan]

Please send email to user-unsubscr...@flink.apache.org if you want to
unsubscribe the mail from user@flink.apache.org, and you can refer [1][2]
for more details.
请发送任意内容的邮件到 user-unsubscr...@flink.apache.org 地址来取消订阅来自
user@flink.apache.org 邮件组的邮件，你可以参考[1][2] 管理你的邮件订阅。

Best,
Hang

[1]
https://flink.apache.org/zh/community/#%e9%82%ae%e4%bb%b6%e5%88%97%e8%a1%a8
[2] https://flink.apache.org/community.html#mailing-lists

tan yao  于2023年7月4日周二 15:51写道：

> Unsubscribe
>
> 获取 Outlook for iOS <https://aka.ms/o0ukef>
>