Hi,
JobManager is not responsible and have no means to restart TaskManager in case
of TaskManager process being killed (it would need to have ssh into the machine
and restart it…). I don’t know, but from your description of the problem I
presume that Flink’s bash startup scripts do not contain
Hi,
I’m not aware of such rules of thumb. Memory consumption is highly application
and workload specific. It depends on how much things you allocate in your user
code and how much memory do you keep on state (in case of heap state backend).
Basically just as with most java applications, you
Thank you for your help Piotrek.
I think it was a combination of a. other processes taking up available
memory and b. flink processes consuming all the memory allocated to them,
that resulted in kernel running out of memory.
Are there any heuristics or best practices which you (or anyone in the
Hi,
Good that we are more or less on track with this problem :) But the problem
here is not that heap size is too small, bot that your kernel is running out of
memory and starts killing processes. Either:
1. some other process is using the available memory
2. Increase memory allocation on
Hi Piotrek,
Thanks for your reply. I checked through the syslogs for that time, and I
see this:
Aug 8 13:20:52 smoketest kernel: [1786160.856662] Out of memory: Kill
process 2305 (java) score 468 or sacrifice child
Aug 8 13:20:52 smoketest kernel: [1786160.859091] Killed process 2305
(java)
Hi,
Please post full TaskManager logs, including stderr and stdout. (Have you
checked the stderr/stdout for some messages?)
I could think of couple reasons:
1. process segfault
2. process killed by OS
3. OS failure
1. Should be visible by some message in stderr/stdout file and can be caused by
Hi,
I hit a similar issue yesterday, the task manager died suspiciously, no
error logs in the task manager logs, but I see the following exceptions in
the job manager logs:
2018-08-05 18:03:28,322 ERROR
akka.remote.Remoting - Association
to
Hi Piotr,
I didn't find anything special in the logs before the failure.
Here are the logs, please take a look:
https://drive.google.com/drive/folders/1zlUDMpbO9xZjjJzf28lUX-bkn_x7QV59?usp=sharing
The configuration is:
3 task managers:
qafdsflinkw011.scl
qafdsflinkw012.scl
qafdsflinkw013.scl -
