I don't there's anything you can do except reducing the parallelism or
the size of your messages.
A separate serializer is used for each channel as the serializers are
stateful; they are capable of writing records partially
to a given MemorySegment to better utilize the allocated memory.
How many messages is each operator instance processing per second? I
would imagine that at this scale
your memory consumption goes through the roof anyway due to the message
size.
Even if every operator instance is only processing 10 records/s you're
already looking at 10TB memory usage
for in-flight data.
On 14.11.2017 11:11, yunfan123 wrote:
In the class org.apache.flink.runtime.io.network.api.writer.RecordWriter, it
has same number of serializers with the numChannels.
If I first operator has 500 parallels and the next operator has 1000
parallels.
And every message in flink is 2MB.
The job takes 500 * 1000 * 2MB as 1TB memory in totally!!!
Can I do anything to reduce the memory usage.
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
