Resource under-utilization when using RocksDb state backend

[Cliff Resnick]

In tests comparing RocksDb to fs state backend we observe much lower
throughput, around 10x slower. While the lowered throughput is expected,
what's perplexing is that machine load is also very low with RocksDb,
typically falling to  < 25% CPU and negligible IO wait (around 0.1%). Our
test instances are EC2 c3.xlarge which are 4 virtual CPUs and 7.5G RAM,
each running a single TaskManager in YARN, with 6.5G allocated memory per
TaskManager. The instances also have 2x40G attached SSDs which we have
mapped to `taskmanager.tmp.dir`.

With FS state and 4 slots per TM, we will easily max out with an average
load average around 5 or 6, so we actually need throttle down the slots to
3. With RocksDb using the Flink SSD configured options we see a load
average at around 1. Also, load (and actual) throughput remain more or less
constant no matter how many slots we use. The weak load is spread over all
CPUs.

Here is a sample top:

Cpu0  : 20.5%us,  0.0%sy,  0.0%ni, 79.5%id,  0.0%wa,  0.0%hi,  0.0%si,
 0.0%st
Cpu1  : 18.5%us,  0.0%sy,  0.0%ni, 81.5%id,  0.0%wa,  0.0%hi,  0.0%si,
 0.0%st
Cpu2  : 11.6%us,  0.7%sy,  0.0%ni, 87.0%id,  0.7%wa,  0.0%hi,  0.0%si,
 0.0%st
Cpu3  : 12.5%us,  0.3%sy,  0.0%ni, 86.8%id,  0.0%wa,  0.0%hi,  0.3%si,
 0.0%st

Our pipeline uses tumbling windows, each with a ValueState keyed to a
3-tuple of one string and two ints.. Each ValueState comprises a small set
of tuples around 5-7 fields each. The WindowFunction simply diffs agains
the set and updates state if there is a diff.

Any ideas as to what the bottleneck is here? Any suggestions welcomed!

-Cliff

Re: Thread 'SortMerger spilling thread' terminated due to an exception: No space left on device

[Miguel Coimbra]

Hello Fabian,

I have created a directory on my host machine user directory (
/home/myuser/mydir ) and I am mapping it as a volume with Docker for the
TaskManager and JobManager containers.
Each container will thus have the following directory /home/flink/htmp

host ---> container
/home/myuser/mydir ---> /home/flink/htmp

I had previously done this successfully with the a host directory which
holds several SNAP data sets.
In the Flink configuration file, I specified /home/flink/htmp to be used as
the tmp dir for the TaskManager.
This seems to be working, as I was able to start the cluster and invoke
Flink for that Friendster dataset.

However, during execution, there were 2 intermediate files which kept
growing until they reached about 30 GB.
At that point, the Flink TaskManager threw the exception again:

java.lang.RuntimeException: Error obtaining the sorted input: Thread
'SortMerger spilling thread' terminated due to an exception: No space left
on device

Here is an ls excerpt of the directory on the host (to which the
TaskManager container was also writing successfully) shortly before the
exception:

*31G *9d177a1971322263f1597c3378885ccf.channel
*31G* a693811249bc5f785a79d1b1b537fe93.channel

Now I believe the host system is capable of storing hundred GBs more, so I
am confused as to what the problem might be.

Best regards,

Miguel E. Coimbra
Email: miguel.e.coim...@gmail.com 
Skype: miguel.e.coimbra

​
>
> Hi Miguel,
>
> the exception does indeed indicate that the process ran out of available
> disk space.
> The quoted paragraph of the blog post describes the situation when you
> receive the IOE.
>
> By default the systems default tmp dir is used. I don't know which folder
> that would be in a Docker setup.
> You can configure the temp dir using the taskmanager.tmp.dirs config key.
> Please see the configuration documentation for details [1].
>
> Hope this helps,
> Fabian
>
> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.1/
> setup/config.html#jobmanager-amp-taskmanager
>
> 2016-12-02 0:18 GMT+01:00 Miguel Coimbra :
> ​
>
>> Hello,
>>
>> I have a problem for which I hope someone will be able to give a hint.
>> I am running the Flink *standalone* cluster with 2 Docker containers (1
>> TaskManager and 1 JobManager) using 1 TaskManager with 30 GB of RAM.
>>
>> The dataset is a large one: SNAP Friendster, which has around 1800 M
>> edges.
>> https://snap.stanford.edu/data/com-Friendster.html
>>
>> I am trying to run the Gelly built-in label propagation algorithm on top
>> of it.
>> As this is a very big dataset, I believe I am exceeding the available RAM
>> and that the system is using secondary storage, which then fails:
>>
>>
>> Connected to JobManager at Actor[akka.tcp://flink@172.19.
>> 0.2:6123/user/jobmanager#894624508]
>> 12/01/2016 17:58:24Job execution switched to status RUNNING.
>> 12/01/2016 17:58:24DataSource (at main(App.java:33) (
>> org.apache.flink.api.java.io.TupleCsvInputFormat))(1/1) switched to
>> SCHEDULED
>> 12/01/2016 17:58:24DataSource (at main(App.java:33) (
>> org.apache.flink.api.java.io.TupleCsvInputFormat))(1/1) switched to
>> DEPLOYING
>> 12/01/2016 17:58:24DataSource (at main(App.java:33) (
>> org.apache.flink.api.java.io.TupleCsvInputFormat))(1/1) switched to
>> RUNNING
>> 12/01/2016 17:58:24Map (Map at fromTuple2DataSet(Graph.java:343))(1/1)
>> switched to SCHEDULED
>> 12/01/2016 17:58:24Map (Map at fromTuple2DataSet(Graph.java:343))(1/1)
>> switched to DEPLOYING
>> 12/01/2016 17:58:24Map (Map at fromTuple2DataSet(Graph.java:343))(1/1)
>> switched to RUNNING
>> 12/01/2016 17:59:51Map (Map at fromTuple2DataSet(Graph.java:343))(1/1)
>> switched to FAILED
>> *java.lang.RuntimeException: Error obtaining the sorted input: Thread
>> 'SortMerger spilling thread' terminated due to an exception: No space left
>> on device*
>> at org.apache.flink.runtime.operators.sort.UnilateralSortMerger
>> .getIterator(UnilateralSortMerger.java:619)
>> at org.apache.flink.runtime.operators.BatchTask.getInput(BatchT
>> ask.java:1098)
>> at org.apache.flink.runtime.operators.MapDriver.run(MapDriver.j
>> ava:86)
>> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.j
>> ava:486)
>> at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTas
>> k.java:351)
>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:585)
>> at java.lang.Thread.run(Thread.java:745)
>> *Caused by: java.io.IOException: Thread 'SortMerger spilling thread'
>> terminated due to an exception: No space left on device*
>> at org.apache.flink.runtime.operators.sort.UnilateralSortMerger
>> $ThreadBase.run(UnilateralSortMerger.java:800)
>> Caused by: java.io.IOException: No space left on device
>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>> at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
>> at 

stackoverflow question

[Abdallah Ghdiri]

Hello

can someone take a look at my question aver at stackoverflow

.

or answer here.

Re: Query regarding state backend for Custom Map Function

[Stefan Richter]

Hi,

unfortunately, I think it is a little unlikely that it will still make it into 
1.2.

Best,
Stefan

> Am 01.12.2016 um 20:29 schrieb Anirudh Mallem :
> 
> Thanks a lot Stefan. I got what I was looking for. Is the MapState 
> functionality coming as a part of the 1.2 release? 
> 
> From: Stefan Richter
> Reply-To: "user@flink.apache.org "
> Date: Thursday, December 1, 2016 at 2:53 AM
> To: "user@flink.apache.org "
> Subject: Re: Query regarding state backend for Custom Map Function
> 
> Hi,
> 
> using the ValueState and RocksDB to store a map inside the value state means 
> that you will have a different map for each key, which is automatically 
> swapped on a per record basis, depending on the record’s key. If you are 
> using a map and Checkpointed, there is only one map and your code is 
> responsible for dispatching state between different keys.
> 
> If you use a map and Checkpointed, the map will be on the heap and the 
> checkpoint will go directly against the filesystem; this is independent of 
> the chosen backend, so no RocksDB is involved.
> 
> On a further note, we are working on an alternative to ValueState that is 
> like a MapState. In contrast to ValueState, MapState does not deserialize the 
> whole map on each access, but can access individual key/value pairs. This 
> might be what you are looking for.
> 
> Best,
> Stefan
> 
> 
>> Am 01.12.2016 um 09:35 schrieb Anirudh Mallem > >:
>> 
>> Hi Everyone,
>> I am trying to understand the Working With State feature page of the Flink 
>> documentation.
>>  My question is in case I am using a ValueState in my CustomMap class to 
>> store my states with the RocksDb as my state backend then it is clear that 
>> every state value is stored in RocksDb. 
>> Now instead of a ValueState if I just use a normal Java Hashmap to store my 
>> states and implement the Checkpointed interface then will the entire HashMap 
>> reside on the RocksDb backend or will the HashMap be in memory and just the 
>> snapshots sent to RocksDb? I am trying to see what will I lose/gain if I 
>> have my own data structure to do state maintenance. Thanks. 
>> 
>> Regards,
>> Anirudh 
> 

Re: Flink 1.1.3 OOME Permgen

[Robert Metzger]

Thank you for reporting the issue Konstantin.
I've filed a JIRA for the jackson issue:
https://issues.apache.org/jira/browse/FLINK-5233.
As I said in the JIRA, I propose to upgrade to Jackson 2.7.8, as this
version contains the fix for the issue, but its not a major jackson upgrade.

Any chance you could try to if 2.7.8 fixes the issue as well?


On Fri, Dec 2, 2016 at 11:12 AM, Fabian Hueske  wrote:

> Hi Konstantin,
>
> Regarding 2): I've opened FLINK-5227 to update the documentation [1].
>
> Regarding the Row type: The Row type was introduced for flink-table and
> was later used by other modules. There is FLINK-5186 to move Row and all
> the related TypeInfo (+serializer and comparator) to flink-core [2]. That
> should solve your issue.
>
> Some of the connector modules which provide TableSource and TableSinks
> have dependencies on flink-table as well. I'll check that these are
> optional dependencies to avoid that we pull in Calcite through connectors
> for jobs that do not not need it.
>
> Thanks,
> Fabian
>
> [1] https://issues.apache.org/jira/browse/FLINK-5227
> [2] https://issues.apache.org/jira/browse/FLINK-5186
>
> 2016-11-30 17:51 GMT+01:00 Konstantin Knauf 
> :
>
>> Hi Stefan,
>>
>> unfortunately, I can not share any heap dumps with you. I was able to
>> resolve some of the issues my self today, the root causes were different
>> for different jobs.
>>
>> 1) Jackson 2.7.2 (which comes with Flink) has a known class loading
>> issue (see https://github.com/FasterXML/jackson-databind/issues/1363).
>> Shipping a shaded version of Jackson 2.8.4 with our user code helped. I
>> recommend upgrading Flink's Jackson version soon.
>>
>> 2) We have a dependency on the flink-table [1] , which ships with
>> Calcite including the Calcite JDBC Driver, which can not been collected
>> cause of the known problem with the java.sql.DriverManager. Putting the
>> flink-table in Flink's lib dir instead of shipping it with the user code
>> helps. You should update the documentation, because this will always
>> happen when using flink-table, I think. So I wonder, why this has not
>> come up before actually.
>>
>> 3) Unresolved: Some Threads in a custom source which are not proberly
>> shut down and keep references to the UserCodeClassLoader. I did not have
>> time to look into this issue so far.
>>
>> Cheers,
>>
>> Konstantin
>>
>> [1] Side note: We only need flink-table for the "Row" class used in the
>> JdbcOutputFormat, so it might make sense to move this class somewhere
>> else. Naturally, we also tried to exclude the "transitive" dependency on
>> org.apache.calcite until we noticed that calcite is packaged with
>> flink-table, so that you can not even exclude it. What is the reasons
>> for this?
>>
>>
>>
>>
>> On 30.11.2016 00:55, Stefan Richter wrote:
>> > Hi,
>> >
>> > could you somehow provide us a heap dump from a TM that run for a while
>> (ideally, shortly before an OOME)? This would greatly help us to figure out
>> if there is a classloader leak that causes the problem.
>> >
>> > Best,
>> > Stefan
>> >
>> >> Am 29.11.2016 um 18:39 schrieb Konstantin Knauf <
>> konstantin.kn...@tngtech.com>:
>> >>
>> >> Hi everyone,
>> >>
>> >> since upgrading to Flink 1.1.3 we observe frequent OOME Permgen
>> Taskmanager Failures. Monitoring the permgen size on one of the
>> Taskamanagers you can see that each Job (New Job and Restarts) adds a few
>> MB, which can not be collected. Eventually, the OOME happens. This happens
>> with all our Jobs, Streaming and Batch, on Yarn 2.4 as well as Stand-Alone.
>> >>
>> >> On Flink 1.0.2 this was not a problem, but I will investigate it
>> further.
>> >>
>> >> The assumption is that Flink is somehow using one of the classes,
>> which comes with our jar and by that prevents the gc of the whole class
>> loader. Our Jars do not include any flink dependencies though
>> (compileOnly), but of course many others.
>> >>
>> >> Any ideas anyone?
>> >>
>> >> Cheers and thank you,
>> >>
>> >> Konstantin
>> >>
>> >> sent from my phone. Plz excuse brevity and tpyos.
>> >> ---
>> >> Konstantin Knauf *konstantin.kn...@tngtech.com * +49-174-3413182
>> >> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
>> >> Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
>> >
>> >
>>
>> --
>> Konstantin Knauf * konstantin.kn...@tngtech.com * +49-174-3413182
>> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
>> Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
>> Sitz: Unterföhring * Amtsgericht München * HRB 135082
>>
>>
>

Re: separation of JVMs for different applications

[Till Rohrmann]

Hi Manu,

with Flip-6 we will be able to support stricter application isolation by
starting for each job a dedicated JobManager which will execute its tasks
on TM reserved solely for this job. But at the same time we will continue
supporting the multi-tenant cluster mode where tasks belonging to multiple
jobs share the same set of TMs and, thus, might share information between
them.

Cheers,
Till

On Fri, Dec 2, 2016 at 11:19 AM, Fabian Hueske  wrote:

> Hi Manu,
>
> As far as I know, there are not plans to change the stand-alone deployment.
> FLIP-6 is focusing on deployments via resource providers (YARN, Mesos,
> etc.) which allow to start Flink processes per job.
>
> Till (in CC) is more familiar with the FLIP-6 effort and might be able to
> add more detail.
>
> Best,
> Fabian
>
> 2016-12-01 4:16 GMT+01:00 Manu Zhang :
>
>> Hi all,
>>
>> It seems tasks of different Flink applications can end up in the same JVM
>> (TaskManager) in standalone mode. Isn't this fragile since errors in one
>> application could crash another ? I checked FLIP-6
>>  
>> but
>> didn't found any mention of changing it in the future.
>>
>> Any thoughts or have I missed anything ?
>>
>> Thanks,
>> Manu Zhang
>>
>
>

Re: Dealing with Multiple sinks in Flink

[vinay patil]

Hi Robert,

I had resolved this issue earlier as I had not set the Kafka source
parallelism to number of partitions, so I was getting the issue of window
not getting triggered.

Now I am facing the same issue, I tried to check the watermark value by
using visualVM locally but I am not seeing that value there, I have attached
the snapahot of visualVM

 

Just to verify , JMX port runs on 9010 by default , right ?, because when I
tried to connect to it locally, I could not connect 



--
View this message in context: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Dealing-with-Multiple-sinks-in-Flink-tp8643p10412.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at 
Nabble.com.

Re: separation of JVMs for different applications

[Fabian Hueske]

Hi Manu,

As far as I know, there are not plans to change the stand-alone deployment.
FLIP-6 is focusing on deployments via resource providers (YARN, Mesos,
etc.) which allow to start Flink processes per job.

Till (in CC) is more familiar with the FLIP-6 effort and might be able to
add more detail.

Best,
Fabian

2016-12-01 4:16 GMT+01:00 Manu Zhang :

> Hi all,
>
> It seems tasks of different Flink applications can end up in the same JVM
> (TaskManager) in standalone mode. Isn't this fragile since errors in one
> application could crash another ? I checked FLIP-6
>  
> but
> didn't found any mention of changing it in the future.
>
> Any thoughts or have I missed anything ?
>
> Thanks,
> Manu Zhang
>

Re: Flink 1.1.3 OOME Permgen

[Fabian Hueske]

Hi Konstantin,

Regarding 2): I've opened FLINK-5227 to update the documentation [1].

Regarding the Row type: The Row type was introduced for flink-table and was
later used by other modules. There is FLINK-5186 to move Row and all the
related TypeInfo (+serializer and comparator) to flink-core [2]. That
should solve your issue.

Some of the connector modules which provide TableSource and TableSinks have
dependencies on flink-table as well. I'll check that these are optional
dependencies to avoid that we pull in Calcite through connectors for jobs
that do not not need it.

Thanks,
Fabian

[1] https://issues.apache.org/jira/browse/FLINK-5227
[2] https://issues.apache.org/jira/browse/FLINK-5186

2016-11-30 17:51 GMT+01:00 Konstantin Knauf :

> Hi Stefan,
>
> unfortunately, I can not share any heap dumps with you. I was able to
> resolve some of the issues my self today, the root causes were different
> for different jobs.
>
> 1) Jackson 2.7.2 (which comes with Flink) has a known class loading
> issue (see https://github.com/FasterXML/jackson-databind/issues/1363).
> Shipping a shaded version of Jackson 2.8.4 with our user code helped. I
> recommend upgrading Flink's Jackson version soon.
>
> 2) We have a dependency on the flink-table [1] , which ships with
> Calcite including the Calcite JDBC Driver, which can not been collected
> cause of the known problem with the java.sql.DriverManager. Putting the
> flink-table in Flink's lib dir instead of shipping it with the user code
> helps. You should update the documentation, because this will always
> happen when using flink-table, I think. So I wonder, why this has not
> come up before actually.
>
> 3) Unresolved: Some Threads in a custom source which are not proberly
> shut down and keep references to the UserCodeClassLoader. I did not have
> time to look into this issue so far.
>
> Cheers,
>
> Konstantin
>
> [1] Side note: We only need flink-table for the "Row" class used in the
> JdbcOutputFormat, so it might make sense to move this class somewhere
> else. Naturally, we also tried to exclude the "transitive" dependency on
> org.apache.calcite until we noticed that calcite is packaged with
> flink-table, so that you can not even exclude it. What is the reasons
> for this?
>
>
>
>
> On 30.11.2016 00:55, Stefan Richter wrote:
> > Hi,
> >
> > could you somehow provide us a heap dump from a TM that run for a while
> (ideally, shortly before an OOME)? This would greatly help us to figure out
> if there is a classloader leak that causes the problem.
> >
> > Best,
> > Stefan
> >
> >> Am 29.11.2016 um 18:39 schrieb Konstantin Knauf <
> konstantin.kn...@tngtech.com>:
> >>
> >> Hi everyone,
> >>
> >> since upgrading to Flink 1.1.3 we observe frequent OOME Permgen
> Taskmanager Failures. Monitoring the permgen size on one of the
> Taskamanagers you can see that each Job (New Job and Restarts) adds a few
> MB, which can not be collected. Eventually, the OOME happens. This happens
> with all our Jobs, Streaming and Batch, on Yarn 2.4 as well as Stand-Alone.
> >>
> >> On Flink 1.0.2 this was not a problem, but I will investigate it
> further.
> >>
> >> The assumption is that Flink is somehow using one of the classes, which
> comes with our jar and by that prevents the gc of the whole class loader.
> Our Jars do not include any flink dependencies though (compileOnly), but of
> course many others.
> >>
> >> Any ideas anyone?
> >>
> >> Cheers and thank you,
> >>
> >> Konstantin
> >>
> >> sent from my phone. Plz excuse brevity and tpyos.
> >> ---
> >> Konstantin Knauf *konstantin.kn...@tngtech.com * +49-174-3413182
> >> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
> >> Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
> >
> >
>
> --
> Konstantin Knauf * konstantin.kn...@tngtech.com * +49-174-3413182
> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
> Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
> Sitz: Unterföhring * Amtsgericht München * HRB 135082
>
>

Re: Thread 'SortMerger spilling thread' terminated due to an exception: No space left on device

[Fabian Hueske]

Hi Miguel,

the exception does indeed indicate that the process ran out of available
disk space.
The quoted paragraph of the blog post describes the situation when you
receive the IOE.

By default the systems default tmp dir is used. I don't know which folder
that would be in a Docker setup.
You can configure the temp dir using the taskmanager.tmp.dirs config key.
Please see the configuration documentation for details [1].

Hope this helps,
Fabian

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.1/setup/config.html#jobmanager-amp-taskmanager

2016-12-02 0:18 GMT+01:00 Miguel Coimbra :

> Hello,
>
> I have a problem for which I hope someone will be able to give a hint.
> I am running the Flink *standalone* cluster with 2 Docker containers (1
> TaskManager and 1 JobManager) using 1 TaskManager with 30 GB of RAM.
>
> The dataset is a large one: SNAP Friendster, which has around 1800 M edges.
> https://snap.stanford.edu/data/com-Friendster.html
>
> I am trying to run the Gelly built-in label propagation algorithm on top
> of it.
> As this is a very big dataset, I believe I am exceeding the available RAM
> and that the system is using secondary storage, which then fails:
>
>
> Connected to JobManager at Actor[akka.tcp://flink@172.19.
> 0.2:6123/user/jobmanager#894624508]
> 12/01/2016 17:58:24Job execution switched to status RUNNING.
> 12/01/2016 17:58:24DataSource (at main(App.java:33) (
> org.apache.flink.api.java.io.TupleCsvInputFormat))(1/1) switched to
> SCHEDULED
> 12/01/2016 17:58:24DataSource (at main(App.java:33) (
> org.apache.flink.api.java.io.TupleCsvInputFormat))(1/1) switched to
> DEPLOYING
> 12/01/2016 17:58:24DataSource (at main(App.java:33) (
> org.apache.flink.api.java.io.TupleCsvInputFormat))(1/1) switched to
> RUNNING
> 12/01/2016 17:58:24Map (Map at fromTuple2DataSet(Graph.java:343))(1/1)
> switched to SCHEDULED
> 12/01/2016 17:58:24Map (Map at fromTuple2DataSet(Graph.java:343))(1/1)
> switched to DEPLOYING
> 12/01/2016 17:58:24Map (Map at fromTuple2DataSet(Graph.java:343))(1/1)
> switched to RUNNING
> 12/01/2016 17:59:51Map (Map at fromTuple2DataSet(Graph.java:343))(1/1)
> switched to FAILED
> *java.lang.RuntimeException: Error obtaining the sorted input: Thread
> 'SortMerger spilling thread' terminated due to an exception: No space left
> on device*
> at org.apache.flink.runtime.operators.sort.UnilateralSortMerger.
> getIterator(UnilateralSortMerger.java:619)
> at org.apache.flink.runtime.operators.BatchTask.getInput(
> BatchTask.java:1098)
> at org.apache.flink.runtime.operators.MapDriver.run(MapDriver.java:86)
> at org.apache.flink.runtime.operators.BatchTask.run(
> BatchTask.java:486)
> at org.apache.flink.runtime.operators.BatchTask.invoke(
> BatchTask.java:351)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:585)
> at java.lang.Thread.run(Thread.java:745)
> *Caused by: java.io.IOException: Thread 'SortMerger spilling thread'
> terminated due to an exception: No space left on device*
> at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$
> ThreadBase.run(UnilateralSortMerger.java:800)
> Caused by: java.io.IOException: No space left on device
> at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
> at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
> at sun.nio.ch.IOUtil.write(IOUtil.java:65)
> at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)
> at org.apache.flink.runtime.io.disk.iomanager.
> SegmentWriteRequest.write(AsynchronousFileIOChannel.java:344)
> at org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync$
> WriterThread.run(IOManagerAsync.java:502)
>
>
> I do not have secondary storage limitations on the host system, so I
> believe the system would be able to handle whatever is spilled to the
> disk...
> Perhaps this is a Docker limitation regarding the usage of the host's
> secondary storage?
>
> Or is there perhaps some configuration or setting for the TaskManager
> which I am missing?
> Running the label propagation of Gelly on this dataset and cluster
> configuration, what would be the expected behavior if the system consumes
> all the memory?
>
>
> I believe the SortMerger thread is associated to the following mechanism
> described in this blog post:
>
> https://flink.apache.org/news/2015/03/13/peeking-into-
> Apache-Flinks-Engine-Room.html
> *The Sort-Merge-Join works by first sorting both input data sets on their
> join key attributes (Sort Phase) and merging the sorted data sets as a
> second step (Merge Phase). The sort is done in-memory if the local
> partition of a data set is small enough. Otherwise, an external merge-sort
> is done by collecting data until the working memory is filled, sorting it,
> writing the sorted data to the local filesystem, and starting over by
> filling the working memory again with more