FLIP-6 and running many "small" jobs

[Maciek Próchniak]

Hi,

we're looking at FLIP-6 and while it looks really great we started to 
wonder how it fits in our use case.

We currently have around 20 processes but the idea is to have many more 
of them. Many of them are pretty "small" - them don't large sources, are 
stateless, mainly filtering data.

As I understand, FLIP-6 makes job even more heavyweight thing than today 
- e.g. each job will have it's own jobmanager process etc.

Our concern is that each job will now require more resources - e.g. the 
number of threads, memory and so on. We are thinking about a way to make 
some jobs share these resources - of course that mean they won't be 
really isolated from each other.

So far the only idea we see is deploying these small jobs together, as 
one job - but this leads to some problems, like how to track which 
version is really deployed (we talk about stateless processes so the 
only problem is maintaining source kafka offsets)

Unfortunatelly our jobs can have many different sources and outcomes, so 
we don't think doing sth similar to King&RBEA would work for us...

Do you have any views/ideas about such use case? Or is common view that 
we should deploy our stuff to mesos and let it handle resource 
allocation? But still - for some jobs we'd need sth like "1/4" slot :)

thanks,

maciek