Re: framework for short living tasks (1-15 seconds)

Try deeplearning4j with spark extensions .. On Mon, Mar 27, 2017 at 1:09 PM, daemeon reiydelle wrote: > Spark cached RDD's > > > *...* > > > > *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198 <(415)%20501-0198>London > (+44) (0) 20 8144 9872 <+44%2020%208144%209872>* > >

framework for short living tasks (1-15 seconds)

Hi, I want to run some image processing tasks on mesos cluster, but at the same time there will be a lot of tasks (1k-10k). What framework do you suggest to use in such case? Regards, Uladzimir

Re: framework for short living tasks (1-15 seconds)

Hi Radek, I thought about this, but in this case I have to scale number of containers manually, but I would like to leverage all the resources on mesos slaves.. In this way, all I need to increase throughput is to add slave nodes to the cluster and that's it. Regards, UladzimirOn Mar 27, 2017

Re: framework for short living tasks (1-15 seconds)

Makes sense. It might be difficult to store the offsets correctly in Kafka. Your framework would be the one consuming Kafka. I guess you could, in case of a failure, submit the message back to Kafka so you do not have to manage the offsets. There’s an interesting challenge there to find out when

Re: framework for short living tasks (1-15 seconds)

Sorry, I replied not to all the mailing list. Regards, UladzimirOn Mar 27, 2017 11:39 PM, vvshvv wrote:When user uploads some image to a REST endpoint, it should be processed in a separate Docker container. Actually, I can implement my custom mesos framework that will consume

Re: framework for short living tasks (1-15 seconds)

Well, you are storing the incoming events in Kafka. So data is not lost. That’s one. Instead of scheduling a container per image, start a number of containers, make each of them a member of consumer group and you’re sorted. By building a framework, you’re making your life really difficult. – Best

RE: framework for short living tasks (1-15 seconds)

Another approach (cli-based & short lived framework) for running multiple arbitrary one-off tasks… https://github.com/asamerh4/mesos-batch Cheers, Hubert From: vvshvv [mailto:vvs...@gmail.com] Sent: Montag, 27. März 2017 20:53 To: user@mesos.apache.org Subject: framework for short living tasks

MesosCon Asia CFP deadline extended

Chinese version below/ 中文详情请见下文 In order to allow for more submissions, we have extended the CFP deadline for MesosCon Asia until April 5th 11:59pm PDT. http://events.linuxfoundation.org/events/mesoscon-asia/program/cfp None of the other submission dates have changed. Community voting on

mesos container cluster came across health check coredump log

https://issues.apache.org/jira/browse/MESOS-6184 anyone give some hint? ``` I0328 11:48:12.922181 48 exec.cpp:162] Version: 1.2.0 I0328 11:48:12.929252 54 exec.cpp:237] Executor registered on agent a29dc3a5-3e3f-4058-8ab4-dd7de2ae58d1-S4 I0328 11:48:12.931640 54 docker.cpp:850] Running docker