Re: multiple frameworks or one big one
Not exactly the Enterprise oriented benchmark, but may give some insight though. https://mesos.apache.org/documentation/latest/powered-by-mesos/ https://www.youtube.com/watch?v=LQxnuPcRl4st=1m31s On Wed, Mar 4, 2015 at 2:18 AM, Diego Medina di...@fmpwizard.com wrote: Well, I deeply think that additionally to the architecture and the organisations concerns, Mesos need to provide some Enterprise oriented benchmark and proof to be able to really prime time on the enterprise world and not only on the Startup style enterprises, but it's not the topic of your post and I'll made my own regarding this specific topic. Looking forward to that discussion. Anyway, thank you very much for your answers. Regarding your choose of Golang instead of Scala because of some pain points, could you send me some exemples (except for compile time)? Even in private if you do not want to steal the thread, as I'm really balancing between those two. I'll send you a separate private message with the reply, I don't mind talking about it, but wouldn't want to distract this mailing list with the topic. Thanks Diego 2015-03-03 14:26 GMT+01:00 Diego Medina di...@fmpwizard.com: Hi Alex, On Tue, Mar 3, 2015 at 7:37 AM, Alex Rukletsov a...@mesosphere.io wrote: Next good big thing would be to handle task state updates. Instead of dying on TASK_LOST, you may want to retry this task several times. Yes, this is definitely something I need to address, for now I use it to help me find bugs in the code, if the app stops, I know I did something wrong :) I also need to find out why some tasks stay in status Staging on the Mesos UI, but I'll start a separate thread for it. Thanks Diego On Tue, Mar 3, 2015 at 10:38 AM, Billy Bones gael.ther...@gmail.com wrote: Oh and you've got a glitch on one of your executor name in your first code block. You've got: *extractorExe := mesos.ExecutorInfo{ ExecutorId: util.NewExecutorID(owl-cralwer-extractor), Name: proto.String(OwlCralwer Fetcher), Source: proto.String(owl-cralwer), Command: mesos.CommandInfo{ Value: proto.String(extractorExecutorCommand), Uris: executorUris, }, }* It should rather be: *extractorExe := mesos.ExecutorInfo{ ExecutorId: util.NewExecutorID(owl-cralwer-extractor), Name: proto.String(OwlCralwer Extractor), Source: proto.String(owl-cralwer), Command: mesos.CommandInfo{ Value: proto.String(extractorExecutorCommand), Uris: executorUris, }, }* 2015-03-03 10:28 GMT+01:00 Billy Bones gael.ther...@gmail.com: Hi Diego, did you already plan to make a benchmark of your result on the Mesos platform VS Bare-metal servers ? It would be really interesting for Enterprise evangelism, they love benchmark and metrics. I'm impress by your project and how it goes fast. I'm myself a fan of Golang, but why did you choose it? 2015-03-03 3:03 GMT+01:00 Diego Medina di...@fmpwizard.com: Hi everyone, based on all the great feedback I got here I updated the code and now I have one scheduler and two executors, one for fetching html and a second one that extracts links and text from the html. I also run the actual work on their own goroutines (like threads for tose not familiar with Go) and it's working great. I wrote about the changes here http://blog.fmpwizard.com/blog/owlcrawler-multiple-executors-using-meso and you can find the updated code here https://github.com/fmpwizard/owlcrawler Again, thanks everyone for your input. Diego On Fri, Feb 27, 2015 at 1:52 PM, Diego Medina di...@fmpwizard.com wrote: Thanks for looking at the code and the feedback Alex. I'll be working on those changes later tonight! Diego On Fri, Feb 27, 2015 at 12:15 PM, Alex Rukletsov a...@mesosphere.io wrote: Diego, I've checked your code, nice effort! Great to see people hacking with mesos and go bindings! One thing though. You do the actual job in the launchTask() of your executor. This prevents you from multiple tasks in parallel on one executor. That means you can't have more simultaneous tasks than executors in your cluster. You may want to spawn a thread for every incoming task and do the job there, while launchTasks() will do solely task initialization (basically, starting a thread). Check the project John referenced to: https://github.com/mesosphere/RENDLER. Best, Alex On Fri, Feb 27, 2015 at 11:03 AM, Diego Medina di...@fmpwizard.com wrote: Hi Billy, comments inline: On Fri, Feb 27, 2015 at 4:07 AM, Billy Bones gael.ther...@gmail.com wrote: Hi diego, as a real fan of the golang, I'm cudoes and clap for your work on this distributed crawler and hope you'll finally release it ;-) Thanks! my 3 month old baby is making sure I don't sleep much and have plenty of time to work on this project :) About your question, the common architecture is to have one scheduler and multiple
Re: multiple frameworks or one big one
Hi Diego, You're welcome ;-) Well, I deeply think that additionally to the architecture and the organisations concerns, Mesos need to provide some Enterprise oriented benchmark and proof to be able to really prime time on the enterprise world and not only on the Startup style enterprises, but it's not the topic of your post and I'll made my own regarding this specific topic. Anyway, thank you very much for your answers. Regarding your choose of Golang instead of Scala because of some pain points, could you send me some exemples (except for compile time)? Even in private if you do not want to steal the thread, as I'm really balancing between those two. 2015-03-03 14:26 GMT+01:00 Diego Medina di...@fmpwizard.com: Hi Alex, On Tue, Mar 3, 2015 at 7:37 AM, Alex Rukletsov a...@mesosphere.io wrote: Next good big thing would be to handle task state updates. Instead of dying on TASK_LOST, you may want to retry this task several times. Yes, this is definitely something I need to address, for now I use it to help me find bugs in the code, if the app stops, I know I did something wrong :) I also need to find out why some tasks stay in status Staging on the Mesos UI, but I'll start a separate thread for it. Thanks Diego On Tue, Mar 3, 2015 at 10:38 AM, Billy Bones gael.ther...@gmail.com wrote: Oh and you've got a glitch on one of your executor name in your first code block. You've got: *extractorExe := mesos.ExecutorInfo{ ExecutorId: util.NewExecutorID(owl-cralwer-extractor), Name: proto.String(OwlCralwer Fetcher), Source: proto.String(owl-cralwer), Command: mesos.CommandInfo{ Value: proto.String(extractorExecutorCommand), Uris: executorUris, }, }* It should rather be: *extractorExe := mesos.ExecutorInfo{ ExecutorId: util.NewExecutorID(owl-cralwer-extractor), Name: proto.String(OwlCralwer Extractor), Source: proto.String(owl-cralwer), Command: mesos.CommandInfo{ Value: proto.String(extractorExecutorCommand), Uris: executorUris, }, }* 2015-03-03 10:28 GMT+01:00 Billy Bones gael.ther...@gmail.com: Hi Diego, did you already plan to make a benchmark of your result on the Mesos platform VS Bare-metal servers ? It would be really interesting for Enterprise evangelism, they love benchmark and metrics. I'm impress by your project and how it goes fast. I'm myself a fan of Golang, but why did you choose it? 2015-03-03 3:03 GMT+01:00 Diego Medina di...@fmpwizard.com: Hi everyone, based on all the great feedback I got here I updated the code and now I have one scheduler and two executors, one for fetching html and a second one that extracts links and text from the html. I also run the actual work on their own goroutines (like threads for tose not familiar with Go) and it's working great. I wrote about the changes here http://blog.fmpwizard.com/blog/owlcrawler-multiple-executors-using-meso and you can find the updated code here https://github.com/fmpwizard/owlcrawler Again, thanks everyone for your input. Diego On Fri, Feb 27, 2015 at 1:52 PM, Diego Medina di...@fmpwizard.com wrote: Thanks for looking at the code and the feedback Alex. I'll be working on those changes later tonight! Diego On Fri, Feb 27, 2015 at 12:15 PM, Alex Rukletsov a...@mesosphere.io wrote: Diego, I've checked your code, nice effort! Great to see people hacking with mesos and go bindings! One thing though. You do the actual job in the launchTask() of your executor. This prevents you from multiple tasks in parallel on one executor. That means you can't have more simultaneous tasks than executors in your cluster. You may want to spawn a thread for every incoming task and do the job there, while launchTasks() will do solely task initialization (basically, starting a thread). Check the project John referenced to: https://github.com/mesosphere/RENDLER. Best, Alex On Fri, Feb 27, 2015 at 11:03 AM, Diego Medina di...@fmpwizard.com wrote: Hi Billy, comments inline: On Fri, Feb 27, 2015 at 4:07 AM, Billy Bones gael.ther...@gmail.com wrote: Hi diego, as a real fan of the golang, I'm cudoes and clap for your work on this distributed crawler and hope you'll finally release it ;-) Thanks! my 3 month old baby is making sure I don't sleep much and have plenty of time to work on this project :) About your question, the common architecture is to have one scheduler and multiple executors rather than one big executor. The basics of mesos is to take any resources, put them together on a pool to then swarm tasks on this pool, so, basically the architecture of your application should share this philosophy and then explode / decouple your application as much as possible but be carreful to not loop lock yourself on threads and tasks if they're dependents. I don't know if I'm explaining myself correctly so do not hesitate
Re: multiple frameworks or one big one
Well, I deeply think that additionally to the architecture and the organisations concerns, Mesos need to provide some Enterprise oriented benchmark and proof to be able to really prime time on the enterprise world and not only on the Startup style enterprises, but it's not the topic of your post and I'll made my own regarding this specific topic. Looking forward to that discussion. Anyway, thank you very much for your answers. Regarding your choose of Golang instead of Scala because of some pain points, could you send me some exemples (except for compile time)? Even in private if you do not want to steal the thread, as I'm really balancing between those two. I'll send you a separate private message with the reply, I don't mind talking about it, but wouldn't want to distract this mailing list with the topic. Thanks Diego 2015-03-03 14:26 GMT+01:00 Diego Medina di...@fmpwizard.com: Hi Alex, On Tue, Mar 3, 2015 at 7:37 AM, Alex Rukletsov a...@mesosphere.io wrote: Next good big thing would be to handle task state updates. Instead of dying on TASK_LOST, you may want to retry this task several times. Yes, this is definitely something I need to address, for now I use it to help me find bugs in the code, if the app stops, I know I did something wrong :) I also need to find out why some tasks stay in status Staging on the Mesos UI, but I'll start a separate thread for it. Thanks Diego On Tue, Mar 3, 2015 at 10:38 AM, Billy Bones gael.ther...@gmail.com wrote: Oh and you've got a glitch on one of your executor name in your first code block. You've got: *extractorExe := mesos.ExecutorInfo{ ExecutorId: util.NewExecutorID(owl-cralwer-extractor), Name: proto.String(OwlCralwer Fetcher), Source: proto.String(owl-cralwer), Command: mesos.CommandInfo{ Value: proto.String(extractorExecutorCommand), Uris: executorUris, }, }* It should rather be: *extractorExe := mesos.ExecutorInfo{ ExecutorId: util.NewExecutorID(owl-cralwer-extractor), Name: proto.String(OwlCralwer Extractor), Source: proto.String(owl-cralwer), Command: mesos.CommandInfo{ Value: proto.String(extractorExecutorCommand), Uris: executorUris, }, }* 2015-03-03 10:28 GMT+01:00 Billy Bones gael.ther...@gmail.com: Hi Diego, did you already plan to make a benchmark of your result on the Mesos platform VS Bare-metal servers ? It would be really interesting for Enterprise evangelism, they love benchmark and metrics. I'm impress by your project and how it goes fast. I'm myself a fan of Golang, but why did you choose it? 2015-03-03 3:03 GMT+01:00 Diego Medina di...@fmpwizard.com: Hi everyone, based on all the great feedback I got here I updated the code and now I have one scheduler and two executors, one for fetching html and a second one that extracts links and text from the html. I also run the actual work on their own goroutines (like threads for tose not familiar with Go) and it's working great. I wrote about the changes here http://blog.fmpwizard.com/blog/owlcrawler-multiple-executors-using-meso and you can find the updated code here https://github.com/fmpwizard/owlcrawler Again, thanks everyone for your input. Diego On Fri, Feb 27, 2015 at 1:52 PM, Diego Medina di...@fmpwizard.com wrote: Thanks for looking at the code and the feedback Alex. I'll be working on those changes later tonight! Diego On Fri, Feb 27, 2015 at 12:15 PM, Alex Rukletsov a...@mesosphere.io wrote: Diego, I've checked your code, nice effort! Great to see people hacking with mesos and go bindings! One thing though. You do the actual job in the launchTask() of your executor. This prevents you from multiple tasks in parallel on one executor. That means you can't have more simultaneous tasks than executors in your cluster. You may want to spawn a thread for every incoming task and do the job there, while launchTasks() will do solely task initialization (basically, starting a thread). Check the project John referenced to: https://github.com/mesosphere/RENDLER. Best, Alex On Fri, Feb 27, 2015 at 11:03 AM, Diego Medina di...@fmpwizard.com wrote: Hi Billy, comments inline: On Fri, Feb 27, 2015 at 4:07 AM, Billy Bones gael.ther...@gmail.com wrote: Hi diego, as a real fan of the golang, I'm cudoes and clap for your work on this distributed crawler and hope you'll finally release it ;-) Thanks! my 3 month old baby is making sure I don't sleep much and have plenty of time to work on this project :) About your question, the common architecture is to have one scheduler and multiple executors rather than one big executor. The basics of mesos is to take any resources, put them together on a pool to then swarm tasks on this pool, so, basically the architecture of your application should share this philosophy and then explode / decouple your
mesos c++ zookeeper blocks indefinately -- any plans to enhance?
hi mesos users and devs, We've observed that that the mesos 0.22.0-rc1 c++ zookeeper code appears to allow indefinite waits on responses. This leads to application hangs blocked inside mesos zookeeper calls. This can happen with a properly running zookeeper presumably able to make all responses. Heres how we hung it for eg. We issue a mesos zk set via int ZooKeeper::set ( const std::string path, const std::string data, int version ) then inside a Watcher we process on CHANGED event to issue a mesos zk get on the same path via int ZooKeeper::get ( const std::string path, boolwatch, std::string * result, Stat * stat ) we end up with two threads in the process both in pthread_cond_waits #0 0x00334e20b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x7f6664ee1cf5 in Gate::arrive (this=0x7f6140, old=0) at ../../../3rdparty/libprocess/src/gate.hpp:82 #2 0x7f6664ecef6e in process::ProcessManager::wait (this=0x7f02e0, pid=...) at ../../../3rdparty/libprocess/src/process.cpp:2476 #3 0x7f6664ed2ce9 in process::wait (pid=..., duration=...) at ../../../3rdparty/libprocess/src/process.cpp:2958 #4 0x7f6664e90558 in process::Latch::await (this=0x7f6ba0, duration=...) at ../../../3rdparty/libprocess/src/latch.cpp:49 #5 0x7f66649452cc in process::Futureint::await (this=0x7fffa0fd9040, duration=...) at ../../3rdparty/libprocess/include/process/future.hpp:1156 #6 0x7f666493a04d in process::Futureint::get (this=0x7fffa0fd9040) at ../../3rdparty/libprocess/include/process/future.hpp:1167 #7 0x7f6664ab1aac in ZooKeeper::set (this=0x803ce0, path=/craig/mo, data= ... and #0 0x00334e20b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x7f6664ee1cf5 in Gate::arrive (this=0x7f66380013f0, old=0) at ../../../3rdparty/libprocess/src/gate.hpp:82 #2 0x7f6664ecef6e in process::ProcessManager::wait (this=0x7f02e0, pid=...) at ../../../3rdparty/libprocess/src/process.cpp:2476 #3 0x7f6664ed2ce9 in process::wait (pid=..., duration=...) at ../../../3rdparty/libprocess/src/process.cpp:2958 #4 0x7f6664e90558 in process::Latch::await (this=0x7f6638000d00, duration=...) at ../../../3rdparty/libprocess/src/latch.cpp:49 #5 0x7f66649452cc in process::Futureint::await (this=0x7f66595fb6f0, duration=...) at ../../3rdparty/libprocess/include/process/future.hpp:1156 #6 0x7f666493a04d in process::Futureint::get (this=0x7f66595fb6f0) at ../../3rdparty/libprocess/include/process/future.hpp:1167 #7 0x7f6664ab18d3 in ZooKeeper::get (this=0x803ce0, path=/craig/mo, watch=false, So, really we are asking whether the mesos zk c++ api will be enhanced to not block indefinitely when results are beyond a time bound. cheers craig