Re: multiple frameworks or one big one
Hi Diego,
You're welcome ;-)
Well, I deeply think that additionally to the architecture and the
organisations concerns, Mesos need to provide some Enterprise oriented
benchmark and "proof" to be able to really prime time on the enterprise
world and not only on the "Startup style enterprises", but it's not the
topic of your post and I'll made my own regarding this specific topic.
Anyway, thank you very much for your answers.
Regarding your choose of Golang instead of Scala because of some pain
points, could you send me some exemples (except for compile time)? Even in
private if you do not want to steal the thread, as I'm really balancing
between those two.
2015-03-03 14:26 GMT+01:00 Diego Medina :
> Hi Alex,
>
> On Tue, Mar 3, 2015 at 7:37 AM, Alex Rukletsov wrote:
>
>> Next good big thing would be to handle task state updates. Instead of
>> dying on TASK_LOST, you may want to retry this task several times.
>>
>
> Yes, this is definitely something I need to address, for now I use it to
> help me find bugs in the code, if the app stops, I know I did something
> wrong :)
> I also need to find out why some tasks stay in status "Staging" on the
> Mesos UI, but I'll start a separate thread for it.
>
> Thanks
>
> Diego
>
>
>
>
>>
>> On Tue, Mar 3, 2015 at 10:38 AM, Billy Bones
>> wrote:
>>
>>> Oh and you've got a glitch on one of your executor name in your first
>>> code block.
>>>
>>> You've got:
>>>
>>> *extractorExe := &mesos.ExecutorInfo{
>>> ExecutorId: util.NewExecutorID("owl-cralwer-extractor"),
>>> Name: proto.String("OwlCralwer Fetcher"),
>>> Source: proto.String("owl-cralwer"),
>>> Command: &mesos.CommandInfo{
>>> Value: proto.String(extractorExecutorCommand),
>>> Uris: executorUris,
>>> },
>>> }*
>>>
>>> It should rather be:
>>>
>>> *extractorExe := &mesos.ExecutorInfo{
>>> ExecutorId: util.NewExecutorID("owl-cralwer-extractor"),
>>> Name: proto.String("OwlCralwer Extractor"),
>>> Source: proto.String("owl-cralwer"),
>>> Command: &mesos.CommandInfo{
>>> Value: proto.String(extractorExecutorCommand),
>>> Uris: executorUris,
>>> },
>>> }*
>>>
>>>
>>>
>>> 2015-03-03 10:28 GMT+01:00 Billy Bones :
>>>
Hi Diego, did you already plan to make a benchmark of your result on
the Mesos platform VS Bare-metal servers ?
It would be really interesting for Enterprise evangelism, they love
benchmark and metrics.
I'm impress by your project and how it goes fast. I'm myself a fan of
Golang, but why did you choose it?
2015-03-03 3:03 GMT+01:00 Diego Medina :
> Hi everyone, based on all the great feedback I got here I updated the
> code and now I have one scheduler and two executors, one for fetching html
> and a second one that extracts links and text from the html.
> I also run the actual work on their own goroutines (like threads for
> tose not familiar with Go) and it's working great.
>
> I wrote about the changes here
> http://blog.fmpwizard.com/blog/owlcrawler-multiple-executors-using-meso
> and you can find the updated code here
> https://github.com/fmpwizard/owlcrawler
>
> Again, thanks everyone for your input.
>
> Diego
>
>
>
>
> On Fri, Feb 27, 2015 at 1:52 PM, Diego Medina
> wrote:
>
>> Thanks for looking at the code and the feedback Alex. I'll be working
>> on those changes later tonight!
>>
>> Diego
>>
>> On Fri, Feb 27, 2015 at 12:15 PM, Alex Rukletsov
>> wrote:
>>
>>> Diego,
>>>
>>> I've checked your code, nice effort! Great to see people hacking
>>> with mesos and go bindings!
>>>
>>> One thing though. You do the actual job in the launchTask() of your
>>> executor. This prevents you from multiple tasks in parallel on one
>>> executor. That means you can't have more simultaneous tasks than
>>> executors
>>> in your cluster. You may want to spawn a thread for every incoming task
>>> and
>>> do the job there, while launchTasks() will do solely task initialization
>>> (basically, starting a thread). Check the project John referenced to:
>>> https://github.com/mesosphere/RENDLER.
>>>
>>> Best,
>>> Alex
>>>
>>> On Fri, Feb 27, 2015 at 11:03 AM, Diego Medina
>>> wrote:
>>>
Hi Billy,
comments inline:
On Fri, Feb 27, 2015 at 4:07 AM, Billy Bones <
gael.ther...@gmail.com> wrote:
> Hi diego, as a real fan of the golang, I'm cudoes and clap for
> your work on this distributed crawler and hope you'll finally release
> it ;-)
>
>
Thanks! my 3 month old baby is making sure I don't sleep much and
have plenty of time to work on this project :)
> About your question, the common architecture is to have one
Re: multiple frameworks or one big one
> Well, I deeply think that additionally to the architecture and the
> organisations concerns, Mesos need to provide some Enterprise oriented
> benchmark and "proof" to be able to really prime time on the enterprise
> world and not only on the "Startup style enterprises", but it's not the
> topic of your post and I'll made my own regarding this specific topic.
>
>
Looking forward to that discussion.
> Anyway, thank you very much for your answers.
>
> Regarding your choose of Golang instead of Scala because of some pain
> points, could you send me some exemples (except for compile time)? Even in
> private if you do not want to steal the thread, as I'm really balancing
> between those two.
>
>
I'll send you a separate private message with the reply, I don't mind
talking about it, but wouldn't want to distract this mailing list with the
topic.
Thanks
Diego
> 2015-03-03 14:26 GMT+01:00 Diego Medina :
>
>> Hi Alex,
>>
>> On Tue, Mar 3, 2015 at 7:37 AM, Alex Rukletsov
>> wrote:
>>
>>> Next good big thing would be to handle task state updates. Instead of
>>> dying on TASK_LOST, you may want to retry this task several times.
>>>
>>
>> Yes, this is definitely something I need to address, for now I use it to
>> help me find bugs in the code, if the app stops, I know I did something
>> wrong :)
>> I also need to find out why some tasks stay in status "Staging" on the
>> Mesos UI, but I'll start a separate thread for it.
>>
>> Thanks
>>
>> Diego
>>
>>
>>
>>
>>>
>>> On Tue, Mar 3, 2015 at 10:38 AM, Billy Bones
>>> wrote:
>>>
Oh and you've got a glitch on one of your executor name in your first
code block.
You've got:
*extractorExe := &mesos.ExecutorInfo{
ExecutorId: util.NewExecutorID("owl-cralwer-extractor"),
Name: proto.String("OwlCralwer Fetcher"),
Source: proto.String("owl-cralwer"),
Command: &mesos.CommandInfo{
Value: proto.String(extractorExecutorCommand),
Uris: executorUris,
},
}*
It should rather be:
*extractorExe := &mesos.ExecutorInfo{
ExecutorId: util.NewExecutorID("owl-cralwer-extractor"),
Name: proto.String("OwlCralwer Extractor"),
Source: proto.String("owl-cralwer"),
Command: &mesos.CommandInfo{
Value: proto.String(extractorExecutorCommand),
Uris: executorUris,
},
}*
2015-03-03 10:28 GMT+01:00 Billy Bones :
> Hi Diego, did you already plan to make a benchmark of your result on
> the Mesos platform VS Bare-metal servers ?
> It would be really interesting for Enterprise evangelism, they love
> benchmark and metrics.
>
> I'm impress by your project and how it goes fast. I'm myself a fan of
> Golang, but why did you choose it?
>
> 2015-03-03 3:03 GMT+01:00 Diego Medina :
>
>> Hi everyone, based on all the great feedback I got here I updated the
>> code and now I have one scheduler and two executors, one for fetching
>> html
>> and a second one that extracts links and text from the html.
>> I also run the actual work on their own goroutines (like threads for
>> tose not familiar with Go) and it's working great.
>>
>> I wrote about the changes here
>>
>> http://blog.fmpwizard.com/blog/owlcrawler-multiple-executors-using-meso
>> and you can find the updated code here
>> https://github.com/fmpwizard/owlcrawler
>>
>> Again, thanks everyone for your input.
>>
>> Diego
>>
>>
>>
>>
>> On Fri, Feb 27, 2015 at 1:52 PM, Diego Medina
>> wrote:
>>
>>> Thanks for looking at the code and the feedback Alex. I'll be
>>> working on those changes later tonight!
>>>
>>> Diego
>>>
>>> On Fri, Feb 27, 2015 at 12:15 PM, Alex Rukletsov >> > wrote:
>>>
Diego,
I've checked your code, nice effort! Great to see people hacking
with mesos and go bindings!
One thing though. You do the actual job in the launchTask() of your
executor. This prevents you from multiple tasks in parallel on one
executor. That means you can't have more simultaneous tasks than
executors
in your cluster. You may want to spawn a thread for every incoming
task and
do the job there, while launchTasks() will do solely task
initialization
(basically, starting a thread). Check the project John referenced to:
https://github.com/mesosphere/RENDLER.
Best,
Alex
On Fri, Feb 27, 2015 at 11:03 AM, Diego Medina >>> > wrote:
> Hi Billy,
>
> comments inline:
>
> On Fri, Feb 27, 2015 at 4:07 AM, Billy Bones <
> gael.ther...@gmail.com> wrote:
>
>> Hi diego, as a real fan of the golang, I'm cudoes and clap fo
mesos c++ zookeeper blocks indefinately -- any plans to enhance?
hi mesos users and devs, We've observed that that the mesos 0.22.0-rc1 c++ zookeeper code appears to allow indefinite waits on responses. This leads to application hangs blocked inside mesos zookeeper calls. This can happen with a properly running zookeeper presumably able to make all responses. Heres how we hung it for eg. We issue a mesos zk set via int ZooKeeper::set ( const std::string & path, const std::string & data, int version ) then inside a Watcher we process on CHANGED event to issue a mesos zk get on the same path via int ZooKeeper::get ( const std::string & path, boolwatch, std::string * result, Stat * stat ) we end up with two threads in the process both in pthread_cond_waits #0 0x00334e20b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x7f6664ee1cf5 in Gate::arrive (this=0x7f6140, old=0) at ../../../3rdparty/libprocess/src/gate.hpp:82 #2 0x7f6664ecef6e in process::ProcessManager::wait (this=0x7f02e0, pid=...) at ../../../3rdparty/libprocess/src/process.cpp:2476 #3 0x7f6664ed2ce9 in process::wait (pid=..., duration=...) at ../../../3rdparty/libprocess/src/process.cpp:2958 #4 0x7f6664e90558 in process::Latch::await (this=0x7f6ba0, duration=...) at ../../../3rdparty/libprocess/src/latch.cpp:49 #5 0x7f66649452cc in process::Future::await (this=0x7fffa0fd9040, duration=...) at ../../3rdparty/libprocess/include/process/future.hpp:1156 #6 0x7f666493a04d in process::Future::get (this=0x7fffa0fd9040) at ../../3rdparty/libprocess/include/process/future.hpp:1167 #7 0x7f6664ab1aac in ZooKeeper::set (this=0x803ce0, path="/craig/mo", data= ... and #0 0x00334e20b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x7f6664ee1cf5 in Gate::arrive (this=0x7f66380013f0, old=0) at ../../../3rdparty/libprocess/src/gate.hpp:82 #2 0x7f6664ecef6e in process::ProcessManager::wait (this=0x7f02e0, pid=...) at ../../../3rdparty/libprocess/src/process.cpp:2476 #3 0x7f6664ed2ce9 in process::wait (pid=..., duration=...) at ../../../3rdparty/libprocess/src/process.cpp:2958 #4 0x7f6664e90558 in process::Latch::await (this=0x7f6638000d00, duration=...) at ../../../3rdparty/libprocess/src/latch.cpp:49 #5 0x7f66649452cc in process::Future::await (this=0x7f66595fb6f0, duration=...) at ../../3rdparty/libprocess/include/process/future.hpp:1156 #6 0x7f666493a04d in process::Future::get (this=0x7f66595fb6f0) at ../../3rdparty/libprocess/include/process/future.hpp:1167 #7 0x7f6664ab18d3 in ZooKeeper::get (this=0x803ce0, path="/craig/mo", watch=false, So, really we are asking whether the mesos zk c++ api will be enhanced to not block indefinitely when results are beyond a time bound. cheers craig
Re: multiple frameworks or one big one
Not exactly the Enterprise oriented benchmark, but may give some insight
though.
https://mesos.apache.org/documentation/latest/powered-by-mesos/
https://www.youtube.com/watch?v=LQxnuPcRl4s&t=1m31s
On Wed, Mar 4, 2015 at 2:18 AM, Diego Medina wrote:
>
> Well, I deeply think that additionally to the architecture and the
>> organisations concerns, Mesos need to provide some Enterprise oriented
>> benchmark and "proof" to be able to really prime time on the enterprise
>> world and not only on the "Startup style enterprises", but it's not the
>> topic of your post and I'll made my own regarding this specific topic.
>>
>>
> Looking forward to that discussion.
>
>
>
>> Anyway, thank you very much for your answers.
>>
>> Regarding your choose of Golang instead of Scala because of some pain
>> points, could you send me some exemples (except for compile time)? Even in
>> private if you do not want to steal the thread, as I'm really balancing
>> between those two.
>>
>>
>
> I'll send you a separate private message with the reply, I don't mind
> talking about it, but wouldn't want to distract this mailing list with the
> topic.
>
> Thanks
>
> Diego
>
>
>
>> 2015-03-03 14:26 GMT+01:00 Diego Medina :
>>
>>> Hi Alex,
>>>
>>> On Tue, Mar 3, 2015 at 7:37 AM, Alex Rukletsov
>>> wrote:
>>>
Next good big thing would be to handle task state updates. Instead of
dying on TASK_LOST, you may want to retry this task several times.
>>>
>>> Yes, this is definitely something I need to address, for now I use it to
>>> help me find bugs in the code, if the app stops, I know I did something
>>> wrong :)
>>> I also need to find out why some tasks stay in status "Staging" on the
>>> Mesos UI, but I'll start a separate thread for it.
>>>
>>> Thanks
>>>
>>> Diego
>>>
>>>
>>>
>>>
On Tue, Mar 3, 2015 at 10:38 AM, Billy Bones
wrote:
> Oh and you've got a glitch on one of your executor name in your first
> code block.
>
> You've got:
>
> *extractorExe := &mesos.ExecutorInfo{
> ExecutorId: util.NewExecutorID("owl-cralwer-extractor"),
> Name: proto.String("OwlCralwer Fetcher"),
> Source: proto.String("owl-cralwer"),
> Command: &mesos.CommandInfo{
> Value: proto.String(extractorExecutorCommand),
> Uris: executorUris,
> },
> }*
>
> It should rather be:
>
> *extractorExe := &mesos.ExecutorInfo{
> ExecutorId: util.NewExecutorID("owl-cralwer-extractor"),
> Name: proto.String("OwlCralwer Extractor"),
> Source: proto.String("owl-cralwer"),
> Command: &mesos.CommandInfo{
> Value: proto.String(extractorExecutorCommand),
> Uris: executorUris,
> },
> }*
>
>
>
> 2015-03-03 10:28 GMT+01:00 Billy Bones :
>
>> Hi Diego, did you already plan to make a benchmark of your result on
>> the Mesos platform VS Bare-metal servers ?
>> It would be really interesting for Enterprise evangelism, they love
>> benchmark and metrics.
>>
>> I'm impress by your project and how it goes fast. I'm myself a fan of
>> Golang, but why did you choose it?
>>
>> 2015-03-03 3:03 GMT+01:00 Diego Medina :
>>
>>> Hi everyone, based on all the great feedback I got here I updated
>>> the code and now I have one scheduler and two executors, one for
>>> fetching
>>> html and a second one that extracts links and text from the html.
>>> I also run the actual work on their own goroutines (like threads for
>>> tose not familiar with Go) and it's working great.
>>>
>>> I wrote about the changes here
>>>
>>> http://blog.fmpwizard.com/blog/owlcrawler-multiple-executors-using-meso
>>> and you can find the updated code here
>>> https://github.com/fmpwizard/owlcrawler
>>>
>>> Again, thanks everyone for your input.
>>>
>>> Diego
>>>
>>>
>>>
>>>
>>> On Fri, Feb 27, 2015 at 1:52 PM, Diego Medina
>>> wrote:
>>>
Thanks for looking at the code and the feedback Alex. I'll be
working on those changes later tonight!
Diego
On Fri, Feb 27, 2015 at 12:15 PM, Alex Rukletsov <
a...@mesosphere.io> wrote:
> Diego,
>
> I've checked your code, nice effort! Great to see people hacking
> with mesos and go bindings!
>
> One thing though. You do the actual job in the launchTask() of
> your executor. This prevents you from multiple tasks in parallel on
> one
> executor. That means you can't have more simultaneous tasks than
> executors
> in your cluster. You may want to spawn a thread for every incoming
> task and
> do the job there, while launchTasks() will do solely task
> initialization
> (basically, starting a thread). Check the project John referenced to:
>>>
Re: mesos c++ zookeeper blocks indefinately -- any plans to enhance?
hi again mesos users and devs, In the prior post i left with description of hanging program with mesos zookeeper c++ api and wondered about enhancement to not wait indefinitely when underlying zookeeper responses dont occur. At that time i thought perhaps the underlying zookeeper and/or its C binding might not be responding up to the mesos api callers. So, while the question is still outstanding, I now see that potentially the hanging issue is with the mesos implementation over zookeeper c binding. In particular i've now tried a similar scenario just with zookeeper c binding api. That is, do zk aget/complete from within a watcher for events for the CHANGED event from a prior aset/complete. i dont see any blocking indefinitely and both the aget and aset completions are invoked and finish. Unless i'm not reproducing this properly, what i determine is a bad behavior from the mesos c++ api. Somehow the mesos c++ zookeeper api implementation is getting itself into pthread condition waits with nothing to notify and break the waits. this seems to occur with get calls from a Watcher on CHANGED events. craig Original Message From: pink...@safe-mail.net Apparently from: user-return-2761-pinktie=safe-mail@mesos.apache.org To: user@mesos.apache.org Subject: mesos c++ zookeeper blocks indefinately -- any plans to enhance? Date: Wed, 4 Mar 2015 10:05:54 -0500 > hi mesos users and devs, > We've observed that that the mesos 0.22.0-rc1 c++ zookeeper code appears to > allow indefinite waits on responses. > This leads to application hangs blocked inside mesos zookeeper calls. > This can happen with a properly running zookeeper presumably able to make all > responses. > > Heres how we hung it for eg. > We issue a mesos zk set via > > int ZooKeeper::set( const std::string & path, > const std::string & data, > int version > ) > > then inside a Watcher we process on CHANGED event to issue a mesos zk get on > the same path via > > int ZooKeeper::get( const std::string & path, > bool watch, > std::string * result, > Stat *stat > ) > > we end up with two threads in the process both in pthread_cond_waits > #0 0x00334e20b43c in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x7f6664ee1cf5 in Gate::arrive (this=0x7f6140, old=0) > at ../../../3rdparty/libprocess/src/gate.hpp:82 > #2 0x7f6664ecef6e in process::ProcessManager::wait (this=0x7f02e0, > pid=...) > at ../../../3rdparty/libprocess/src/process.cpp:2476 > #3 0x7f6664ed2ce9 in process::wait (pid=..., duration=...) > at ../../../3rdparty/libprocess/src/process.cpp:2958 > #4 0x7f6664e90558 in process::Latch::await (this=0x7f6ba0, duration=...) > at ../../../3rdparty/libprocess/src/latch.cpp:49 > #5 0x7f66649452cc in process::Future::await (this=0x7fffa0fd9040, > duration=...) > at ../../3rdparty/libprocess/include/process/future.hpp:1156 > #6 0x7f666493a04d in process::Future::get (this=0x7fffa0fd9040) > at ../../3rdparty/libprocess/include/process/future.hpp:1167 > #7 0x7f6664ab1aac in ZooKeeper::set (this=0x803ce0, path="/craig/mo", > data= > ... > > and > #0 0x00334e20b43c in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x7f6664ee1cf5 in Gate::arrive (this=0x7f66380013f0, old=0) > at ../../../3rdparty/libprocess/src/gate.hpp:82 > #2 0x7f6664ecef6e in process::ProcessManager::wait (this=0x7f02e0, > pid=...) > at ../../../3rdparty/libprocess/src/process.cpp:2476 > #3 0x7f6664ed2ce9 in process::wait (pid=..., duration=...) > at ../../../3rdparty/libprocess/src/process.cpp:2958 > #4 0x7f6664e90558 in process::Latch::await (this=0x7f6638000d00, > duration=...) > at ../../../3rdparty/libprocess/src/latch.cpp:49 > #5 0x7f66649452cc in process::Future::await (this=0x7f66595fb6f0, > duration=...) > at ../../3rdparty/libprocess/include/process/future.hpp:1156 > #6 0x7f666493a04d in process::Future::get (this=0x7f66595fb6f0) > at ../../3rdparty/libprocess/include/process/future.hpp:1167 > #7 0x7f6664ab18d3 in ZooKeeper::get (this=0x803ce0, path="/craig/mo", > watch=false, > > > So, really we are asking whether the mesos zk c++ api will be enhanced to not > block indefinitely when results are beyond a time bound. > > cheers > craig
