Re: Prefork vs threaded when I want to spawn my own threads

[Andre Terra]

On Wed, Sep 4, 2013 at 2:36 PM, Javier Guerra Giraldez
wrote:

> above that, Redis is a clean, fast and very scalable platform for
> queues.  if you don't have heavy buzzword dependency, i wouldn't
> bother checking AMQP systems and stay with Redis.
>

A thousand times this. You don't need AMQP systems until you need them.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Prefork vs threaded when I want to spawn my own threads

[Javier Guerra Giraldez]

On Wed, Sep 4, 2013 at 12:28 PM,   wrote:
> What were people using back in PHP days, cron all the way?


MQs have existed for a long time; but weren't so popular.  I think the
Twitter problems were the wake-up call for most people.

in part because ghetto queues do work, and work well up to a point.
when you grow past its capacity it gets ugly, but below that it's a
real solution.   As first estimate, on anything that gets less than a
hundred tasks a minute, I wouldn't feel bad about using them.

above that, Redis is a clean, fast and very scalable platform for
queues.  if you don't have heavy buzzword dependency, i wouldn't
bother checking AMQP systems and stay with Redis.

-- 
Javier

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Prefork vs threaded when I want to spawn my own threads

[mailbox . tec]

On 3 september 2013 15:25:42 UTC+2 Russell Keith-Magee wrote:
>
>
> On Tue, Sep 3, 2013 at 9:50 PM,  wrote:
>
>> In my Django application I need to schedule some tasks to run in future. 
>> Celery is not an option here because some crucial options - mainly, the 
>> ability to revoke() a task - are only available with fully-featured MQ 
>> backend and I only have a limited shared hosting where I can't install this 
>> kind of software. After some reading, I've found that supposedly "right" 
>> way to do that in Python is to spawn a subprocess and call sleep(). 
>> However, spawning a process does have an overhead, especially considering a 
>> rather simple nature of tasks I want to defer (retrieval and conditional 
>> deletion of one object).
>>
>> Then I decided to use threads instead. They have little to no overhead, 
>> share memory and Python has a nice little threading.Timer class which is 
>> exactly what I need. My code looks like this:
>>
>> http://pastebin.com/K6RukAX9
>>
>> Here's when my questions come in. I admit my knowledge of threading is 
>> somewhat low and Python operates on high-level too, so nuances are not 
>> easily noticeable. From what I've learned, threads (can?) share memory with 
>> main thread/process. My primary concern here is that, as far as I 
>> understand, method=prefork (the default) creates a pool of processes 
>> waiting for work. It seems possible that it may impact my scheduler 
>> somehow, i.e. if process that spawned it sits idle (sleeps?) when countdown 
>> ends.
>>
>> On the other hand, threaded mode is supposedly more efficient because, 
>> being free from overhead of process forking, it can create child threads at 
>> will. In my mind this raises a similar concern to above: what if framework 
>> decides to destroy a thread that happens to be the one where Timer has been 
>> previously spawned?
>>
>> It is entirely possible that my concerns are completely baseless and both 
>> methods have nothing to do with my scheduler. I'm asking for input from 
>> smarter people than me :)
>>
>  
> Lets back up a bit here. 
>
> Web servers aren't designed to do the sort of thing you're describing. Web 
> requests are not designed to be long lived, and web servers are tuned with 
> this in mind. If you try to make them long lived, you're going to hit all 
> sorts of problems with thread/process lifecycle management. 
>
> The solution here isn't to try and make a web server do something it 
> wasn't designed to do. The solution is to work out how to get a task queue 
> working on your hardware setup.
>
> You say you can't install a fully-fledged MQ service. Ok - that's fine -- 
> but don't throw out the baby with the bathwater. There are other ways to 
> back Celery -- if need be, you can use a database as your data store.
>
> Or, you could consider using an entirely different task queue. If your 
> service provider will give you access to Redis, look into RQ. It's 
> lightweight, but 
>
> Or, if neither of those are viable, you can do a "ghetto queue" -- a 
> combination of cron and a database can be used to implement task queue-like 
> behaviour, without any need for celery.
>
> It's difficult to give more specific advice without knowing the specifics 
> of your problem domain and your deployment options -- but I implore you -- 
> don't keep down the path you're on. That way leads to madness :-)
>
> Yours,
> Russ Magee %-)
>

I get the overall message. I would use threads because that method looks 
simple enough, but if it isn't - I won't insist on it.

Regarding Celery, I was specifically referring to revoke() method, it's 
absolutely crucial for me to be able to revoke a scheduled task. Internet 
says it's only supported with AMQP and Redis ( 
http://stackoverflow.com/questions/7415657/celery-tasks-dont-get-revoked ) 
though now that I look at it, docs also claim compatibility with MongoDB. 
However, I didn't managed to get it to work on such a setup and complete 
lack of error messages didn't help (the call just fails silently).

I don't have Redis on my host, but I guess a free 20 MB instance on a cloud 
will be enough for my needs, so I'll settle with one of these small-ish 
queuers. On a side note, I am a bit surprised that every sane method to do 
scheduled task execution depends on MQs or Redis. Those are fairly new 
tools while the problem definitely didn't appear yesterday. What were 
people using back in PHP days, cron all the way?

Thank you very much for your remarks :)

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Prefork vs threaded when I want to spawn my own threads

[Javier Guerra Giraldez]

On Tue, Sep 3, 2013 at 9:28 AM, Dig  wrote:
> My django application is host in uwsgi, which is support timer(interval). So
> I insert my task parameters into database and execute tasks in timer
> handler.


note that uwsgi also has a spooler facility...

-- 
Javier

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Prefork vs threaded when I want to spawn my own threads

[Dig]

I have a similar need. And I implement this feature with uwsgi.

My django application is host in uwsgi, which is support timer(interval).
So I insert my task parameters into database and execute tasks in timer
handler.

Regards,
Dig
On Sep 3, 2013 9:26 PM, "Russell Keith-Magee" 
wrote:

>
> On Tue, Sep 3, 2013 at 9:50 PM,  wrote:
>
>> In my Django application I need to schedule some tasks to run in future.
>> Celery is not an option here because some crucial options - mainly, the
>> ability to revoke() a task - are only available with fully-featured MQ
>> backend and I only have a limited shared hosting where I can't install this
>> kind of software. After some reading, I've found that supposedly "right"
>> way to do that in Python is to spawn a subprocess and call sleep().
>> However, spawning a process does have an overhead, especially considering a
>> rather simple nature of tasks I want to defer (retrieval and conditional
>> deletion of one object).
>>
>> Then I decided to use threads instead. They have little to no overhead,
>> share memory and Python has a nice little threading.Timer class which is
>> exactly what I need. My code looks like this:
>>
>> http://pastebin.com/K6RukAX9
>>
>> Here's when my questions come in. I admit my knowledge of threading is
>> somewhat low and Python operates on high-level too, so nuances are not
>> easily noticeable. From what I've learned, threads (can?) share memory with
>> main thread/process. My primary concern here is that, as far as I
>> understand, method=prefork (the default) creates a pool of processes
>> waiting for work. It seems possible that it may impact my scheduler
>> somehow, i.e. if process that spawned it sits idle (sleeps?) when countdown
>> ends.
>>
>> On the other hand, threaded mode is supposedly more efficient because,
>> being free from overhead of process forking, it can create child threads at
>> will. In my mind this raises a similar concern to above: what if framework
>> decides to destroy a thread that happens to be the one where Timer has been
>> previously spawned?
>>
>> It is entirely possible that my concerns are completely baseless and both
>> methods have nothing to do with my scheduler. I'm asking for input from
>> smarter people than me :)
>>
>
> Lets back up a bit here.
>
> Web servers aren't designed to do the sort of thing you're describing. Web
> requests are not designed to be long lived, and web servers are tuned with
> this in mind. If you try to make them long lived, you're going to hit all
> sorts of problems with thread/process lifecycle management.
>
> The solution here isn't to try and make a web server do something it
> wasn't designed to do. The solution is to work out how to get a task queue
> working on your hardware setup.
>
> You say you can't install a fully-fledged MQ service. Ok - that's fine --
> but don't throw out the baby with the bathwater. There are other ways to
> back Celery -- if need be, you can use a database as your data store.
>
> Or, you could consider using an entirely different task queue. If your
> service provider will give you access to Redis, look into RQ. It's
> lightweight, but
>
> Or, if neither of those are viable, you can do a "ghetto queue" -- a
> combination of cron and a database can be used to implement task queue-like
> behaviour, without any need for celery.
>
> It's difficult to give more specific advice without knowing the specifics
> of your problem domain and your deployment options -- but I implore you --
> don't keep down the path you're on. That way leads to madness :-)
>
> Yours,
> Russ Magee %-)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Prefork vs threaded when I want to spawn my own threads

[Russell Keith-Magee]

On Tue, Sep 3, 2013 at 9:50 PM,  wrote:

> In my Django application I need to schedule some tasks to run in future.
> Celery is not an option here because some crucial options - mainly, the
> ability to revoke() a task - are only available with fully-featured MQ
> backend and I only have a limited shared hosting where I can't install this
> kind of software. After some reading, I've found that supposedly "right"
> way to do that in Python is to spawn a subprocess and call sleep().
> However, spawning a process does have an overhead, especially considering a
> rather simple nature of tasks I want to defer (retrieval and conditional
> deletion of one object).
>
> Then I decided to use threads instead. They have little to no overhead,
> share memory and Python has a nice little threading.Timer class which is
> exactly what I need. My code looks like this:
>
> http://pastebin.com/K6RukAX9
>
> Here's when my questions come in. I admit my knowledge of threading is
> somewhat low and Python operates on high-level too, so nuances are not
> easily noticeable. From what I've learned, threads (can?) share memory with
> main thread/process. My primary concern here is that, as far as I
> understand, method=prefork (the default) creates a pool of processes
> waiting for work. It seems possible that it may impact my scheduler
> somehow, i.e. if process that spawned it sits idle (sleeps?) when countdown
> ends.
>
> On the other hand, threaded mode is supposedly more efficient because,
> being free from overhead of process forking, it can create child threads at
> will. In my mind this raises a similar concern to above: what if framework
> decides to destroy a thread that happens to be the one where Timer has been
> previously spawned?
>
> It is entirely possible that my concerns are completely baseless and both
> methods have nothing to do with my scheduler. I'm asking for input from
> smarter people than me :)
>

Lets back up a bit here.

Web servers aren't designed to do the sort of thing you're describing. Web
requests are not designed to be long lived, and web servers are tuned with
this in mind. If you try to make them long lived, you're going to hit all
sorts of problems with thread/process lifecycle management.

The solution here isn't to try and make a web server do something it wasn't
designed to do. The solution is to work out how to get a task queue working
on your hardware setup.

You say you can't install a fully-fledged MQ service. Ok - that's fine --
but don't throw out the baby with the bathwater. There are other ways to
back Celery -- if need be, you can use a database as your data store.

Or, you could consider using an entirely different task queue. If your
service provider will give you access to Redis, look into RQ. It's
lightweight, but

Or, if neither of those are viable, you can do a "ghetto queue" -- a
combination of cron and a database can be used to implement task queue-like
behaviour, without any need for celery.

It's difficult to give more specific advice without knowing the specifics
of your problem domain and your deployment options -- but I implore you --
don't keep down the path you're on. That way leads to madness :-)

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
For more options, visit https://groups.google.com/groups/opt_out.

Prefork vs threaded when I want to spawn my own threads

[mailbox . tec]

In my Django application I need to schedule some tasks to run in future. 
Celery is not an option here because some crucial options - mainly, the 
ability to revoke() a task - are only available with fully-featured MQ 
backend and I only have a limited shared hosting where I can't install this 
kind of software. After some reading, I've found that supposedly "right" 
way to do that in Python is to spawn a subprocess and call sleep(). 
However, spawning a process does have an overhead, especially considering a 
rather simple nature of tasks I want to defer (retrieval and conditional 
deletion of one object).

Then I decided to use threads instead. They have little to no overhead, 
share memory and Python has a nice little threading.Timer class which is 
exactly what I need. My code looks like this:

http://pastebin.com/K6RukAX9

Here's when my questions come in. I admit my knowledge of threading is 
somewhat low and Python operates on high-level too, so nuances are not 
easily noticeable. From what I've learned, threads (can?) share memory with 
main thread/process. My primary concern here is that, as far as I 
understand, method=prefork (the default) creates a pool of processes 
waiting for work. It seems possible that it may impact my scheduler 
somehow, i.e. if process that spawned it sits idle (sleeps?) when countdown 
ends.

On the other hand, threaded mode is supposedly more efficient because, 
being free from overhead of process forking, it can create child threads at 
will. In my mind this raises a similar concern to above: what if framework 
decides to destroy a thread that happens to be the one where Timer has been 
previously spawned?

It is entirely possible that my concerns are completely baseless and both 
methods have nothing to do with my scheduler. I'm asking for input from 
smarter people than me :)

Regards,
Peter

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
For more options, visit https://groups.google.com/groups/opt_out.