[web2py] Re: Has anyone integrated user-driven multiprocessing?

2015-09-18 Thread Phillip
My basic interpretation: The scheduler shouldn't be managed by the webserver (shouldn't be controlled by user requests) which could basically create zombie processess and / or will drop long-running processes. If you see no reason the scheduler shouldn't work for this purpose (while

[web2py] Re: Has anyone integrated user-driven multiprocessing?

2015-09-18 Thread Dave S
On Friday, September 18, 2015 at 12:01:30 PM UTC-7, Niphlod wrote: > > imho the scenario isn't clear. Why having a sleuth of workers waiting for > tasks enqueued by your app is not viable for multiple users ? > > On Friday, September 18, 2015 at 8:14:31 PM UTC+2, Phillip wrote: >> >> If I

[web2py] Re: Has anyone integrated user-driven multiprocessing?

2015-09-18 Thread Niphlod
imho the scenario isn't clear. Why having a sleuth of workers waiting for tasks enqueued by your app is not viable for multiple users ? On Friday, September 18, 2015 at 8:14:31 PM UTC+2, Phillip wrote: > > If I understand, using the scheduler in my case would only be a viable > option for my

[web2py] Re: Has anyone integrated user-driven multiprocessing?

2015-09-18 Thread Niphlod
I still can't get the """viable option for my own processing purposes, not multiple users""" if they're viable for you, they're viable for multiple users. To a limit. BTW: you can't run external processes on GAE, there's only the TaskQueue (with some limitations) As for the number of

[web2py] Re: Has anyone integrated user-driven multiprocessing?

2015-09-18 Thread Phillip
If I understand, using the scheduler in my case would only be a viable option for my own processing purposes, not multiple users. If so, it appears that my only option would be to export a downloadable desktop version of the interface to be used for this processing. If I am at all offtrack

[web2py] Re: Has anyone integrated user-driven multiprocessing?

2015-09-18 Thread Anthony
On Friday, September 18, 2015 at 3:37:26 PM UTC-4, Phillip wrote: > > This was my basic interpretation of your post: Scheduler processes > shouldn't be managed by the webserver (shouldn't be controlled by user > requests) which could basically create zombie processess and / or will drop >

[web2py] Re: Has anyone integrated user-driven multiprocessing?

2015-09-17 Thread Anthony
Niphlod can probably provide better advice regarding managing scheduler workers and system resources, but yes, you should be able to have a few workers running ready to handle incoming tasks. On Wednesday, September 16, 2015 at 6:11:08 PM UTC-4, Phillip wrote: > > Are you saying that if I start

[web2py] Re: Has anyone integrated user-driven multiprocessing?

2015-09-17 Thread Niphlod
The basic concept of the scheduler is to have a process (or multiple ones) NOT managed by the webserver, that are ready to do some work when told to do so. Why ? 99% of the cases is composed by: - long-running computations that will incur in the webserver dropping the process for timeout -

[web2py] Re: Has anyone integrated user-driven multiprocessing?

2015-09-16 Thread Phillip
Are you saying that workers should be started before any tasks are queued (by an arbitrary number of users) which will be running for an arbitrary amount of time? If the answer is no Here is the set up: There is a grid of files from which a user can generate 'offspring' files in all possible

[web2py] Re: Has anyone integrated user-driven multiprocessing?

2015-09-15 Thread Anthony
Why can't you start one or more workers and then have the user simply schedule a task in the queue? On Tuesday, September 15, 2015 at 3:12:31 PM UTC-4, Phillip wrote: > > It is my understanding that, despite being able to queue tasks from the > controller, workers can only be started from the