http://code.google.com/appengine/docs/python/taskqueue/
On Nov 20, 7:29 pm, Phyo Arkar <[email protected]> wrote: > 2) could be a compatibility layer on top of google's task queue. > > google's task queue? on GAE? > > for many orginizations and companys , GAE or any other public clouds are not > an option. They want their sensitive data to be private, and paranoid to go > it online. The firm i am working on is building their own private cloud and > for their clients. My extractor will also need to run over it too , later. > > So many private or goverment organisation , they will want non public cloud > related technologies. > > On Sun, Nov 21, 2010 at 7:49 AM, Phyo Arkar <[email protected]>wrote: > > > For my problem i have to use multiprocessing + Queue is: > > > I am developing a file indexer for a Digital Forensic Firm which extract > > huge archive of files , pst (ms Outlook archives), Read them . parse them , > > extract Mime and Indexable text out of many different type of documents , as > > much as possible. And put them into the database for later indexing by > > search engine. > > > Target is to process over 10GB of files at once. > > > When we tested 800MB archives, takes over 20 mins (without multiprocessing) > > . > > > machine is dual Core2 8GB DDR > > > Problems are , > > 1. it stop web2py's from responding both normal requests and ajax requests > > , that makes it impossible for showing progress to user. > > 2. it also stops Rocket's Keealive responds for 20 mins , making browser > > think its dead > > > my solution is to separate all processing out into a separate python > > process, spawn it via subprocess.Popen , without waiting it to end. > > > With that i can use multiprocessing module to spread loads across 4 > > processess , without bombarding web2py. > > With that web2py can function welll.The process is running at background. > > and with Ajax i can make a progress bar by checking the progress report of > > the separated process. > > > Problem there are : > > > 1. As it is outside of web2py theres no way of knowing if there error > > occoured, unless i monitoring the output of web2py process. > > 2. The way to communicate back to web2py is via files. I wrote process > > progress or error to 4 different files , which ajax requests monitor them > > for progress. > > 3. Between 4 processes it is easy to communicate parsed results via > > multiprocessing Queue , but as it is outside of web2py scope , cannot > > communicate using Queue > > 4. Also DAL have to be use as outside of web2py to put back results into > > database , and its ugly. > > > If tasks and ques of celery are integrated , there will be many benefits. > > Can easily load balance across multiple machines too. and communicate to and > > fro easily. > > > On Sun, Nov 21, 2010 at 6:06 AM, mdipierro <[email protected]>wrote: > > >> If we here to integrate a queue functionality in web2py what features > >> would you consider most valuable? > > >> I can three different applications: > >> 1) a message queue (one app sends a message, another one receives it) > >> 2) a task queue (an app can submit a task to be executed later): task > >> is executed by a remote process (or cloud) > >> 3) a task queue: task is executed by the app itself (in a separate > >> thread) but triggered by a queue callback. > > >> There is some overlap but they are subject to different optimizations. > >> 2) could be a compatibility layer on top of google's task queue. > >
