No.
On Nov 20, 8:52 pm, Phyo Arkar <[email protected]> wrote: > I was never into GAE , but if GAE SDK works locally without any need > from Google GAE ? > > Can we build cloud using it? > > On 11/21/10, mdipierro <[email protected]> wrote: > > >http://code.google.com/appengine/docs/python/taskqueue/ > > > On Nov 20, 7:29 pm, Phyo Arkar <[email protected]> wrote: > >> 2) could be a compatibility layer on top of google's task queue. > > >> google's task queue? on GAE? > > >> for many orginizations and companys , GAE or any other public clouds are > >> not > >> an option. They want their sensitive data to be private, and paranoid to > >> go > >> it online. The firm i am working on is building their own private cloud > >> and > >> for their clients. My extractor will also need to run over it too , later. > > >> So many private or goverment organisation , they will want non public > >> cloud > >> related technologies. > > >> On Sun, Nov 21, 2010 at 7:49 AM, Phyo Arkar > >> <[email protected]>wrote: > > >> > For my problem i have to use multiprocessing + Queue is: > > >> > I am developing a file indexer for a Digital Forensic Firm which extract > >> > huge archive of files , pst (ms Outlook archives), Read them . parse > >> > them , > >> > extract Mime and Indexable text out of many different type of documents > >> > , as > >> > much as possible. And put them into the database for later indexing by > >> > search engine. > > >> > Target is to process over 10GB of files at once. > > >> > When we tested 800MB archives, takes over 20 mins (without > >> > multiprocessing) > >> > . > > >> > machine is dual Core2 8GB DDR > > >> > Problems are , > >> > 1. it stop web2py's from responding both normal requests and ajax > >> > requests > >> > , that makes it impossible for showing progress to user. > >> > 2. it also stops Rocket's Keealive responds for 20 mins , making browser > >> > think its dead > > >> > my solution is to separate all processing out into a separate python > >> > process, spawn it via subprocess.Popen , without waiting it to end. > > >> > With that i can use multiprocessing module to spread loads across 4 > >> > processess , without bombarding web2py. > >> > With that web2py can function welll.The process is running at > >> > background. > >> > and with Ajax i can make a progress bar by checking the progress report > >> > of > >> > the separated process. > > >> > Problem there are : > > >> > 1. As it is outside of web2py theres no way of knowing if there error > >> > occoured, unless i monitoring the output of web2py process. > >> > 2. The way to communicate back to web2py is via files. I wrote process > >> > progress or error to 4 different files , which ajax requests monitor > >> > them > >> > for progress. > >> > 3. Between 4 processes it is easy to communicate parsed results via > >> > multiprocessing Queue , but as it is outside of web2py scope , cannot > >> > communicate using Queue > >> > 4. Also DAL have to be use as outside of web2py to put back results into > >> > database , and its ugly. > > >> > If tasks and ques of celery are integrated , there will be many > >> > benefits. > >> > Can easily load balance across multiple machines too. and communicate to > >> > and > >> > fro easily. > > >> > On Sun, Nov 21, 2010 at 6:06 AM, mdipierro > >> > <[email protected]>wrote: > > >> >> If we here to integrate a queue functionality in web2py what features > >> >> would you consider most valuable? > > >> >> I can three different applications: > >> >> 1) a message queue (one app sends a message, another one receives it) > >> >> 2) a task queue (an app can submit a task to be executed later): task > >> >> is executed by a remote process (or cloud) > >> >> 3) a task queue: task is executed by the app itself (in a separate > >> >> thread) but triggered by a queue callback. > > >> >> There is some overlap but they are subject to different optimizations. > >> >> 2) could be a compatibility layer on top of google's task queue. > >
