I was never into GAE , but if GAE SDK works locally without any need from Google GAE ?
Can we build cloud using it? On 11/21/10, mdipierro <mdipie...@cs.depaul.edu> wrote: > http://code.google.com/appengine/docs/python/taskqueue/ > > On Nov 20, 7:29 pm, Phyo Arkar <phyo.arkarl...@gmail.com> wrote: >> 2) could be a compatibility layer on top of google's task queue. >> >> google's task queue? on GAE? >> >> for many orginizations and companys , GAE or any other public clouds are >> not >> an option. They want their sensitive data to be private, and paranoid to >> go >> it online. The firm i am working on is building their own private cloud >> and >> for their clients. My extractor will also need to run over it too , later. >> >> So many private or goverment organisation , they will want non public >> cloud >> related technologies. >> >> On Sun, Nov 21, 2010 at 7:49 AM, Phyo Arkar >> <phyo.arkarl...@gmail.com>wrote: >> >> > For my problem i have to use multiprocessing + Queue is: >> >> > I am developing a file indexer for a Digital Forensic Firm which extract >> > huge archive of files , pst (ms Outlook archives), Read them . parse >> > them , >> > extract Mime and Indexable text out of many different type of documents >> > , as >> > much as possible. And put them into the database for later indexing by >> > search engine. >> >> > Target is to process over 10GB of files at once. >> >> > When we tested 800MB archives, takes over 20 mins (without >> > multiprocessing) >> > . >> >> > machine is dual Core2 8GB DDR >> >> > Problems are , >> > 1. it stop web2py's from responding both normal requests and ajax >> > requests >> > , that makes it impossible for showing progress to user. >> > 2. it also stops Rocket's Keealive responds for 20 mins , making browser >> > think its dead >> >> > my solution is to separate all processing out into a separate python >> > process, spawn it via subprocess.Popen , without waiting it to end. >> >> > With that i can use multiprocessing module to spread loads across 4 >> > processess , without bombarding web2py. >> > With that web2py can function welll.The process is running at >> > background. >> > and with Ajax i can make a progress bar by checking the progress report >> > of >> > the separated process. >> >> > Problem there are : >> >> > 1. As it is outside of web2py theres no way of knowing if there error >> > occoured, unless i monitoring the output of web2py process. >> > 2. The way to communicate back to web2py is via files. I wrote process >> > progress or error to 4 different files , which ajax requests monitor >> > them >> > for progress. >> > 3. Between 4 processes it is easy to communicate parsed results via >> > multiprocessing Queue , but as it is outside of web2py scope , cannot >> > communicate using Queue >> > 4. Also DAL have to be use as outside of web2py to put back results into >> > database , and its ugly. >> >> > If tasks and ques of celery are integrated , there will be many >> > benefits. >> > Can easily load balance across multiple machines too. and communicate to >> > and >> > fro easily. >> >> > On Sun, Nov 21, 2010 at 6:06 AM, mdipierro >> > <mdipie...@cs.depaul.edu>wrote: >> >> >> If we here to integrate a queue functionality in web2py what features >> >> would you consider most valuable? >> >> >> I can three different applications: >> >> 1) a message queue (one app sends a message, another one receives it) >> >> 2) a task queue (an app can submit a task to be executed later): task >> >> is executed by a remote process (or cloud) >> >> 3) a task queue: task is executed by the app itself (in a separate >> >> thread) but triggered by a queue callback. >> >> >> There is some overlap but they are subject to different optimizations. >> >> 2) could be a compatibility layer on top of google's task queue. >> >>
