so i am confused :D Google taskqueues wont work without running on Google GAE Cloud right?
On 11/21/10, mdipierro <[email protected]> wrote: > No. > > On Nov 20, 8:52 pm, Phyo Arkar <[email protected]> wrote: >> I was never into GAE , but if GAE SDK works locally without any need >> from Google GAE ? >> >> Can we build cloud using it? >> >> On 11/21/10, mdipierro <[email protected]> wrote: >> >> >http://code.google.com/appengine/docs/python/taskqueue/ >> >> > On Nov 20, 7:29 pm, Phyo Arkar <[email protected]> wrote: >> >> 2) could be a compatibility layer on top of google's task queue. >> >> >> google's task queue? on GAE? >> >> >> for many orginizations and companys , GAE or any other public clouds >> >> are >> >> not >> >> an option. They want their sensitive data to be private, and paranoid >> >> to >> >> go >> >> it online. The firm i am working on is building their own private cloud >> >> and >> >> for their clients. My extractor will also need to run over it too , >> >> later. >> >> >> So many private or goverment organisation , they will want non public >> >> cloud >> >> related technologies. >> >> >> On Sun, Nov 21, 2010 at 7:49 AM, Phyo Arkar >> >> <[email protected]>wrote: >> >> >> > For my problem i have to use multiprocessing + Queue is: >> >> >> > I am developing a file indexer for a Digital Forensic Firm which >> >> > extract >> >> > huge archive of files , pst (ms Outlook archives), Read them . parse >> >> > them , >> >> > extract Mime and Indexable text out of many different type of >> >> > documents >> >> > , as >> >> > much as possible. And put them into the database for later indexing >> >> > by >> >> > search engine. >> >> >> > Target is to process over 10GB of files at once. >> >> >> > When we tested 800MB archives, takes over 20 mins (without >> >> > multiprocessing) >> >> > . >> >> >> > machine is dual Core2 8GB DDR >> >> >> > Problems are , >> >> > 1. it stop web2py's from responding both normal requests and ajax >> >> > requests >> >> > , that makes it impossible for showing progress to user. >> >> > 2. it also stops Rocket's Keealive responds for 20 mins , making >> >> > browser >> >> > think its dead >> >> >> > my solution is to separate all processing out into a separate python >> >> > process, spawn it via subprocess.Popen , without waiting it to end. >> >> >> > With that i can use multiprocessing module to spread loads across 4 >> >> > processess , without bombarding web2py. >> >> > With that web2py can function welll.The process is running at >> >> > background. >> >> > and with Ajax i can make a progress bar by checking the progress >> >> > report >> >> > of >> >> > the separated process. >> >> >> > Problem there are : >> >> >> > 1. As it is outside of web2py theres no way of knowing if there error >> >> > occoured, unless i monitoring the output of web2py process. >> >> > 2. The way to communicate back to web2py is via files. I wrote >> >> > process >> >> > progress or error to 4 different files , which ajax requests monitor >> >> > them >> >> > for progress. >> >> > 3. Between 4 processes it is easy to communicate parsed results via >> >> > multiprocessing Queue , but as it is outside of web2py scope , cannot >> >> > communicate using Queue >> >> > 4. Also DAL have to be use as outside of web2py to put back results >> >> > into >> >> > database , and its ugly. >> >> >> > If tasks and ques of celery are integrated , there will be many >> >> > benefits. >> >> > Can easily load balance across multiple machines too. and communicate >> >> > to >> >> > and >> >> > fro easily. >> >> >> > On Sun, Nov 21, 2010 at 6:06 AM, mdipierro >> >> > <[email protected]>wrote: >> >> >> >> If we here to integrate a queue functionality in web2py what >> >> >> features >> >> >> would you consider most valuable? >> >> >> >> I can three different applications: >> >> >> 1) a message queue (one app sends a message, another one receives >> >> >> it) >> >> >> 2) a task queue (an app can submit a task to be executed later): >> >> >> task >> >> >> is executed by a remote process (or cloud) >> >> >> 3) a task queue: task is executed by the app itself (in a separate >> >> >> thread) but triggered by a queue callback. >> >> >> >> There is some overlap but they are subject to different >> >> >> optimizations. >> >> >> 2) could be a compatibility layer on top of google's task queue. >> >>
