If you really want to keep the code "decoupled" you could potentially use separate versions and specify a 'target' version for the queue.
Or, this is what I would probably suggest, include the 'consumer' as a separate WSGI App (for Python) or Servlet (in Java). http://code.google.com/appengine/docs/python/config/appconfig.html#Script_Handlers http://code.google.com/appengine/docs/java/config/webxml.html#Servlets_and_URL_Paths On Fri, Apr 29, 2011 at 07:03, footy <[email protected]> wrote: > Okay, so as for my design, how do I make this decoupled. > I hope I have given a fair idea, on what I am trying to do. > So if I were to use a single app, what would be the process workflow > between Producer and Consumer? > > On Apr 29, 11:54 am, "Nick Johnson (Google)" <[email protected]> > wrote: >> Hi, >> >> >> >> >> >> >> >> >> >> On Fri, Apr 29, 2011 at 8:36 PM, footy <[email protected]> wrote: >> > Thanks for the reply Ravi, >> >> > No3. The worry is not scaling. I know GAE will do it for me. >> > What I was trying to say is that the level of scaling required for >> > Producer and Consumer is different. >> >> > It is critical for me to present the front end (App1: Producer) to the >> > users under heavy load. Thus the producer needs to be scaled >> > differently. for eg: I might need 25 instances. >> > As for the worker back end (App2: Consumer) it is not that critical, >> > and I am okay with a bit of delay as the consumer churns through the >> > work load of various tasks queued in the task queue. >> > Thus the consumer needs to be scaled differently. for eg: I might only >> > need 5 instances to run through the tasks. >> >> This is not an issue: App Engine will serve user-facing traffic in >> preference to task queue traffic, and will scale the resources provisioned >> to handle both where possible. Having two separate apps won't give you any >> more scalability than one. >> >> -Nick Johnson >> >> >> >> >> >> >> >> >> >> >> >> > By having it as part of a single app this might not be possible as the >> > entire app will be scaled as a whole. >> >> > Also, having it as 2 separate apps, I can make the workflow completely >> > decoupled and asynchronous. App2 in this case would run like a daemon >> > to process the jobs in the queue. >> > Being new to GAE I am not sure how that can be done via a single App. >> >> > On Apr 29, 11:25 am, Ravi Sharma <[email protected]> wrote: >> > > Hi Anuj, >> > > I dont think you can do it. >> > > 1) Task queue cant be share between two apps, although you can do like >> > this >> > > App1 gives one url which App2 can call and then App1 Controller will >> > create >> > > task for App2 in App1. Or you can use Amazon Queues or send XMPP message >> > > from APP1 to APP2 >> > > 2) Even if you implement such controller/queus/xmpp still your design >> > > pattern wont work on GAE as App2 will not be able to access files written >> > in >> > > datastore by App1,(people have been asking such implementaion ia m not >> > sure >> > > if it has been implemented till now). >> > > 3) And even if somehow you managed to do all(by doing lots of extra >> > overhead >> > > and realzing it woulkd have been better to stay on one app only) it may >> > go >> > > against the Google terms and conditions as you are dividing the same work >> > on >> > > different app. >> > > you will be using 2 times of FREE cpu, 2 times of free Datastore limit >> > etc. >> >> > > I dont understand the full reasoning behind this design, as i dont know >> > full >> > > requirments,designs,constraints etc but i can say your 3rd reason is kind >> > of >> > > invalid as you dont have to worry about scaling much...GAE will do it for >> > > your self. >> >> > > On Fri, Apr 29, 2011 at 10:01 AM, footy <[email protected]> wrote: >> > > > Thanks Nick and Steve for your answers. >> >> > > > I wasn't planning on storing data in the queue, but just a job message >> > > > with a payload giving enough information to the Consumer ( photo >> > > > processor ) for further >> > > > processing of the images. >> >> > > > Also, by consumer I don' mean the user itself, but as a design pattern >> > > > of Produce - Consumer. The images are produced ( uploaded ) by the >> > > > producer. >> >> > > > My main query was regarding sharing the task queue between two apps. >> >> > > > App1 = Producer - produces (uploads) images >> > > > App2 = Consumer - consumers ( processes ) images >> >> > > > Steps: >> > > > 1. App1 finishes the upload of images to datastore. >> > > > 2. App1 creates a Job message in the task queue called Job1 >> > > > 3. App2 picks Job1 message from the task queue >> > > > 4. App2 finds the image specified in the payload of Job1 from >> > > > datastore >> > > > 5. App2 does processing on the image and stores it back in the >> > > > datastore >> > > > 6. App2 notifies App1 that the processing is finished. >> >> > > > Reasons for doing this >> > > > - separate, isolated code base >> > > > - can be managed separately >> > > > - can easily deal with the different level of scaling requirement for >> > > > both App1 and App2 >> >> > > > Would it be possible? >> >> > > > On Apr 29, 8:08 am, "Nick Johnson (Google)" <[email protected]> >> > > > wrote: >> > > > > On Fri, Apr 29, 2011 at 4:12 PM, stevep <[email protected]> wrote: >> > > > > > You wrote: >> >> > > > > > > Producer: Produces photos. These are uploaded by the user. >> > > > > > > Once the upload is done, I would like to put up a task in the >> > queue, >> > > > > > > to be ready for further processing >> >> > > > > > Keep in mind that data payloads sent to a taskqueue are quite >> > > > > > constrained re: size. >> >> > > > > > I tried sending a 600x400 pixel jpeg image, and it would error out. >> > > > > > Nick, though, infers there is a workable means. >> >> > > > > Yes - simply store the original image to the datastore. If he's >> > dealing >> > > > with >> > > > > large images, they were probably already uploaded to blobstore, in >> > which >> > > > > case he needs merely pass the blobkey to the task. >> >> > > > > -Nick Johnson >> >> > > > > > TQs in my experience are great in concept, but oft lack for >> > > > > > applicability for real-world problems. >> >> > > > > > Despite that, I use them extensively. >> >> > > > > > Best of luck, >> > > > > > stevep >> >> > > > > > -- >> > > > > > You received this message because you are subscribed to the Google >> > > > Groups >> > > > > > "Google App Engine" group. >> > > > > > To post to this group, send email to >> > [email protected] >> > > > . >> > > > > > To unsubscribe from this group, send email to >> > > > > > [email protected]. >> > > > > > For more options, visit this group at >> > > > > >http://groups.google.com/group/google-appengine?hl=en. >> >> > > > > -- >> > > > > Nick Johnson, Developer Programs Engineer, App Engine >> >> > > > -- >> > > > You received this message because you are subscribed to the Google >> > Groups >> > > > "Google App Engine" group. >> > > > To post to this group, send email to [email protected] >> > . >> > > > To unsubscribe from this group, send email to >> > > > [email protected]. >> > > > For more options, visit this group at >> > > >http://groups.google.com/group/google-appengine?hl=en. >> >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "Google App Engine" group. >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]. >> > For more options, visit this group at >> >http://groups.google.com/group/google-appengine?hl=en. >> >> -- >> Nick Johnson, Developer Programs Engineer, App Engine > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
