Thanks for the reply Ravi, No3. The worry is not scaling. I know GAE will do it for me. What I was trying to say is that the level of scaling required for Producer and Consumer is different.
It is critical for me to present the front end (App1: Producer) to the users under heavy load. Thus the producer needs to be scaled differently. for eg: I might need 25 instances. As for the worker back end (App2: Consumer) it is not that critical, and I am okay with a bit of delay as the consumer churns through the work load of various tasks queued in the task queue. Thus the consumer needs to be scaled differently. for eg: I might only need 5 instances to run through the tasks. By having it as part of a single app this might not be possible as the entire app will be scaled as a whole. Also, having it as 2 separate apps, I can make the workflow completely decoupled and asynchronous. App2 in this case would run like a daemon to process the jobs in the queue. Being new to GAE I am not sure how that can be done via a single App. On Apr 29, 11:25 am, Ravi Sharma <[email protected]> wrote: > Hi Anuj, > I dont think you can do it. > 1) Task queue cant be share between two apps, although you can do like this > App1 gives one url which App2 can call and then App1 Controller will create > task for App2 in App1. Or you can use Amazon Queues or send XMPP message > from APP1 to APP2 > 2) Even if you implement such controller/queus/xmpp still your design > pattern wont work on GAE as App2 will not be able to access files written in > datastore by App1,(people have been asking such implementaion ia m not sure > if it has been implemented till now). > 3) And even if somehow you managed to do all(by doing lots of extra overhead > and realzing it woulkd have been better to stay on one app only) it may go > against the Google terms and conditions as you are dividing the same work on > different app. > you will be using 2 times of FREE cpu, 2 times of free Datastore limit etc. > > I dont understand the full reasoning behind this design, as i dont know full > requirments,designs,constraints etc but i can say your 3rd reason is kind of > invalid as you dont have to worry about scaling much...GAE will do it for > your self. > > > > > > > > On Fri, Apr 29, 2011 at 10:01 AM, footy <[email protected]> wrote: > > Thanks Nick and Steve for your answers. > > > I wasn't planning on storing data in the queue, but just a job message > > with a payload giving enough information to the Consumer ( photo > > processor ) for further > > processing of the images. > > > Also, by consumer I don' mean the user itself, but as a design pattern > > of Produce - Consumer. The images are produced ( uploaded ) by the > > producer. > > > My main query was regarding sharing the task queue between two apps. > > > App1 = Producer - produces (uploads) images > > App2 = Consumer - consumers ( processes ) images > > > Steps: > > 1. App1 finishes the upload of images to datastore. > > 2. App1 creates a Job message in the task queue called Job1 > > 3. App2 picks Job1 message from the task queue > > 4. App2 finds the image specified in the payload of Job1 from > > datastore > > 5. App2 does processing on the image and stores it back in the > > datastore > > 6. App2 notifies App1 that the processing is finished. > > > Reasons for doing this > > - separate, isolated code base > > - can be managed separately > > - can easily deal with the different level of scaling requirement for > > both App1 and App2 > > > Would it be possible? > > > On Apr 29, 8:08 am, "Nick Johnson (Google)" <[email protected]> > > wrote: > > > On Fri, Apr 29, 2011 at 4:12 PM, stevep <[email protected]> wrote: > > > > You wrote: > > > > > > Producer: Produces photos. These are uploaded by the user. > > > > > Once the upload is done, I would like to put up a task in the queue, > > > > > to be ready for further processing > > > > > Keep in mind that data payloads sent to a taskqueue are quite > > > > constrained re: size. > > > > > I tried sending a 600x400 pixel jpeg image, and it would error out. > > > > Nick, though, infers there is a workable means. > > > > Yes - simply store the original image to the datastore. If he's dealing > > with > > > large images, they were probably already uploaded to blobstore, in which > > > case he needs merely pass the blobkey to the task. > > > > -Nick Johnson > > > > > TQs in my experience are great in concept, but oft lack for > > > > applicability for real-world problems. > > > > > Despite that, I use them extensively. > > > > > Best of luck, > > > > stevep > > > > > -- > > > > You received this message because you are subscribed to the Google > > Groups > > > > "Google App Engine" group. > > > > To post to this group, send email to [email protected] > > . > > > > To unsubscribe from this group, send email to > > > > [email protected]. > > > > For more options, visit this group at > > > >http://groups.google.com/group/google-appengine?hl=en. > > > > -- > > > Nick Johnson, Developer Programs Engineer, App Engine > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
