Hi, On Fri, Apr 29, 2011 at 8:36 PM, footy <[email protected]> wrote:
> Thanks for the reply Ravi, > > No3. The worry is not scaling. I know GAE will do it for me. > What I was trying to say is that the level of scaling required for > Producer and Consumer is different. > > It is critical for me to present the front end (App1: Producer) to the > users under heavy load. Thus the producer needs to be scaled > differently. for eg: I might need 25 instances. > As for the worker back end (App2: Consumer) it is not that critical, > and I am okay with a bit of delay as the consumer churns through the > work load of various tasks queued in the task queue. > Thus the consumer needs to be scaled differently. for eg: I might only > need 5 instances to run through the tasks. > This is not an issue: App Engine will serve user-facing traffic in preference to task queue traffic, and will scale the resources provisioned to handle both where possible. Having two separate apps won't give you any more scalability than one. -Nick Johnson > > By having it as part of a single app this might not be possible as the > entire app will be scaled as a whole. > > Also, having it as 2 separate apps, I can make the workflow completely > decoupled and asynchronous. App2 in this case would run like a daemon > to process the jobs in the queue. > Being new to GAE I am not sure how that can be done via a single App. > > On Apr 29, 11:25 am, Ravi Sharma <[email protected]> wrote: > > Hi Anuj, > > I dont think you can do it. > > 1) Task queue cant be share between two apps, although you can do like > this > > App1 gives one url which App2 can call and then App1 Controller will > create > > task for App2 in App1. Or you can use Amazon Queues or send XMPP message > > from APP1 to APP2 > > 2) Even if you implement such controller/queus/xmpp still your design > > pattern wont work on GAE as App2 will not be able to access files written > in > > datastore by App1,(people have been asking such implementaion ia m not > sure > > if it has been implemented till now). > > 3) And even if somehow you managed to do all(by doing lots of extra > overhead > > and realzing it woulkd have been better to stay on one app only) it may > go > > against the Google terms and conditions as you are dividing the same work > on > > different app. > > you will be using 2 times of FREE cpu, 2 times of free Datastore limit > etc. > > > > I dont understand the full reasoning behind this design, as i dont know > full > > requirments,designs,constraints etc but i can say your 3rd reason is kind > of > > invalid as you dont have to worry about scaling much...GAE will do it for > > your self. > > > > > > > > > > > > > > > > On Fri, Apr 29, 2011 at 10:01 AM, footy <[email protected]> wrote: > > > Thanks Nick and Steve for your answers. > > > > > I wasn't planning on storing data in the queue, but just a job message > > > with a payload giving enough information to the Consumer ( photo > > > processor ) for further > > > processing of the images. > > > > > Also, by consumer I don' mean the user itself, but as a design pattern > > > of Produce - Consumer. The images are produced ( uploaded ) by the > > > producer. > > > > > My main query was regarding sharing the task queue between two apps. > > > > > App1 = Producer - produces (uploads) images > > > App2 = Consumer - consumers ( processes ) images > > > > > Steps: > > > 1. App1 finishes the upload of images to datastore. > > > 2. App1 creates a Job message in the task queue called Job1 > > > 3. App2 picks Job1 message from the task queue > > > 4. App2 finds the image specified in the payload of Job1 from > > > datastore > > > 5. App2 does processing on the image and stores it back in the > > > datastore > > > 6. App2 notifies App1 that the processing is finished. > > > > > Reasons for doing this > > > - separate, isolated code base > > > - can be managed separately > > > - can easily deal with the different level of scaling requirement for > > > both App1 and App2 > > > > > Would it be possible? > > > > > On Apr 29, 8:08 am, "Nick Johnson (Google)" <[email protected]> > > > wrote: > > > > On Fri, Apr 29, 2011 at 4:12 PM, stevep <[email protected]> wrote: > > > > > You wrote: > > > > > > > > Producer: Produces photos. These are uploaded by the user. > > > > > > Once the upload is done, I would like to put up a task in the > queue, > > > > > > to be ready for further processing > > > > > > > Keep in mind that data payloads sent to a taskqueue are quite > > > > > constrained re: size. > > > > > > > I tried sending a 600x400 pixel jpeg image, and it would error out. > > > > > Nick, though, infers there is a workable means. > > > > > > Yes - simply store the original image to the datastore. If he's > dealing > > > with > > > > large images, they were probably already uploaded to blobstore, in > which > > > > case he needs merely pass the blobkey to the task. > > > > > > -Nick Johnson > > > > > > > TQs in my experience are great in concept, but oft lack for > > > > > applicability for real-world problems. > > > > > > > Despite that, I use them extensively. > > > > > > > Best of luck, > > > > > stevep > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > Groups > > > > > "Google App Engine" group. > > > > > To post to this group, send email to > [email protected] > > > . > > > > > To unsubscribe from this group, send email to > > > > > [email protected]. > > > > > For more options, visit this group at > > > > >http://groups.google.com/group/google-appengine?hl=en. > > > > > > -- > > > > Nick Johnson, Developer Programs Engineer, App Engine > > > > > -- > > > You received this message because you are subscribed to the Google > Groups > > > "Google App Engine" group. > > > To post to this group, send email to [email protected] > . > > > To unsubscribe from this group, send email to > > > [email protected]. > > > For more options, visit this group at > > >http://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
