Okay, so as for my design, how do I make this decoupled. I hope I have given a fair idea, on what I am trying to do. So if I were to use a single app, what would be the process workflow between Producer and Consumer?
On Apr 29, 11:54 am, "Nick Johnson (Google)" <[email protected]> wrote: > Hi, > > > > > > > > > > On Fri, Apr 29, 2011 at 8:36 PM, footy <[email protected]> wrote: > > Thanks for the reply Ravi, > > > No3. The worry is not scaling. I know GAE will do it for me. > > What I was trying to say is that the level of scaling required for > > Producer and Consumer is different. > > > It is critical for me to present the front end (App1: Producer) to the > > users under heavy load. Thus the producer needs to be scaled > > differently. for eg: I might need 25 instances. > > As for the worker back end (App2: Consumer) it is not that critical, > > and I am okay with a bit of delay as the consumer churns through the > > work load of various tasks queued in the task queue. > > Thus the consumer needs to be scaled differently. for eg: I might only > > need 5 instances to run through the tasks. > > This is not an issue: App Engine will serve user-facing traffic in > preference to task queue traffic, and will scale the resources provisioned > to handle both where possible. Having two separate apps won't give you any > more scalability than one. > > -Nick Johnson > > > > > > > > > > > > > By having it as part of a single app this might not be possible as the > > entire app will be scaled as a whole. > > > Also, having it as 2 separate apps, I can make the workflow completely > > decoupled and asynchronous. App2 in this case would run like a daemon > > to process the jobs in the queue. > > Being new to GAE I am not sure how that can be done via a single App. > > > On Apr 29, 11:25 am, Ravi Sharma <[email protected]> wrote: > > > Hi Anuj, > > > I dont think you can do it. > > > 1) Task queue cant be share between two apps, although you can do like > > this > > > App1 gives one url which App2 can call and then App1 Controller will > > create > > > task for App2 in App1. Or you can use Amazon Queues or send XMPP message > > > from APP1 to APP2 > > > 2) Even if you implement such controller/queus/xmpp still your design > > > pattern wont work on GAE as App2 will not be able to access files written > > in > > > datastore by App1,(people have been asking such implementaion ia m not > > sure > > > if it has been implemented till now). > > > 3) And even if somehow you managed to do all(by doing lots of extra > > overhead > > > and realzing it woulkd have been better to stay on one app only) it may > > go > > > against the Google terms and conditions as you are dividing the same work > > on > > > different app. > > > you will be using 2 times of FREE cpu, 2 times of free Datastore limit > > etc. > > > > I dont understand the full reasoning behind this design, as i dont know > > full > > > requirments,designs,constraints etc but i can say your 3rd reason is kind > > of > > > invalid as you dont have to worry about scaling much...GAE will do it for > > > your self. > > > > On Fri, Apr 29, 2011 at 10:01 AM, footy <[email protected]> wrote: > > > > Thanks Nick and Steve for your answers. > > > > > I wasn't planning on storing data in the queue, but just a job message > > > > with a payload giving enough information to the Consumer ( photo > > > > processor ) for further > > > > processing of the images. > > > > > Also, by consumer I don' mean the user itself, but as a design pattern > > > > of Produce - Consumer. The images are produced ( uploaded ) by the > > > > producer. > > > > > My main query was regarding sharing the task queue between two apps. > > > > > App1 = Producer - produces (uploads) images > > > > App2 = Consumer - consumers ( processes ) images > > > > > Steps: > > > > 1. App1 finishes the upload of images to datastore. > > > > 2. App1 creates a Job message in the task queue called Job1 > > > > 3. App2 picks Job1 message from the task queue > > > > 4. App2 finds the image specified in the payload of Job1 from > > > > datastore > > > > 5. App2 does processing on the image and stores it back in the > > > > datastore > > > > 6. App2 notifies App1 that the processing is finished. > > > > > Reasons for doing this > > > > - separate, isolated code base > > > > - can be managed separately > > > > - can easily deal with the different level of scaling requirement for > > > > both App1 and App2 > > > > > Would it be possible? > > > > > On Apr 29, 8:08 am, "Nick Johnson (Google)" <[email protected]> > > > > wrote: > > > > > On Fri, Apr 29, 2011 at 4:12 PM, stevep <[email protected]> wrote: > > > > > > You wrote: > > > > > > > > Producer: Produces photos. These are uploaded by the user. > > > > > > > Once the upload is done, I would like to put up a task in the > > queue, > > > > > > > to be ready for further processing > > > > > > > Keep in mind that data payloads sent to a taskqueue are quite > > > > > > constrained re: size. > > > > > > > I tried sending a 600x400 pixel jpeg image, and it would error out. > > > > > > Nick, though, infers there is a workable means. > > > > > > Yes - simply store the original image to the datastore. If he's > > dealing > > > > with > > > > > large images, they were probably already uploaded to blobstore, in > > which > > > > > case he needs merely pass the blobkey to the task. > > > > > > -Nick Johnson > > > > > > > TQs in my experience are great in concept, but oft lack for > > > > > > applicability for real-world problems. > > > > > > > Despite that, I use them extensively. > > > > > > > Best of luck, > > > > > > stevep > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > Groups > > > > > > "Google App Engine" group. > > > > > > To post to this group, send email to > > [email protected] > > > > . > > > > > > To unsubscribe from this group, send email to > > > > > > [email protected]. > > > > > > For more options, visit this group at > > > > > >http://groups.google.com/group/google-appengine?hl=en. > > > > > > -- > > > > > Nick Johnson, Developer Programs Engineer, App Engine > > > > > -- > > > > You received this message because you are subscribed to the Google > > Groups > > > > "Google App Engine" group. > > > > To post to this group, send email to [email protected] > > . > > > > To unsubscribe from this group, send email to > > > > [email protected]. > > > > For more options, visit this group at > > > >http://groups.google.com/group/google-appengine?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. > > -- > Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
