Thx again Nick. Yes, I need to tune total_storage_limit and I just find actually the document about queue.yaml does talk about that, but i missed that part before. This is what exactly i'm looking for.
cheers, eric On 9 December 2010 11:24, Nicholas Verne <[email protected]> wrote: > from T2-T4, the attempts to add tasks to the queue will fail because > the Task Queue Stored Task Bytes is at or over quota. > > The solution is to give yourself head room. You do this by setting > your limit to, say, 500 MB with the follwing in queue.yaml > > total_storage_limit: 500M > > Alternatively, in queue.xml > > <total-storage-limit> 500M </total-storage-limit> > > Note that your Task Queue Stored Task Bytes also counts toward your > Total Stored Data. More Total Stored Data can be obtained by enabling > billing. > > Nick Verne > > On Thu, Dec 9, 2010 at 2:14 PM, Eric Ka Ka Ng <[email protected]> wrote: > > Hi Nick, > > Thank for your reply! guess i have more understanding on how the Task > Queue > > Stored Task Bytes be counted > > as Task Queue Stored Task Bytes increases in 'near real time' when task > is > > being added, but decreases 'every couple of hours', so within this > 'couple > > of hours', this counter is sort of accumulative? > > for example, would the following scenario be possible? > > assume T4-T3=T3-T2=T2-T1=T1-T0=30 mins, and the counter decreases 2 hours > > after T0 (at T4) > > before T0, there were no tasks, Task Queue Stored Task Byte = 0 > > from T0-T1, many tasks are added, and Task Queue Stored Task Byte = 100MB > > from T1-T2, all previous added tasks has been finished processing, > leaving > > the queue empty. Task Queue Stored Task Byte is still 100MB > > from T2-T4, some tasks are added, but since Task Queue Stored Task Byte = > > 100MB (over the quota), those tasks would be accumulated in the queue > > but cannot be executed, making Task Queue Stored Task Byte continue to > be > > increasing to let say 200MB > > at T4, Task Queue Stored Task Byte started to decrease, by the amount of > > task being executed from T1-T2. Task Queue Stored Task Byte = 100MB (over > > quota). newly added tasks since T2 still cannot be executed > > > > > > anyway, seems being able to increase the quota for Task Queue Stored Task > > Byte would be a MUST-HAVE for us, in order to avoid the tasks being stuck > > and hanging the system when there is a short burst of tasks . could you > > advise how we could configure this quota on GAE? > > Many thanks, > > Eric Ng > > On 8 December 2010 10:15, Nicholas Verne <[email protected]> wrote: > >> > >> Eric, > >> > >> Task Queue Stored Task Count is updated in "near real" time. Each task > >> that is added to a queue is counted at that time, and each task that > >> is deleted after sucessful execution is subtracted. > >> Task Queue Stored Task Bytes increases as each task is added, but > >> decreases only every couple of hours when a process runs which takes > >> an accurate measurement of real usage. > >> > >> To estimate the average number of bytes/task, you could pause the > >> relevant queue, then watch the Task Queue Stored Task Bytes change as > >> you add tasks. > >> > >> Nick Verne > >> > >> On Tue, Dec 7, 2010 at 8:46 PM, Eric Ka Ka Ng <[email protected]> wrote: > >> > Once upon a time we experienced problem when we reach the Task Queue > >> > Stored > >> > Task Bytes Quota Limit (i believe is 100MB for paid application) . It > >> > happened when 'producer' produce too many tasks into the queue before > >> > the > >> > 'consumer' can consume them, leaving many many tasks being in the > queue. > >> > Although we fixed the problem for 'producer' faster than 'consumer' > >> > problem, > >> > this trigger us to have a better understanding on how the quota 'Task > >> > Queue > >> > Stored Task Bytes' be counted and configured > >> > > >> > Specially i have 3 questions > >> > 1. how to count the 'bytes' of a task queue stored task? > >> > let say each task consumes 50K, as we should have 100MB of such quota, > >> > it > >> > should have the capacity to store 20000 tasks. but how could we > >> > calculate / > >> > estimate the task stored size? > >> > 2. is this a counter showing the 'real-time current' total stored task > >> > size, > >> > or a counter being updated regularly, or an accumulative counter being > >> > reset > >> > in daily basis? > >> > i was thinking it should not be an accumulative counter, but sometimes > >> > when > >> > i check the system we dont have any tasks in the queue, but the "Tasek > >> > Queue > >> > Stored Task Bytes" is not 0 > >> > > >> > 3. how to configure the quota for Task Queue Stored Task Bytes' > >> > in GAE > >> > > >> > documentation > http://code.google.com/appengine/docs/quotas.html#Task_Queue, > >> > it is stated that we could configure it should be configurable up to > >> > "Stored > >> > Data" (billable) . however from our current GAE admin console and > >> > billing > >> > setting, we found no way to configure it. > >> > > >> > appreciateif anyone can share (partial) answers for these or your > >> > thoughts > >> > > >> > Regards, > >> > Eric ng > >> > > >> > -- > >> > You received this message because you are subscribed to the Google > >> > Groups > >> > "Google App Engine" group. > >> > To post to this group, send email to > [email protected]. > >> > To unsubscribe from this group, send email to > >> > [email protected]<google-appengine%[email protected]> > . > >> > For more options, visit this group at > >> > http://groups.google.com/group/google-appengine?hl=en. > >> > > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "Google App Engine" group. > >> To post to this group, send email to [email protected]. > >> To unsubscribe from this group, send email to > >> [email protected]<google-appengine%[email protected]> > . > >> For more options, visit this group at > >> http://groups.google.com/group/google-appengine?hl=en. > >> > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]<google-appengine%[email protected]> > . > > For more options, visit this group at > > http://groups.google.com/group/google-appengine?hl=en. > > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<google-appengine%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
