I'm not sure that we document or guarantee this anywhere, but we currently seem to be preserving these for 90 days. I don't know how feasible it is to retrieve all 90 days of data via appcfg, though.
On Thu, Mar 11, 2010 at 8:50 PM, Spines <[email protected]> wrote: > Thanks for your help Don. Just one more question (hopefully :)). > > I think just the request logs will be good enough for my purposes, and > I won't actually have to use the diagnostic logs. How much past data > is stored for the request logs? > > On Mar 11, 8:16 am, Don Schwarz <[email protected]> wrote: > > On Wed, Mar 10, 2010 at 8:05 PM, Spines <[email protected]> wrote: > > > Sorry, I don't think I really understood the task queue approach. To > > > ensure no loss of data, would the task have to update in the datastore > > > every time? And the benefit over just doing it directly in the > > > servlet handler would be the faster response time to the user? > > > > Basically, yes. > > > > > I think the memcache solution may be my best bet. I can tolerate some > > > chance of loss of data. I'm wondering how likely loss of data would > > > be? How often does the data get booted out? If I persist from memcache > > > to datastore every 10 seconds would data loss be super rare? > > > > You would have to experiment with this as we don't make any guarantees, > but > > I believe that memcache data generally survives much longer than 10 > seconds > > of inactivity, yes. > > > > On Mar 10, 4:11 pm, Spines <[email protected]> wrote: > > > > > > > > > > Thanks Don, > > > > I thought about the task queue, but that caps at being able to > execute > > > > like 5 tasks per second right? > > > > > > So, as long as the log data doesn't get full before I download it > then > > > > it would be fine? > > > > > > On Mar 10, 3:42 pm, Don Schwarz <[email protected]> wrote: > > > > > > > Yeah, those are diagnostic logs. They effectively go into a ring > > > buffer per > > > > > logging level, so the maximum data stored at any given time is > capped. > > > The > > > > > more you log, the more frequently you would have to download the > logs > > > to > > > > > avoid missing any. You would also be competing with log space with > any > > > > > other log messages generated by your application. > > > > > > > What I would suggest instead is either to increment counters in > > > memcache, > > > > > and flush them to the datastore periodically if you need durability > > > (I'm > > > > > assuming you can tolerate some chance of data loss here). If you > > > cannot > > > > > tolerate any loss of data, then I would suggest enqueueing tasks to > a > > > task > > > > > queue for each request that maintains a summary in memcache and/or > the > > > > > datastore. > > > > > > > On Wed, Mar 10, 2010 at 4:59 PM, Spines <[email protected]> > wrote: > > > > > > I'm talking about the logs that get written when I call > > > > > > Logger.info("something"). > > > > > > > > Basically this is what I'm thinking: I have certain data that > needs > > > to > > > > > > get written very often, but hardly ever needs to be read (stuff > like > > > > > > what users view what pages of my site). The datastore is > optimized > > > > > > for read efficiency. So, I want to output this data to the logs. > I > > > > > > will have an offsite computer download these logs, do > calculations on > > > > > > them, and upload the result of the calculations to the datastore. > > > > > > > > On Mar 10, 2:03 pm, Don Schwarz <[email protected]> wrote: > > > > > > > Are you talking about request logs or diagnostic logs? > Although we > > > > > > conflate > > > > > > > them a bit in both the Admin Console viewer and the appcfg > command, > > > but > > > > > > they > > > > > > > are stored and tracked separately. > > > > > > > > > On Wed, Mar 10, 2010 at 4:01 PM, Spines <[email protected]> > > > wrote: > > > > > > > > Hmm, that is my biggest concern, log reliability. Can > someone > > > from > > > > > > > > Google confirm whether or not I can rely on the logs having > all > > > of the > > > > > > > > log data? Or might certain entries just disappear? > > > > > > > > > > On Mar 10, 1:24 pm, thierry Le conniat <[email protected] > > > > > wrote: > > > > > > > > > Hello, > > > > > > > > > I think google log are stored in file. > > > > > > > > > My experience about log reliability is that when the app is > > > very > > > > > > > > > strong working, not all the log are stored. > > > > > > > > > It's confusing, but i can't not explain it. > > > > > > > > > > > Bye > > > > > > > > > > > On 10 mar, 22:04, Spines <[email protected]> wrote: > > > > > > > > > > > > Where does Google store the logs when you do a Logging > > > statement? > > > > > > > > > > Logging statements seem to be pretty fast, so it doesn't > seem > > > like > > > > > > > > > > they are stored in the datastore. > > > > > > > > > > > > How reliable are the logs? If I do a logging statement > and it > > > > > > > > > > succeeds, is it pretty much guaranteed that it will show > up > > > in the > > > > > > > > > > logs? > > > > > > > > > > > > How much past history of logs is stored? > > > > > > > > > > > > The reason I'm interested in this is because I'm making a > > > question > > > > > > and > > > > > > > > > > answer website, and I want to keep track of views by each > > > unique > > > > > > > > > > logged in user to each question, and display the view > count > > > on the > > > > > > > > > > question page. So if 10 different users visit the > question > > > page 100 > > > > > > > > > > times, it still only counts as 10 unique views. > > > > > > > > > > > > I have an offsite computer that does background > processing > > > for my > > > > > > app. > > > > > > > > > > I'm planning to have this offsite computer download the > logs > > > about > > > > > > > > > > every 30 minutes, and calculate what the view count > should be > > > for > > > > > > each > > > > > > > > > > question based off of the logs. By doing this, I don't > have > > > to > > > > > > create > > > > > > > > > > a datastore entity for each different question each user > > > views. > > > > > > > > > > > > What do you guys think? Does anyone see any problems with > > > this? > > > > > > > > > > -- > > > > > > > > You received this message because you are subscribed to the > > > Google > > > > > > Groups > > > > > > > > "Google App Engine for Java" group. > > > > > > > > To post to this group, send email to > > > > > > > > [email protected]. > > > > > > > > To unsubscribe from this group, send email to > > > > > > > > [email protected]<google-appengine-java%[email protected]><google-appengine-java%2B > [email protected]><google-appengine-java%2B > > > [email protected]><google-appengine-java%2B > > > > > > [email protected]> > > > > > > > > . > > > > > > > > For more options, visit this group at > > > > > > > >http://groups.google.com/group/google-appengine-java?hl=en. > > > > > > > > -- > > > > > > You received this message because you are subscribed to the > Google > > > Groups > > > > > > "Google App Engine for Java" group. > > > > > > To post to this group, send email to > > > > > > [email protected]. > > > > > > To unsubscribe from this group, send email to > > > > > > [email protected]<google-appengine-java%[email protected]><google-appengine-java%2B > [email protected]><google-appengine-java%2B > > > [email protected]> > > > > > > . > > > > > > For more options, visit this group at > > > > > >http://groups.google.com/group/google-appengine-java?hl=en. > > > > > -- > > > You received this message because you are subscribed to the Google > Groups > > > "Google App Engine for Java" group. > > > To post to this group, send email to > > > [email protected]. > > > To unsubscribe from this group, send email to > > > [email protected]<google-appengine-java%[email protected]><google-appengine-java%2B > [email protected]> > > > . > > > For more options, visit this group at > > >http://groups.google.com/group/google-appengine-java?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to > [email protected]. > To unsubscribe from this group, send email to > [email protected]<google-appengine-java%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
