On Wed, Mar 10, 2010 at 8:05 PM, Spines <[email protected]> wrote: > Sorry, I don't think I really understood the task queue approach. To > ensure no loss of data, would the task have to update in the datastore > every time? And the benefit over just doing it directly in the > servlet handler would be the faster response time to the user? > > Basically, yes.
> I think the memcache solution may be my best bet. I can tolerate some > chance of loss of data. I'm wondering how likely loss of data would > be? How often does the data get booted out? If I persist from memcache > to datastore every 10 seconds would data loss be super rare? > > You would have to experiment with this as we don't make any guarantees, but I believe that memcache data generally survives much longer than 10 seconds of inactivity, yes. On Mar 10, 4:11 pm, Spines <[email protected]> wrote: > > Thanks Don, > > I thought about the task queue, but that caps at being able to execute > > like 5 tasks per second right? > > > > So, as long as the log data doesn't get full before I download it then > > it would be fine? > > > > On Mar 10, 3:42 pm, Don Schwarz <[email protected]> wrote: > > > > > > > > > Yeah, those are diagnostic logs. They effectively go into a ring > buffer per > > > logging level, so the maximum data stored at any given time is capped. > The > > > more you log, the more frequently you would have to download the logs > to > > > avoid missing any. You would also be competing with log space with any > > > other log messages generated by your application. > > > > > What I would suggest instead is either to increment counters in > memcache, > > > and flush them to the datastore periodically if you need durability > (I'm > > > assuming you can tolerate some chance of data loss here). If you > cannot > > > tolerate any loss of data, then I would suggest enqueueing tasks to a > task > > > queue for each request that maintains a summary in memcache and/or the > > > datastore. > > > > > On Wed, Mar 10, 2010 at 4:59 PM, Spines <[email protected]> wrote: > > > > I'm talking about the logs that get written when I call > > > > Logger.info("something"). > > > > > > Basically this is what I'm thinking: I have certain data that needs > to > > > > get written very often, but hardly ever needs to be read (stuff like > > > > what users view what pages of my site). The datastore is optimized > > > > for read efficiency. So, I want to output this data to the logs. I > > > > will have an offsite computer download these logs, do calculations on > > > > them, and upload the result of the calculations to the datastore. > > > > > > On Mar 10, 2:03 pm, Don Schwarz <[email protected]> wrote: > > > > > Are you talking about request logs or diagnostic logs? Although we > > > > conflate > > > > > them a bit in both the Admin Console viewer and the appcfg command, > but > > > > they > > > > > are stored and tracked separately. > > > > > > > On Wed, Mar 10, 2010 at 4:01 PM, Spines <[email protected]> > wrote: > > > > > > Hmm, that is my biggest concern, log reliability. Can someone > from > > > > > > Google confirm whether or not I can rely on the logs having all > of the > > > > > > log data? Or might certain entries just disappear? > > > > > > > > On Mar 10, 1:24 pm, thierry Le conniat <[email protected]> > wrote: > > > > > > > Hello, > > > > > > > I think google log are stored in file. > > > > > > > My experience about log reliability is that when the app is > very > > > > > > > strong working, not all the log are stored. > > > > > > > It's confusing, but i can't not explain it. > > > > > > > > > Bye > > > > > > > > > On 10 mar, 22:04, Spines <[email protected]> wrote: > > > > > > > > > > Where does Google store the logs when you do a Logging > statement? > > > > > > > > Logging statements seem to be pretty fast, so it doesn't seem > like > > > > > > > > they are stored in the datastore. > > > > > > > > > > How reliable are the logs? If I do a logging statement and it > > > > > > > > succeeds, is it pretty much guaranteed that it will show up > in the > > > > > > > > logs? > > > > > > > > > > How much past history of logs is stored? > > > > > > > > > > The reason I'm interested in this is because I'm making a > question > > > > and > > > > > > > > answer website, and I want to keep track of views by each > unique > > > > > > > > logged in user to each question, and display the view count > on the > > > > > > > > question page. So if 10 different users visit the question > page 100 > > > > > > > > times, it still only counts as 10 unique views. > > > > > > > > > > I have an offsite computer that does background processing > for my > > > > app. > > > > > > > > I'm planning to have this offsite computer download the logs > about > > > > > > > > every 30 minutes, and calculate what the view count should be > for > > > > each > > > > > > > > question based off of the logs. By doing this, I don't have > to > > > > create > > > > > > > > a datastore entity for each different question each user > views. > > > > > > > > > > What do you guys think? Does anyone see any problems with > this? > > > > > > > > -- > > > > > > You received this message because you are subscribed to the > Google > > > > Groups > > > > > > "Google App Engine for Java" group. > > > > > > To post to this group, send email to > > > > > > [email protected]. > > > > > > To unsubscribe from this group, send email to > > > > > > [email protected]<google-appengine-java%[email protected]><google-appengine-java%2B > [email protected]><google-appengine-java%2B > > > > [email protected]> > > > > > > . > > > > > > For more options, visit this group at > > > > > >http://groups.google.com/group/google-appengine-java?hl=en. > > > > > > -- > > > > You received this message because you are subscribed to the Google > Groups > > > > "Google App Engine for Java" group. > > > > To post to this group, send email to > > > > [email protected]. > > > > To unsubscribe from this group, send email to > > > > [email protected]<google-appengine-java%[email protected]><google-appengine-java%2B > [email protected]> > > > > . > > > > For more options, visit this group at > > > >http://groups.google.com/group/google-appengine-java?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to > [email protected]. > To unsubscribe from this group, send email to > [email protected]<google-appengine-java%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
