The high-level summary and complete transcript of the April 7th edition of the IRC office hours is pasted below. Please join us on the first and third Wednesday of every month in the #appengine channel on irc.freenode.net. On the first Wednesday, we meet in the channel from 7:00-8:00 p.m. PST (evening hours), and on the third Wednesday (e.g. TOMORROW, 4/21), we're available from 9:00-10:00 a.m. PST (morning hours).
- Jason --SUMMARY----------------------------------------------------------- - Discussion of existing App Engine-compatible OpenID libraries and forthcoming built-in support for OpenID and OAuth [7:03-7:09] - Remember, back up your code or use a secure version control system -- we cannot help you recover your code in the event of corruption or theft, so backup early and often. Also, see http://stackoverflow.com/questions/2479087/can-i-restore-my-source-code-that-has-been-uploaded-into. [7:11-7:16] - Q: Is it feasible to push every write to the task queue instead of making the user wait for the write to complete inside a handler? A: While this should work, there are two things to keep in mind: 1) the number of task insertions is currently limited to 1M per day -- if you make more than one million writes each day, this solution will fail. 2) Tasks don't necessarily run in the order they were inserted; e.g. if a task fails and has to be re-tried, it's possible that a newer task will be processed ahead of it. Thus, you'll want to version your entities so you don't mistakenly overwrite new data with old data. [7:18-7:19, 7:21, 7:58-7:59] - Discussion on IDE/editor preference [7:21, 7:23, 7:24] - Discussion on paging using cursors and the inherent challenges of paging backwards when entities are consistently being inserted [7:24-7:30] - Q: Should I be concerned if I see a red or cold caution sign in the Admin Console indicating that CPU usage for a particular handler is high? A: In general, no. These symbols are present to show you which handlers are using the most CPU and to help point out which handlers you should optimize, but the quotas page is the definitive source on resource usage. Keep in mind that enabling billing for your application automatically boosts a lot of your quotas, regardless of whether you actually exceed any of the billable quotas. [7:31-7:35] - Discussion on SSL support, ETAs on SSL for custom domains, and a potential SSL bug affecting non-default versions of an app. [7:39-7:40] - Q: Are sessions supported in Python? A: Sessions aren't supported directly by the webapp framework, but you can build your own implementation fairly easily using cookies and memcache/datastore. [7:49, 7:51-7:53] - Q: Are keys-only queries a good way to get a simple count on the number of entities matching a query? A: Depending on your queries, this may be the only effective method. In general, we prefer pre- computation when possible -- if you have a limited number of known queries, you can just increment a counter when a new entity is written or updated. For systems where the set of possible queries is large and/ or unknown, then performing a keys-only query is a reasonable solution. Be sure to cache the result so you don't have to continually execute this query for every request. Also, remember the datastore statistics API is available if you need a straight count of every entity in your datastore or of every entity of a particular kind. [7:50-7:52, 7:54-7:59, 8:03] --FULL TRANSCRIPT----------------------------------------------------------- [7:00pm] apijason_google: Good evening (or morning as the case may be)! I and a few other App Engine engineers will be here for the next hour to answer your questions, so fire away. [7:01pm] chilts: sweet, didn't realise it was now [7:01pm] chilts: (not that I have a question) [7:02pm] apijason_google: Well, just in case you have one later. [7:03pm] chilts: apijason_google: any idea if Google are working on patching the python-openid library to make it Google Apps compatible (ie. does the host-meta discovery method)? [7:04pm] chilts: it would be good so Python people on AppEngine can start doing Marketplace apps more easily for _all_ Google Apps Enterprise customers [7:04pm] AreEmmKay: I saw a commit on AEOID about that [7:04pm] • chilts goes to check it out [7:05pm] AreEmmKay: http://github.com/Arachnid/aeoid "Add preliminary support for Google Apps openid " [7:05pm] chilts: I did see Nick's post about putting an /openid page on your site, but that would be much better [7:05pm] chilts: ah, March 13th [7:05pm] chilts: cool, thanks AreEmmKay [7:05pm] AreEmmKay: np [7:06pm] apijason_google: I'm not positive on the implementation details, but OpenID enhancements will be added in a future release. I don't know if you caught the "Built-in support for OAuth & OpenID" item in our latest product roadmap. [7:06pm] chilts: saw that on the roadmap yeah [7:06pm] chilts: can't wait [7:06pm] molson: Trying to view a specific class in my datastore via the datastore viewer generates an "Error: Server Error" error page from app engine every time. Is there a known bug for viewing the datastore? [7:07pm] apijason_google: chilts: That will definitely make Apps Marketplace development a lot easier, I think. [7:07pm] AreEmmKay: Does "built in" oath mean easy authentication with other services, or easily exposing services via oauth? [7:07pm] chilts: yeah, it'll be lovely [7:07pm] apijason_google: molson: Hmm, interesting. It's not widespread as far as I know. What is your application ID and the name of the kind you're trying to view? [7:08pm] molson: apijason_google: my application id is 'seismocs' and the entity kind is SensorReadings. Attempting to execute "SELECT * FROM SensorReading" will crash the datastore viewer. [7:09pm] apijason_google: AreEmmKay: Good question. I believe it will definitely help in authenticating signed requests, but it may also be relevant if you want to expose your own OAuth services -- not as sure on this part, but it will be a nice enhancement. [7:09pm] apijason_google: molson: I'll take a look. [7:09pm] molson: apijason_google: Thanks! [7:10pm] apijason_google: Welcome Wesley! [7:10pm] AreEmmKay: There's been some chatter on the mailing lists about emails going out even when send_mail raises an exception-- have you guys identified the issue, and is it something you plan to fix? [7:10pm] AreEmmKay: ie, will we ever be able to trust that when send_mail fails, no mail goes out [7:11pm] Wesley_Google: thanks! glad to be here on App Engine's 2nd birthday!! http://bit.ly/9cFBsv [7:11pm] xdcdx: I recently got my laptop stolen, and lost the code for an appengine app I was working on. I acknowledge I should have kept backups, or a version control system, and I have learned my lesson the hard way. Is there any possibility anyone at Google can help me recover my code from the appengine servers? I Googled extensively and found http://stackoverflow.com/questions/2479087/can-i-restore-my-source-code-that-has-been-uploaded-into [7:11pm] molson: Happy birthday, App Engine [7:11pm] apijason_google: molson: Yep, I can confirm the issue. Can you file a new production issue in the issue tracker, and I'll take a closer look when I get back into the office tomorrow. http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue [7:12pm] xdcdx: but I am afraid I didn't have the remote-api and deferred handlers in place. My app is programmed in python [7:12pm] apijason_google: molson: When did you start seeing this behavior? [7:12pm] molson: apijason_google: no problem, will do. I started seeing the problem pretty much as soon as I started creating these objects. [7:12pm] AreEmmKay: I'd offer to buy you guys a birthday drink, but google is made of money [7:12pm] apijason_google: AreEmmKay: Perhaps, but I'm not. [7:12pm] molson: apijason_google: FWIW, the problem does not occur locally, or at least not for records with small file sizes [7:13pm] apijason_google: xdcdx: Can you ping me privately by email? [email protected] [7:14pm] Wesley_Google: xdcdx> what is described in the post is the newest (and perhaps) easiest way for you to get your code. we can't get it for you unfortunately. can you imagine the privacy issues?!? [7:14pm] xdcdx: apijason_google: sure, going to send you an email right now, thank you in advance [7:15pm] Wesley_Google: AreEmmKay: you mean Google AdWords is made of money... App Engine has a generous free quota! [7:16pm] xdcdx: Wesley_Google: I acnowledge that, the problem is that I didn't have the remote-api and deferred handlers in place, which seem to be a must for using that method. I can put the handlers now, but on a new code-less version of my application of course [7:17pm] Wesley_Google: jason may have a way to help you, but most of us don't have access. again, it's for privacy reasons. [7:17pm] amalgameate: hey, is this a room for developers to freely discuss GAE? [7:17pm] Wesley_Google: amalgameate> yes, it is [7:18pm] amalgameate: sweeet, i've been looking for a place to talk with other devs foreverr [7:18pm] apijason_google: amalgameate: [7:18pm] amalgameate: do you know if it's feasible to add every single write to the task queue instead of making the user wait? [7:18pm] amalgameate: [7:19pm] apijason_google: amalgameate: That depends on the number of writes you do. Currently there's an upper limit to the number of tasks you can add to your queues per day, but there's no upper limit on the number of writes. [7:19pm] amalgameate: mm gotcha [7:19pm] amalgameate: are you guys google staff? [7:20pm] xdcdx: apijason_google: mail sent, thanks for your interest into this problem caused by my lack of backup prevision [7:20pm] amalgameate: err all the *_google sn's? [7:20pm] apijason_google: amalgameate: Yes. We hold two one-hour office hour sessions per month, and you just happened to come at a good time. [7:21pm] Wesley_Google: datastore writes aren't *that* bad... but i don't see anything wrong with your proposal [7:21pm] amalgameate: ah ok [7:21pm] Wooble: amalgameate: this is the bi-weekly developer chat (see the topic) [7:21pm] amalgameate: got it [7:21pm] apijason_google: Hmm, I guess we're in PDT now though. I'll have to bug Nick. [7:21pm] Wesley_Google: if we have "_Google" at the end of our names, then we are indeed Googlers [7:21pm] Sthebig: Not exactly strictly development-related but I'll ask anyways: What IDEs for win/nix/mac do the Google staff prefer for developing for app engine in python? I'm just using notepad++ at the moment, would prefer something more comprehensive. [7:22pm] Wesley_Google: amalgameate> just keep in mind what jason suggested... you do have a cap on TQs but not DS writes [7:22pm] Wooble: the channel's pretty good the rest of the time too, if with fewer google people here... [7:22pm] amalgameate: ok got it [7:23pm] xdcdx: Wesley_Google: I understand how Googlers don't have general access to user uploaded data for privacy concerns. It's quite understandable. [7:23pm] Wesley_Google: Sthebig> some people prefer plain text, others an IDE, so it depends what is your comfort level... [7:23pm] apijason_google: Sthebig: I personally use TextMate on my Mac. Wesley might have a stronger opinion though -- he's a Python master. [7:23pm] luddep: so how did the meetup go yesterday? sadly couldn't make it. [7:23pm] Wesley_Google: luddep: the meeting went GREAT! the only bad news is that we didn't notify the auto-recording folks so it wasn't taped [7:24pm] apijason_google: luddep: You missed a good one. In honor of App Engine's birthday, we actually had food. [7:24pm] amalgameate: I just finished watching all of the Google I/O talks from 08/09 where Brett and others recommended doing paging by property sort. Is this still the preferred method even over using cursors? [7:24pm] Wesley_Google: next month will be a really good talk as well (but can't say anything yet as it hasn't been confirmed) [7:24pm] xdcdx: Sthebig: if that helps I used to use TextMate too, it's a very confortable editor, and you won't get all the bloat of a full fledged IDE like Eclipse. Although people says Eclipse is very nice actually [7:24pm] luddep: aw! good to hear though [7:24pm] apijason_google: amalgameate: No, use cursors. [7:24pm] ikai_google: Hi guys, just arrived to this party. What's shakin'? [7:24pm] amalgameate: apijason_google: mmk that's what i thought [7:24pm] apijason_google: Cursors were added fairly recently and are the preferred method of paging. [7:25pm] ikai_google: Note that cursors only go forwards, not backwards [7:25pm] Sthebig: Well I'll put it this way, Wesley_Google. I'm a bit of a newbie at python; jumping in head first. [7:25pm] apijason_google: We're going to update our online docs to reflect that before this upcoming I/O. [7:25pm] amalgameate: haha ok [7:26pm] Wooble: ikai_google: they go both ways if you remember to save the previous ones [7:26pm] ikai_google: wooble: not if you insert things [7:26pm] Wooble: bah. [7:27pm] Wesley_Google: PRIVMSG ikai_google: hey there [7:27pm] ikai_google: Wooble: Yeah, Wesley and I have been trying to cook up how to get pagination working similar to how folks would do it in mysql [7:27pm] ikai_google: Wooble: It isn't a 1:1 match [7:27pm] Wooble: inserting messes up the pagination anyway. [7:27pm] ikai_google: true that. [7:28pm] amalgameate: ok so for now, the best way to do pagination with cursors is to just save the previous cursor to go backwards? [7:28pm] Wesley_Google: Sthbig: what tool have you been coding with for your other programming languages? [7:28pm] Wooble: well, unless you have a pre-blogging "oldest items first" mindset, I suppose. [7:28pm] molson: apijason_google: Okay, the issue is in: http://code.google.com/p/googleappengine/issues/detail?id=3061 - I couldn't find the log files I wanted using the online tool, so I may comb through the downloads later and try to update the record. [7:28pm] amalgameate: and deal with the fact that if data is inserted you may lose the last entry? [7:29pm] apijason_google: molson: Thanks. We'll try to get that sorted out for you as soon as possible. [7:29pm] ikai_google: amalgameate: Yeah, but there are a lot of weird edge cases here. You could also precalculate cursors both ways. There are a lot of solutions [7:29pm] ikai_google: amalgameate: You won't "lose" an entry [7:29pm] molson: apijason_google: Great; thanks for the help. [7:29pm] ikai_google: amalgameate: A cursor points to a place where the query stopped. So when you query, you query for cursor location + N [7:29pm] ikai_google: where N is your page size [7:29pm] ikai_google: so if you insert something [7:29pm] ikai_google: you "miss" the last element last used in that query. if that makes sense [7:30pm] amalgameate: ikai_google: yea ok that makes sense [7:31pm] mrleroylee: Probably a common question, but I am wondering if there is any documentation on the range of numbers that is under quota in the "average cpu" column in the admin. I get warnings often but the numbers don't mean anything to me. [7:31pm] ikai_google: mrleroylee: They don't mean anything individually. You should only be concerned if you get them frequently AND are hitting quota [7:32pm] Wesley_Google: mrleroylee: the avg CPU or QPS is actually pretty low. but if your app is very low latency, you can scale as high as our hardware allows... [7:33pm] apijason_google: mrleroylee: The numbers are actually a holdover from the period before billing was available. Since there was no way for devs to pay for additional CPU time, they were there to help caution devs as to which pages were using the most CPU on average. But like Ikai says, the overall quota is more important. [7:34pm] svm_invictvs: Heya [7:34pm] svm_invictvs: IT's Google day! [7:34pm] mrleroylee: ikai_google: Ok, I have never approached my overall quota. I was just worried that the warnings might indicate running over my CPU per minute. Thanks. [7:34pm] Wesley_Google: you'll *know* when you go over quota... the graphs go red, and we ask you to turn on billing [7:35pm] svm_invictvs: hey ikai_google our app handled quite a load w/ o any problems and no quota denials. [7:35pm] apijason_google: svm_invictvs: Welcome! I thought every day was Google day! Er, maybe not. But it is App Engine's birthday, so there's that. [7:35pm] Wesley_Google: luddep asked earlier about the meetup... if anyone else is in the San Francisco Bay Area or Silicon Valley, we invite you to join and come to the meetings http://www.meetup.com/appengine/ [7:35pm] ikai_google: svm_invictvs: Awesome. That's ... awesome [7:35pm] svm_invictvs: [7:36pm] svm_invictvs: ikai_google: Yeah, usage is down though b/c the client is buggy and we've had bad drop-off but it's been performing well. [7:36pm] amalgameate: Am i correct in saying that a parent entity doesnt actually retrieve the properties of child entities unless explicitly called? [7:36pm] ikai_google: amalgameate: Correct. [7:37pm] apijason_google: Yes [7:37pm] ikai_google: svm_invictvs: Can you talk about your app? [7:37pm] svm_invictvs: ikai_google: I posted it on the mailing lists. It's an iPhone game. [7:37pm] svm_invictvs: ikai_google: iDuel Online. [7:37pm] mrleroylee: Some one mentioned a "timeline". Where would that be located online? [7:37pm] yodler12: Hey trying to understand where your going with 'mapping operations across datasets' in roadmap. Is this a way to write to entities without incurring serialization/deserialization overhead? [7:38pm] ikai_google: svm_invictvs: Oh yeah!! Saw that. And I asked you when you were making an Android version [7:38pm] ikai_google: yodler12: map reduce [7:38pm] Wesley_Google: mrleroylee: what timeline are you talking about? our public roadmap? [7:38pm] svm_invictvs: It's all GAE/J, it's got a few thousand accounts, and has served like 6,000,000 advertising impressions since it's launch two weeks ago. [7:38pm] luddep: some days I'm really glad I use app engine, 4x traffic jump in a day? No biggie. [7:38pm] xdcdx: apijason_google: excuse me, can you confirm you got my email message? No hurries, just have a look at it when you have time [7:38pm] apijason_google: mrleroylee: If you're looking for the roadmap, it's located at http://code.google.com/appengine/docs/roadmap.html. [7:38pm] apijason_google: xdcdx: Yes, I received it. [7:38pm] svm_invictvs: ikai_google: Those are hte vital stats on it ^^ [7:38pm] ikai_google: yodler12: Everything you do will have overhead from serialization/deserialization. [7:39pm] xdcdx: thank you [7:39pm] mrleroylee: apijason_google: thanks. [7:39pm] ikai_google: luddep: You'll be even gladder the days you hit 40x or 400x (one can hope) [7:39pm] amalgameate: can you guys give an ETA on SSL support? [7:39pm] yodler12: got it, thanks. Follow up: how best to do 'Relation index entities' when you need to add to them over time? [7:39pm] ikai_google: amalgameate: Wish we could, but in general we don't give ETAs [7:39pm] ikai_google: amagameate: SSL is supported - just not for custom domains. [7:40pm] apijason_google: amalgameate: No ETA. But it recently made the roadmap for the first time, so that indicates that it's something the team is dedicated to delivering sooner rather than later. [7:40pm] svm_invictvs: Oh, yeah. I waanted to ask about SSL. Are you guys aware that if you access your app at its default URL all works fine...but if you access it a specific version the signing fails? [7:40pm] xdcdx: can you give an ETA on when Apple will release new Macbooks Pros? [7:40pm] amalgameate: right right [7:40pm] luddep: ikai_google: hehe, yeah. [7:40pm] amalgameate: that's great SSL is huge for us [7:40pm] Wesley_Google: someone wanted private SSL for their App Engine birthday wish [7:40pm] svm_invictvs: Yeah, private SSL would be nice. [7:40pm] ikai_google: xdcdx: I've heard a refresh is coming end of this month [7:40pm] apijason_google: Wesley: If you say the wish, it won't come true. [7:41pm] luddep: so, what kind of apps are people working on, if I may ask? [7:41pm] svm_invictvs: I've got a game in appengine [7:42pm] xdcdx: yep, I read that too. They are quite busy with the iPad thing now, hehe [7:42pm] svm_invictvs: Was the other SSL issue brought up, btw? [7:42pm] svm_invictvs: Is that a known issue? [7:42pm] xdcdx: Mine is a Google Calendar frontend for concerts around my city [7:42pm] apijason_google: svm_invictvs: I haven't come across that one before. I'll verify and file a bug. [7:43pm] ikai_google: xdcdx: Link? And what city? [7:43pm] apijason_google: svm_invictvs: Have you filed an issue in the tracker? [7:43pm] Wesley_Google: xdcdx: keep an eye out at WWDC isn't there going to be an iPhone announcement soon too? [7:43pm] luddep: this is my shameless plug: http://www.observerapp.com/ [7:43pm] luddep: svm_invictvs: cool! what was the name? (cant scroll up in irssi on a mac) [7:43pm] apijason_google: Wesley_Google: Tomorrow -- iPhone OS 4.0 announcement. [7:43pm] svm_invictvs: luddep: iDuel It's an iPhone game [7:44pm] amalgameate: luddep: will it be for free? [7:44pm] xdcdx: well, not exactly a GCal frontend, it's a static calendar that only the content manager can modify, but it uses an internal Google Calendar to store events www.musicual.com [7:44pm] mrleroylee: I would like to use some pretty urls in my app. Is that doable with the standard python webapp implementation or do I need to install a framework like Django? [7:44pm] xdcdx: city is Valencia (Spain) [7:44pm] svm_invictvs: apijason_google: No, I havne't. [7:44pm] xdcdx: 4:44 am right here [7:44pm] ikai_google: mrleroylee: Yes, you can do it with regular expressions [7:44pm] luddep: amalgameate: hehe, probably not, unless I'm able to get some sort of ad deal for free accounts. [7:44pm] luddep: like, fusionads.net. [7:44pm] apijason_google: svm_invictvs: Go ahead and start there, and I'll verify independently. [7:44pm] amalgameate: have you heard of mixpanel? [7:44pm] svm_invictvs: I dont' see any defects listed in the issue tracker. [7:45pm] ikai_google: xdcdx: Ah, I love Chrome's "translate" extension. So awesome [7:45pm] svm_invictvs: Something about hte certificate isn't issued for *.*.appspot.com just *.appspot.com [7:45pm] luddep: amalgameate: yup [7:45pm] Wesley_Google: myleroylee: yes, you can do it... as ikai said, regexes in your app.yaml or controller. it's an xml file for java [7:45pm] mrleroylee: ikai_google: is there documentation about that? I would not be sure how to start. [7:45pm] ikai_google: luddep: "Is it ready yet? Well not quite yet." hahaha nice. [7:46pm] ikai_google: mrleroylee: You'd match something like /* (this *should*) work, then just process it in your handler [7:46pm] xdcdx: ikai_google: yeah, works wonders for german ebays listing. Do you know if there is a way to invoce the translate bar when Chrome doesn't automatically detect a foreign language? [7:46pm] ikai_google: xdcdx: No I NEVER see the Chrome guys. When I do, I forgot to bug em about stuff [7:46pm] xdcdx: hahah [7:46pm] luddep: ikai_google: hehe, well, it is getting pretty damn ready the actual front-end is in cappuccino, a quick teaser: http://dl.dropbox.com/u/24582/sites.mov [7:46pm] xdcdx: that would be handy, it misses some foreign pages sometimes [7:46pm] apijason_google: svm_invictvs: That might not be something we can fix right away, but it will be good to have an open issue for it. Surprisingly, that's the first time it's come up, at least as far as I'm aware. [7:47pm] Wesley_Google: myleroylee: start here where you can see an example of an app.yaml file [7:47pm] Wesley_Google: http://code.google.com/appengine/docs/python/gettingstarted/helloworld.html [7:47pm] xdcdx: and we need Gears for Chrome on Snow Leopard too... can't live without Offline Gmail [7:47pm] apijason_google: luddep: re Cappuccino: nice! I'm wearing my 280 North shirt today, as a matter of fact. [7:47pm] svm_invictvs: apijason_google: Yeah, it's not just a Firefox thing the iPHone throws an error too. It's really bad for me because most of the interaction with appengine right now is just through my app's API so I can't have versioning in the clients. At least not using the URL. [7:48pm] svm_invictvs: apijason_google: Would that be appropriate to file as a "production issue?" [7:48pm] apijason_google: svm_invictvs: No, just file it as a general defect. [7:48pm] svm_invictvs: Okay [7:48pm] luddep: apijason_google: oh, nice! I should actaully be at their weekly meetup right now, but, couldn't make it today. [7:49pm] xdcdx: back to my app, something broke with the Google Calendar API and we cannot longer edit or delete events, just add them. I think it's just a matter of updatind GData... but then came the laptop theft and the code loss [7:49pm] apijason_google: luddep: You wanted to spend it here with us instead. I'll let them know that the next time I see them. [7:49pm] apijason_google: (jk) [7:49pm] mrleroylee: Wesley_Google: I understand using the regex to route the request, I just don't understand how to retrieve the info from the url. [7:49pm] Brandon\: Any plans to natively support sessions in GAE? [7:50pm] amalgameate: Is there an advantage to using the Gdatatypes like PhoneNumberProperty and EmailProperty vs just a StringProperty? [7:50pm] svm_invictvs: Is a keys-only query appropriate to use to count entities matching a certain criteria? [7:50pm] apijason_google: amalgameate: No, not really. It's useful if you want to keep your data structured, but I usually use StringProperty for email addresses and other data. [7:50pm] Wesley_Google: mrleroylee: what info? the stuff your users entered/POSTed? [7:51pm] svm_invictvs: Let's say, I want to know how many entities have a property between 0 and 42, could I just do a keys only query and count the keys? Is that reasonable or will my performance tank doing that? [7:51pm] ikai_google: Brandon: What do you mean? There already exists session support. If you are talking about stateful servlets, then no [7:51pm] Brandon\: There is session support already? [7:51pm] Brandon\: Hmmm [7:51pm] luddep: apijason_google: haha [7:51pm] Wesley_Google: myleroylee: just use self.request.get(VARIABLE) to get all the stuff from the user [7:51pm] svm_invictvs: 'cause keys only just looks at the index, correct, and doesn't fetch the whole datastore, right? [7:51pm] Brandon\: Haven't been able to find that in the API [7:51pm] ikai_google: Brandon: are you writing Python or Java? [7:51pm] Brandon\: Python [7:52pm] apijason_google: svm_invictvs: Yes. Keeping it keys-only will definitely save on resource usage. The best method is to do the count at write time, but this isn't feasible for ad hoc queries. [7:52pm] Wesley_Google: mrleroylee: or if you want the actual URL itself, then try self.request.uri [7:52pm] Brandon\: And yes, when I'm referring to sessions, I'm referring to ways in which data is kept between page views for server and client. [7:52pm] Brandon\: Which can be any number of methods. [7:52pm] mrleroylee: Wesley_Google: by stuff I mean the data in the url... /post/314... I would like to pull the post id from the url [7:52pm] ikai_google: Brandon: Maybe I'm incorrect then. But sessions really are just a thin layer over Memcache [7:53pm] ikai_google: Brandon: Set a cookie, map the cookie to Memcache, get from memcache. Wesley_Google: Does webapp have any other session support? [7:53pm] amalgameate: Do you guys have any pointers on how to do full- text search? [7:53pm] xdcdx: mrleroylee: I read somewhere that you can use the '#' character in an url and read/write characters after that one, kinda like gmail does [7:54pm] chilts: xdcdx: they're called fragments and lots of sites use it when they want to do Ajax [7:54pm] amalgameate: besides the poor man's search [7:54pm] xdcdx: that was it [7:54pm] svm_invictvs: apijason_google: I tried keeping a count for that sort of thing but trying to actually use that count resulted in deadline exceptioons left and right. [7:55pm] ikai_google: amalgameate: Not at the moment. Check out the "poor man's search" or http://www.google.com/cse/ [7:55pm] amalgameate: thanks [7:55pm] Wesley_Google: mrleroylee: self.request.uri is your friend. here is a more complex example: http://stackoverflow.com/questions/2004691/regex-match-of-hexdigest-in-google-app-engine-webapp-wsgiapplication [7:55pm] ikai_google: xdcdx: No, Gmail does that to support AJAX history [7:55pm] svm_invictvs: It was just too much to count and I ended up with something like 70,000 shards [7:55pm] svm_invictvs: [7:55pm] apijason_google: svm_invictvs: Interesting. Depending on the amount of new writes, you would have to shard your counter. Otherwise, you'll see a lot of write contention which usually results in the DeadlineExceededExceptions like you say. [7:56pm] apijason_google: Ah [7:56pm] apijason_google: I see. [7:56pm] svm_invictvs: Yeah, the sharding just ended crapping all over itself. [7:56pm] svm_invictvs: I maybe used too many shards. [7:56pm] apijason_google: svm_invictvs: You really need all those shards? [7:56pm] svm_invictvs: *shrug* [7:56pm] svm_invictvs: no. [7:56pm] svm_invictvs: That's what it resulted in, but I probably did it wrong. [7:57pm] svm_invictvs: I was trying to keep a count in such a way that given a range of two numbers I could count the number of entities between those two numbers. [7:57pm] apijason_google: svm_invictvs: In general, you should count on not being able to write to a single entity/entity group more than once per second. So depending on your level of write traffic, you can generally figure how many shards you'll need. [7:57pm] xdcdx: ikai_google: you mean correct back button behaviour? I thought it did, I remember reading something about it in gmail's blog a while ago [7:57pm] mrleroylee: Wesley_Google: thanks [7:57pm] Wooble: mrleroylee: if you;re using webapp, you can set up capture groups in your regexes and they'll be passed to your handlers as parameters [7:58pm] svm_invictvs: apijason_google: Yeah,I had counter shard that had a count and then the actual rank of the object. SO then I'd query for shards that were within the ranges I wanted and then summed up the counts. [7:58pm] taaz: speaking of shards... anyone know of a good way to reduce shards for a counter in some way so that the count is always accurate during the shard removal process? [7:58pm] ikai_google: xdcdx: Yeah, it does that. But back button behavior is a mess. I've custom rolled enough of my own code to not want to go there. [7:58pm] lurkdev_: hmm I thought sharded counters were falliing out of favor to task queue updates. Guess you can use both together [7:58pm] ikai_google: xdcdx: Much better using really simple history or GWT [7:58pm] ikai_google: xdcdx: The reason I rolled my own was because I was inside an OpenSocial gadget container ... in a different domain (needed hashtag for cross-domain IPC) [7:59pm] svm_invictvs: apijason_google: The problem is, for each rank, I could have a few shards and ended up having n * rank shards which resulted in a bunch of shards that were too much to count. [7:59pm] Wooble: mrleroylee: there's an example at http://code.google.com/appengine/docs/python/tools/webapp/running.html [7:59pm] Wesley_Google: mrleroylee: sure, no problem! [7:59pm] apijason_google: lurkdev_: Again, you have the problem where the number of new tasks per day is limited, while the general number of writes to the datastore isn't. But task queues are a fine solution otherwise. [8:00pm] lurkdev_: i see [8:00pm] mrleroylee: thanks [8:01pm] amalgameate: Do you guys know why I would be getting 500 Internal Errors when using bulkloader.py to backup blobs? [8:01pm] Wesley_Google: mrleroylee: the page that wooble sent it also helpful as another medium complexity example [8:02pm] amalgameate: bulkloader is working for all my kinds except for my photos entity [8:02pm] svm_invictvs: apijason_google: Make sense? [8:03pm] Wesley_Google: amagameate: not sure about the bulkloader issue with the 500s. if this is happening live, file a production issue at http://code.google.com/p/googleappengine/issues/list [8:03pm] yodler12: how much CPU overhead in serving photos from the datastore vs. putting them onto S3? Am fearful of when I have more photos [8:03pm] apijason_google: svm_invictvs: Yeah, I understand what you were trying to do. I wouldn't advocate for the sharding in that case. Keys-only query would take a bit longer to execute than a straight get, but the other solution is untenable. [8:03pm] Wooble: If you have really consistent contention for the counters, your taskqueues might get overwhelmed anyway; if you're really seeing 5+ writes per second and you're executing the tasks at a slower rate, they'll literally never finish... [8:03pm] Wesley_Google: ok guys, it's 8pm PDT now... thanks for coming to office hours. the next one is in several weeks but in the morning from 9-10a PDT... hope to see you then! [8:03pm] Wooble: yodler12: have you considered blobstore? [8:04pm] svm_invictvs: apijason_google: Well, I don't actually fetch the keys. I just use PreparedQuery.countEntities() and just count the number of entities of a rank higher thatn the entity I'm trying to rank... [8:04pm] mrleroylee: Thanks [8:04pm] amalgameate: thanks guys! [8:04pm] svm_invictvs: THen just estimate that entity's ranking against the others. [8:04pm] apijason_google: svm_invictvs: I see. [8:04pm] ikai_google: thanks for coming out, guys! [8:04pm] mrleroylee: Are these chats archived anywhere? [8:05pm] ikai_google: mrleroylee: In the Google Group [8:05pm] ikai_google: Jason sends these out, along with a summary [8:05pm] chilts: it'll be summarised too [8:05pm] chilts: yeah [8:05pm] mrleroylee: cool [8:05pm] chilts: thanks Googlers! [8:05pm] svm_invictvs: Basically, I can estimate where a player ranks in relation to the rest of the users based on how many people are "above" that player [8:05pm] apijason_google: chilts: Yeah, I haven't been able to send the last few out though... hopefully I'll get this one up soon. [8:06pm] apijason_google: I have to head out, but thanks for the great chat Everyone. [8:06pm] svm_invictvs: Thanks! [8:06pm] apijason_google: See you on the 21st, 9-10 a.m. PDT. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
