Hey everyone, Posted below is the transcript from IRC office hours on November 3rd, 2010. The only App Engine team member in attendance was myself.
Notes: - Go Giants! Yes, San Francisco was crazy this day and a lot of us had challenges getting to work. - Is it possible to set logic in a class constructor to add new fields in Java ( http://michael.robellard.com/2010/10/updating-appengine-kinds-without-taking.html)? Yes, but it might not be worth it to do it this way - Mapper API is probably better for schema migrations. - Is it possible to check in a non-blocking fashion if an asynchronous URLFetch has finished? Yes in Java (it's just a Future) - Is the team trying to make it out to Pycon 2011? Yep! We're gonna try. - Should I do joins in my data layer? You can ( http://gae-java-persistence.blogspot.com/2010/03/executing-simple-joins-across-owned.html), but it might make more sense to denormalize and reduce the number of queries that need to hit the datastore. - General memcache questions - no immediate plans to implement CAS. ------- [7:00pm] ikai_google: Alright everybody! it's time for App Engine office hours [7:00pm] mbw: ikai_google: I hope you guys are not as exhausted as I am... *sigh* [7:00pm] ikai_google: mbw: I am pretty exhausted. [7:00pm] xirkus: ikai_google: do you have some time to answer a question about PersistenceManager for java? [7:00pm] ikai_google: mbw: but the team is pretty taxed in general [7:00pm] ikai_google: xirkus: I'll do what I can, but if I don't know the answer off the top of my head I'll refer you to the groups [7:01pm] xeer1 left the chat room. (Read error: Connection reset by peer) [7:01pm] ikai_google: So the team was ... challenged this morning getting to work. Radio estimates 1.5 million people gathering in downtown San Francisco for Giants victory parade! [7:01pm] __key__: ikai_google, I wanted to let you know of an odd error i've seen a few times. I try to .put() and get either deadline exceeded OR datastore timeout, however, in fact, the put() ends up being successful behind the scenes. I've wittnessed this about 4 times... maybe something to look at. [7:01pm] mbw: great, more contention issues :D [7:01pm] ikai_google: public transit in San Francisco experienced a meltdown. [7:01pm] xeer joined the chat room. [7:02pm] ikai_google: __key__: Python? That's strange. You get deadline exceeded from the API call or the handler? [7:02pm] xeer left the chat room. [7:02pm] xirkus: np. I was wondering if you can migrate existing entities by putting logic in a class's constructor. There is a python article which suggests this : http://michael.robellard.com/2010/10/updating-appengine-kinds-without-taking.html [7:02pm] ikai_google: __key__: Can you set a datastore deadline instead? [7:02pm] xirkus: I was just wondering if this would work for java as well. [7:02pm] robertk joined the chat room. [7:02pm] __key__: ikai_google, i saw it once with deadline exceeded, so I started setting all my rpc timeouts to 22 seconds...however I saw it once again with the rpc timeout set, after getting the datastore timeout [7:03pm] __key__: yes pythong [7:03pm] ikai_google: xirkus: MMmmm not sure about JDO/JPA but if you have a wrapper class on top of entity, why not? [7:03pm] ikai_google: xirkus: Alternatively, you can use the Mapper API [7:03pm] ikai_google: xirkus: That's the way I'd do it. These constructor schemes only work once [7:03pm] ikai_google: xirkus: Once you have more than one schema migration it gets nasty [7:03pm] ikai_google: xirkus: I'm speaking from experience migrating JSON schemas stored in MySQL blobs lazily the way he describes [7:04pm] xirkus: ikai_google: thanks. I read about the Mapper API. I won't waste my time pursuing this avenue. I've added a version property to my entites; I'll look at Mapper for the migration. Thanks. [7:05pm] ikai_google: xirkus: Great! Time for some self promotion http://ikaisays.com/2010/07/09/using-the-java-mapper-framework-for-app-engine/ [7:05pm] ikai_google: (I wish I had more time to do more blog posts ... ) [7:06pm] xirkus: ikai_google: thanks for the link. I'll have a look. [7:06pm] jwbnyc: Given the need to randomly instantiate a (Java) handler class, I need a list of available classes. I'm thinking of putting the list into a file under war/WEB-INF/classes. I haven't tried it yet, so I wonder if the use of getResourceAsStream sounds like a good way to go as opposed to generating a class to contain the list (statically) and getting the class loaded as usual. (Of course, the... [7:06pm] jwbnyc: ...randomly selected classes would be loaded via Class.forName().) [7:08pm] ikai_google: jwbnyc: ... good question. I don't know off the top of my head. I'd classify that as one of those "try it first, then if it doesn't work, figure out why you need it" type of problems [7:08pm] dgu: ikai_google: Is there a non-blocking way to check if an RPC has completed or not? (in particular, a urlfetch RPC). [7:09pm] dgu: I tried to trace wait()'s implementation, but unfortunately it eventually calls into some built-in code which doesn't seem to be open source. [7:09pm] ikai_google: dgu: are you using Java? [7:09pm] dgu: Python [7:09pm] ikai_google: dgu: With Java, since the async call returns a Future, you can just call isDone() [7:09pm] ikai_google: dgu: oh, poor you. [7:09pm] ikai_google: KIDDING! pythonistas [7:09pm] dgu: :) [7:10pm] ikai_google: dgu: I don't know off the top of my head, but here's the Java implementation http://download.oracle.com/javase/6/docs/api/java/util/concurrent/Future.html#isDone() [7:10pm] dgu: Hmm, that's handy. I don't think I saw anything like that in the Python API or code :/. [7:11pm] mbw: ikai_google: do you know if anyone from your team is going to make it out to pycon 11? Brett was there last year. [7:11pm] jasonphd joined the chat room. [7:11pm] ikai_google: dgu: Can you ask in the groups? Wesley and Nick are python experts [7:11pm] u1001101012 left the chat room. (Quit: Leaving.) [7:11pm] ikai_google: mbw: We are ALL trying to get into Pycon [7:11pm] ikai_google: mbw: Wesley submitted like, 6 talks [7:11pm] dgu: Will do, thanks Ikai. [7:11pm] ikai_google: mbw: I submitted one that has nothing to do with GAE. It's about the joy of TornadoWeb/MongoDB [7:11pm] mbw: ikai_google: I plan on being there also, along with nickjoyce [7:11pm] ikai_google: mbw: wesley will be there almost for sure [7:12pm] ikai_google: mbw: Oh and Guido kind of has to go, I guess [7:12pm] mbw: i'd love to actually get a chance to show you guys code.. still havent ever had time [7:12pm] droid joined the chat room. [7:12pm] droid is now known as Guest68840. [7:12pm] ikai_google: mbw: I wouldn't be surprised if a ton of GAE core team members go to Pycon [7:13pm] jasonphd: can anyone help with this datastore question i have about joins? my question is the latest comment on this post: http://gae-java-persistence.blogspot.com/2010/03/executing-simple-joins-across-owned.html [7:13pm] ikai_google: mbw: Yeah ... we'll see if Wesley and Nick go. Brett is very likely to go and always happy to talk GAE [7:13pm] mbw: ikai_google: I loved last years GIL talk, that was pure geek comedy gold [7:13pm] ikai_google: mbw: I didn't see it, guess I'll have to look it up [7:14pm] ikai_google: jasonphd: I don't know off hte top of my head. Instead of designing like that, why not make the classes a list property and do an IN search? [7:14pm] ikai_google: jasonphd: App Engine's datastore highly favors denormalization, so try not to think in terms of relational concepts such as joins when possible [7:15pm] nigini left the chat room. (Ping timeout: 265 seconds) [7:15pm] ikai_google: jasonphd: only sadness lies down that road. [7:15pm] mbw: ikai_google: http://blip.tv/file/3254256 [7:16pm] ikai_google: mbw: what's up with the first comment? wtf? [7:16pm] jasonphd: in that example, "credits" is a list, right? [7:16pm] mbw: who knows, spam. I could not find the youtube copy of the vid [7:17pm] dgu: ikai_google: Any reason the time (timeout) parameter isn't accept by memcache.incr()/decr()? [7:17pm] dgu: If I pass a value for default_value and there's nothing in memcache for the key I specified, then it seems like it will end up adding a non-expiring entry to the cache (well, until it gets evicted due to memory pressure or something...). [7:17pm] ikai_google: dgu: I think that's just the memcache API [7:17pm] ikai_google: dgu: I don't recall ever setting an expiry on it [7:18pm] __key__: dgu i've run into that and I have tried unsuccessfully to set expiration with memcache.incr [7:18pm] [Outcast] left the chat room. (Quit: [Outcast]) [7:18pm] __key__: basically you have to build your own [7:18pm] ikai_google: jasonphd: Hrmm ... have to think about your example [7:18pm] ikai_google: jasonphd: But what you'd do is place Philosophy and Math in a List property on student [7:19pm] ikai_google: jasonphd: But the resulting query would do TWO queries anyway and join them in memory [7:19pm] dgu: __key__: How would you build your own? If you omit default_value and then just use set() if incr() fails, you can pass a timeout, but set() doesn't play nicely if a lot of instances are trying to incr() the same key at once. [7:19pm] ikai_google: dgu: Yeah, but when you're working with values that need incr and decr chances are good you are okay with some volatility with the value, and being off by a few is not that bad [7:20pm] dgu: true [7:20pm] ikai_google: dgu: Basically ... what I'm getting is that the "ballpark" of the value is more important than the exact value [7:20pm] __key__: like mentioned in the "walkscore" io vid [7:20pm] ikai_google: dgu: Example memcache usage: view counts that are periodically flushed to datastore. You're okay if the true value is 100,000 but you recorded 99,990 [7:21pm] jwbnyc: Can you say if we're likely to be able to buy IP addresses so that our apps can make requests to services that require the remote address to be whitelisted? [7:21pm] dgu: Sure; I definitely agree that it will be good enough. [7:21pm] mbw: ikai_google: what if I am not ok with volatility... isnt it safe to use incr decr still? its atomic more or less right? [7:21pm] ikai_google: mbw: it's atomic. but the problem is that memcache can get flushed at any time [7:22pm] mbw: oh, ya.. thats fine [7:22pm] ikai_google: mbw: so chances are you are backing up te value periodically and using memcache as an optimization sacrificing some accuracy for performance [7:22pm] mbw: which we do, sharded counter just layers mc on top [7:22pm] mbw: ya [7:22pm] ikai_google: mbw: yep, that's how you do it [7:22pm] ikai_google: mbw: We really need to support CAS [7:23pm] robertk: ikai_google: what happen to memcahce cas? [7:23pm] mbw: i know... we broght that up at the thing ... [7:23pm] ikai_google: CAS = check and set. think transaction on a single memcache key/value pair [7:23pm] ikai_google: robertk: We actually use a fork of memcache that has some security things in place [7:23pm] ikai_google: robertk: I don't know why we don't support CAS [7:23pm] robertk: i sort of heard maybe it could happen "soon" though. [7:23pm] jasonphd: What I am really try do do is have a "Document" class, a "DocumentSearchKey" class, and a "DocumentSearchKeyValues" class.... the "DocumentSearchKeyValues" class would be contained as a List element of the "Document" class and would contain a "foreign key" linking to the "DocumentSearchKey" class.... so my query really needs to do more than see if a string exists in a list... it needs to say for a [7:23pm] jasonphd: certain DocumentSearchKey find documents with matching DocumentSearchKeyValue elements... that was probably tough to follow [7:23pm] robertk: but that was a few months ago. [7:23pm] ikai_google: robertk: Really? I didn't hear that [7:24pm] mbw: for the most part, using .set and .add in the right way does what we need. One area that we don't have a good solution for yet is mc.set in transactions that get rolled back... we need to delete the memcache keys as part of the rollback [7:24pm] rafanunes joined the chat room. [7:25pm] ikai_google: jasonphd: If you had to think of entities as a HashMap, how would you persist the data? That's how you approach data modeling in GAE [7:25pm] mbw: all we can do now is catch rollbacks and handle that on our own, but if it were to happen in an odd situation like a deadline, we never get a chance [7:25pm] ikai_google: mbw: Yeah, I can't think of another way to do that [7:26pm] ikai_google: mbw: Hrmm... yeah the only other thing I can think of is 2 memcache keys [7:26pm] ikai_google: one that you set or incr after the transaction completes [7:26pm] mbw: well.. .you have transactional tasks, how about a task that only fires if a rollback occurs? [7:26pm] jasonphd: so it is typically not a good idea to use any relationships? try to make data as denormalized as possible? [7:26pm] mbw: ikai_google: two keys might work, but feels so dirty... ill have to think about it [7:27pm] ikai_google: jasonphd: Yes. But there are times when structured data makes more sense. In these cases, you would store the Key of related entities [7:27pm] mbw: ikai_google: thats on my todo list though [7:27pm] perlmonkey2 left the chat room. (Ping timeout: 245 seconds) [7:27pm] ikai_google: jasonphd: But anytime you query on multiple entity kinds it's more queries that hit the datastore [7:27pm] rafanunes left the chat room. (Read error: Connection reset by peer) [7:27pm] jasonphd: thanks for your help [7:27pm] rafanunes joined the chat room. [7:27pm] rafanunes left the chat room. (Remote host closed the connection) [7:28pm] rafanunes joined the chat room. [7:28pm] rafanunes: ping [7:29pm] mbw: rafanunes: as in ... 'hi' or like icmp ? [7:30pm] fpotter left the chat room. (Quit: fpotter) [7:31pm] rafanunes: more like 'is my connection working?' [7:31pm] xzcvczx: rafanunes: no its not :( [7:31pm] xzcvczx: :P [7:33pm] rafanunes: tks...i will restart to fix it.... [7:33pm] jwbnyc: I'm also interested in whether the request to provide a webservice (appcfg command) allowing the default version to be changed ( http://code.google.com/p/googleappengine/issues/detail?id=1996) is likely to be resolved in a near-term release. [7:33pm] ikai_google: mbw: Well, if it makes you feel better "transaction journaling" is pretty much how everything else works [7:33pm] ikai_google: mbw: a version number somewhere. it just so happens that you're going to end up implementing it yourself, which is annoying [7:34pm] ikai_google: mbw: I'm personally annoyed whenever I need to implement something that has already been implemented a million times 100x better than I can do it [7:34pm] ozburo joined the chat room. [7:35pm] mbw: ikai_google: What makes my job fun is we get to implement a lot of things that we don't see anyone else doing anything like... :D [7:36pm] mbw: I have been having fun with namespaces lately. I have some api patches which allow me to leave certain Kinds in default namespace while others default to current namespace, then I set namespace in the middleware based on logged in account... and you get automatic namespace management by kind [7:37pm] rafanunes left the chat room. (Read error: Connection reset by peer) [7:37pm] mbw: Also working on channel a bit, we want to support that asap. [7:37pm] ikai_google: mbw: =P one of these days I'll have the time to do cool things again too. [7:39pm] mbw: the namespace patches are fun because ignoring migration work, I didnt have to refactor any code to get data split up by account [7:39pm] jasonphd left the chat room. (Quit: HydraIRC -> http://www.hydrairc.com <- The alternative IRC client) [7:42pm] mbw: ikai_google: one other question. I was bugging Fred about this before too. Any chance we can get Application/x-amf setup for gzip on the front ends? [7:42pm] ikai_google: mbw: Maybe ... keep bugging fred. I don't want to say no because I think it can be done, we just have to work with the front end teams [7:42pm] mbw: ikai_google: they did get text/x-amf setup for gzip, and it works, but it breaks charles proxy, which we just plain need for troubleshooting... plus we like using proper mime types [7:43pm] mbw: ikai_google: ok, ill keep bugging him [7:43pm] guest01 joined the chat room. [7:43pm] ikai_google: mbw: bug either fred or chris [7:43pm] guest01 left the chat room. (Client Quit) [7:44pm] mbw: ikai_google: I pretty much always to/cc both these days [7:44pm] ikai_google: mbw: you can also bug me but I'm just going to bug fred or chris. that's how messages propagate [7:44pm] mbw: the question is... are you sick of all the WF people bugging you yet? :P [7:44pm] steph_ joined the chat room. [7:45pm] ikai_google: mbw: Not really, it's actually not so bad at all if it's someone who 1. already is a happy customer 2. knows what they're doing 3. very successful [7:45pm] steph_ left the chat room. (Client Quit) [7:45pm] ikai_google: mbw: being bugged is what we're about. it's only bad if it's folks who don't read the docs and don't try things first themselves [7:46pm] vanriper joined the chat room. [7:46pm] robertk: oh, yeah. hey i got some message about my my being sent too many sms messages. is that documented _anywhere_? ;) [7:47pm] robertk: *my mobile [7:48pm] mbw: robertk: the proper way to handle that is 1. go to google i/o 2. get 2 free phones 3. register accounts during free trial period [7:48pm] robertk: ahhhhh. now i get it! [7:48pm] mbw: lol [7:48pm] dgu: ikai_google: Any suggestions for monitoring the status of an app without being glued to the dashboard (or screen-scraping it)? Maybe in the future there will be an API for interrogating the dashboard (e.g., my task queue is backed up) :)? [7:49pm] ikai_google: dgu: Unfortunately not. We're looking at how to design a dashboard API [7:49pm] ikai_google: dgu: We've heard from developers who are screen scraping. Kind of sucks [7:50pm] robertk: ikai_google, any hints about what will be addressed in the upcoming maintenance? Or, maybe more answerable is it a 'feature rollout' or 'maintenance'? [7:50pm] mbw: dju: the log buffers roll over really really quickly, so in order to get all the log level data we pull our logs every 15 minutes into a massive mysql database, then we can crunch the data and generate tables that tell us average latency by url, user facing, tasks, etc... It takes some work but its very useful [7:50pm] ikai_google: robertk: It's a real maintenane [7:50pm] ikai_google: robertk: Hoping to resolve many of the issues we've had since the sept maintenance [7:50pm] dgu: Yeah, slighty better than checking it yourself - but not as good as an API :). Well, cool to here that you guys are brainstorming a solution. [7:51pm] robertk: ok, excellent. [7:51pm] robertk: mbw: are you using 'their' log download or your own parsing? [7:51pm] ikai_google: yeah, what would be awesome if if all the data got logged into bigquery. maybe as a paid feature http://code.google.com/apis/bigquery/ [7:52pm] mbw: ikai_google: I heard a bit about what is going to happen sat, are you also going to change the gae status site at all to reflect that? It already doesn't seem very accurate compared to the health of our app [7:52pm] mbw: robertk: we pull logs with appcfg request_logs or whatever, then we parse it ourselves [7:52pm] ikai_google: mbw: I don't think we're making any changes to status site on saturday [7:52pm] mbw: parse and put into mysql [7:52pm] Guest68840 left the chat room. [7:52pm] dgu: mbw: Yeah, lots of other good data you can get from the logs too. (should be able to run it through awstats or similar tools) [7:53pm] mbw: dgu: we look at the custom stuff they add like loading, pending, etc also. we also look for certain errors, all sorts of things [7:53pm] dgu: ikai_google: Interesting, hadn't thought about you guys just dumping into bigquery as a paid feature ... that would work too :). [7:54pm] dgu: mbw: Nice. We *should* do that ... but it languishes as a todo list item for now. [7:54pm] ikai_google: dgu: Yeah, well there's been a reorg recently, now all the different "cloud" solutions report to the same hierarchy [7:54pm] mbw: dgu: but the most important part of it is just tracking our average user facing latency by url and totals, over certain time periods, so we can accurately report back to google about our apps health [7:54pm] ikai_google: dgu: So I'd anticipate long time more (buzzword alert) *synergy* among the different APIs [7:55pm] mbw: ikai_google: as part of the reorg I hope you get more people... [7:55pm] dgu: haha, well buzzword or not it sounds like a change for the good [7:55pm] ikai_google: mbw: Well, we've asked. heh [7:55pm] vanriper left the chat room. (Ping timeout: 265 seconds) [7:56pm] dgu: Regarding the mapreduce library, has anybody written an input which only maps over just a subset of an entity perhaps? [7:56pm] • mbw needs to head out [7:56pm] mbw: thanks ikai_google for running office hours today [7:57pm] robertk: hey mbw, one more quick q on your log analysis. do you have any way to distinguish different app instances? [7:57pm] robertk: to look for borked instances. [7:57pm] jwbnyc: I was recently asked what the biggest app on GAE is. Any nominees? [7:57pm] ikai_google: dgu: Subset of an entity? Or all entites? [7:57pm] mbw: robertk: yes, we have that as a field in our sql db, so we can filter by appid... but our production site is just one appid [7:57pm] dgu: ikai_google: subset of an entity [7:58pm] ikai_google: dgu: What does that mean? [7:58pm] ikai_google: dgu: You return part of one entity? [7:58pm] dgu: Oh no, I mean a subset of all entities. Sorry. [7:58pm] mbw: i'm out... peace, love and low latency [7:58pm] ikai_google: dgu: Can't be done. You get entities whole or not at all. Can't do it piecemiel [7:58pm] daws left the chat room. (Remote host closed the connection) [7:58pm] ikai_google: dgu: heh. Nope not yet [7:58pm] ikai_google: mbw: good night! [7:58pm] dgu: k, that's why I couldn't find one then :) [7:59pm] ikai_google: Okay then! Thanks everyone for coming out to office hours [7:59pm] dgu: thanks Ikai [7:59pm] ikai_google: as usual, I like to call out the regulars: mbw, robertk [8:00pm] ikai_google: when things die down a bit I try to hang out in IRC as much as possible [8:00pm] ikai_google: so catch me here, in the groups, etc [8:00pm] ikai_google: good night everyone! -- Ikai Lan Developer Programs Engineer, Google App Engine Blogger: http://googleappengine.blogspot.com Reddit: http://www.reddit.com/r/appengine Twitter: http://twitter.com/app_engine -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
