This past Wednesday, the App Engine team hosted the latest session of its bimonthly IRC office hours. A transcript of the session and a summary of the topics covered is provided below. The next session will take place on Wednesday, January 6th from 7:00-8:00 p.m. PST in the #appengine channel on irc.freenode.net.
- Jason --SUMMARY----------------------------------------------------------- - Q: What kind of data redundancy do uploaded files and stored data have and is this redundancy counted towards the Stored Data quota? A: We use GFS for data storage, which has its own redundancy, and we also replicate to multiple datacenters and take backups. The cost of all of this replication is included in the per-gigabyte cost, so you are only charged for one copy of your data plus the indexes needed to query it. [9:02-9:03] - Discussion on feature request (http://code.google.com/p/ googleappengine/issues/detail?id=2377) to add more detail to the App Engine status page and/or create a per-app status page to check whether specific apps are being impacted by a particular issue since not all apps are affected by all issues. Note: we do post planned and unplanned downtime to the downtime-notify mailing list, so please subscribe to http://groups.google.com/group/google-appengine-downtime-notify to receive reminders about planned maintenance and status updates in the event of system issues. [9:04-9:10, 9:16] - Discussion on planned maintenances in general -- every 30-60 days, App Engine enters read-only mode where apps continue serving but the datastore can only be read, not written to. We're working on reducing the occurrences of these planned maintenances as well as the time it takes to complete each one, and will have a calendar set up soon so developers can keep up to date with the maintenance schedule. [9:34-9:39] - Q: Can one achieve better peformance by writing entities in different entity groups outside of a transaction over writing multiple entitites in the same entity group inside of a transaction? A: Yes! App Engine supports batch (parallel) writes for entities in distinct entity groups. Updates to multiple entities in the same entity group, regardless of whether you use transactions, forces the entities to be written serially which takes more time. In order to achieve the best performance, keep your entity groups as small as possible and only use transactions if you absolutely need atomic writes to two or more entities. [9:09-9:11, 9:13-9:15] - Q: Can you map subdomains to versions? A: You can do this right now -- when you deploy a new version, you can access it by hitting <version_string>.latest.<app_id>.appspot.com, even if it's not the default version. As of release 1.2.4, App Engine supports wildcard subdomains, which work without any explicit configuration. To determine which subdomain is being used, inspect the 'Host' request header -- see http://googleappengine.blogspot.com/2009/08/new-features-in-124.html. [9:11, 9:14, 9:16, 9:18] - Discussion on feature request to allow app deployment to domains other than appspot.com because of the crowded namespace. Note: App Engine already allows you to deploy your application to any Google Apps domain, so if you don't plan to host directly on appspot.com, a recognizable app ID isn't as important. You can always prefix your own username/alias to any app ID you choose to reduce the possibility that the app ID is taken. [9:16-9:17, 9:21] - Q: Is there any way to query for "not null" without using an inequality? A: "not equal to" or != is implemented as two inequality filters, so the answer is no. One workaround is using a calculated property instead -- at write time, determine if a certain property contains a value of interest. If so, store True in a boolean property, False otherwise, then use an equality filter on this boolean property. Or just filter the results in memory. Since empty property values (displayed as <missing> in the datastore viewer) are not indexed, you can just leave a particular property without a value so when you issue a query that filters on that property, entities with <missing> values don't get returned as results. You can't currently query on whether a field is empty yet, although you can query on whether a certain property value is null since null is an actual value. [9:19-9:26] - Q: Are there any plans to increase the size of a result set from 1,000? A: No, but datastore cursors, which are on the public roadmap, should make it easier to page through larger result sets. A cursor is an object that points to a particular point in the result set of a query which can be serialized and reconstituted so that you can pick up from where you left off in a query -- essentially, it's an object that helps you page through your result set, even if there are more than 1,000 results. [9:26-9:29] - Q: Are there any limits to the number of "blobs" you can store using the new Blobstore service? A: No, but space needed for blobs are counted against your storage quota, so you will have to enable billing for your app if you exceed 1 GB of total storage space for all blobs, entities, and indexes. [9:44-9:45] - Discussion on using app-engine-patch for Django versus using Django with the Python datastore service directly including discussion on performance drawbacks inherent in using Django and app-engine-patch. [9:41-9:47, 9:49-9:53, 9:56, 9:58, 10:00-10:01, 10:05] - Q: Is using the Java runtime necessarily more cost effective than using the Python runtime due to lower CPU usage? A: There are many factors to consider, but using Java isn't necessarily more cost effective. There may be some operations that require less CPU in Java, but this is usually tempered by the performance of other operations which are faster in Python, e.g. cold start-up time. [9:56-9:57] - Discussion on support for JIT for App Engine's Java runtime and recent peformance improvements for Java specifically. (http:// googleappengine.blogspot.com/2009/12/request-performance-in-java.html) [9:59-10:03] --FULL TRANSCRIPT--------------------------------------------------- [09:01am] ikai_google: What's going on, everybody? [09:01am] nickjohnson: Welcome to the fortnightly App Engine developer chat. With us tonight are myself, Wesley, Ikai, and others should be joining shortly. [09:02am] daskalou: What kind of data redundancy do uploaded files and data in the Datastore have? And is this redundancy counted towards our Stored Data quota? [09:02am] nickjohnson: Here's Jason, too. [09:02am] Jason_Google: Hi Everyone. [09:03am] daskalou: hi GAE team! [09:03am] nickjohnson: daskalou: We use GFS for data storage, which has its own redundancy. We also replicate to multiple datacenters and take backups. The cost of all of this is included in the per-gigabyte cost; the amount used is only for one copy of your data. [09:03am] Wesley_Google: Hi, this is Wesley, I'm on the App Engine team as well, and your Python doctor [09:04am] daskalou: nickjohnson: great! [09:04am] _matt_: Q: Any plans to add more information and detail to the 'appengine status page'? [09:04am] daskalou: Ok another question... [09:04am] nickjohnson: _matt_: What sort of information are you looking for? [09:05am] ikai_google: _matt_: What kind of information are you looking for? [09:05am] ikai_google: um [09:05am] _matt_: well, the specific thing I'm thinking of is related to an issue I filed: http://code.google.com/p/googleappengine/issues/detail?id=2377 [09:05am] ikai_google: we are different people [09:05am] ikai_google: I swear [09:05am] _matt_: haha [09:06am] daskalou: Q: I know we can change the version in app.yaml and use the Admin Console to switch between versions, but can we also switch between revisions of the same version? [09:06am] ikai_google: daskalou: Revisions of the same version? You mean like SCM revisions? [09:06am] Wesley_Google: Oops, sorry I didn't send mine, which was "What type of information are you looking for?" [09:07am] ikai_google: _matt_: I think we'll need to look at this carefully. The reason is that in instances of degraded performance, it usually applies to a subset of users [09:07am] Jason_Google: daskalou: No, you can't currently switch between minor versions of an app. This might be possible at some point but isn't exposed through the UI right now. [09:07am] ikai_google: _matt_: We try to provide a % of users impacted in the groups [09:08am] ikai_google: _matt_: Are you subscribed to appengine- downtime-notify? [09:08am] daskalou: ah ok, Jason_Google answered it [09:08am] ikai_google: _matt_: We could probably keep that one a bit more up to date while we figure out a strategy for making the status page more useful in these situations [09:09am] lent: java: there have been quotes of update rate in appengine being up to 10 per second for entities within an entity group in a transaction. Is the update rate higher if the updates are being done to different entity groups outside a transaction? [09:09am] _matt_: ikai_google: not sure what the solution is, but when you are one of the affected users, and you go the status page and it says everything is good, it seems oddd.... maybe a status page for a given app (which shows the status of the places where it is deployed?) ... not sure. [09:09am] daskalou_: DC'd [09:10am] _matt_: ika_google: bottom line....some way to know if you're one of the 'affected' 'subset' of users [09:10am] maxoizo: Hi all. Do you have any plans to reduce the cost of data storage and bandwidth? [09:10am] Jason_Google: lent: Writes to separate entity groups can be done in parallel. When writing to the same entity group in a transaction, the writes have to occur one after the other, hence they may take longer. [09:11am] Jason_Google: maxoizo: Not that I know of. Any billing changes will be announced at least 3 months before they go into effect, in line with our TOS. [09:11am] lent: Jason_Google: is there any way to achieve parallel updates for within a single request? Does a batch put get processed in parallel? [09:11am] ikai_google: maxoizo: There are no current plans, but as storage and bandwidth become cheaper we will always look at adjusting our prices, kind of like what we did with Picasa [09:11am] jeverling: Hi all, is there a chance that it will be possible to map subdomains to versions? So that you can use different versions for localized instances of your app, for example? [09:13am] Jason_Google: lent: Not to the same entity group, no. Writes to the same entity group must necessarily be serial, which is why we encourage you to keep your entity groups as small as possible or just stick with root entities if transactions aren't required. [09:13am] ikai_google: jeverling: For localization, it's probably best to use framework leve capabilities [09:13am] jeverling: ikai_google: ok, thanks [09:14am] Jason_Google: jeverling: Interesting idea. For localization, I second Ikai's suggestion, but it might be an interesting feature down the line for other reasons. Please file it in the issue tracker if you haven't already. [09:14am] lent: Jason_Google: How about a batch put to entities in separate entity groups outside a transaction? Does that get processed in parallel? [09:14am] ikai_google: jeverling: I dodged your question, though You can access different versions at version.latest.appid.appspot.com. I'm not sure off the top of my head if a subdomain can be mapped from an outside DNS control panel [09:15am] Jason_Google: lent: Yes. [09:16am] _matt_: Q: any plans to offer deploying apps on a base domain other than appspot.com in the future? The availability of names seems scarce there. Took me 4 tries just come up with a name for a project I was using to test an api [09:16am] maxoizo: 2ikai_google: Yes, 20GB for 5$ per year - it's very nice. [09:16am] nickjohnson: We're back, baby! [09:16am] nickjohnson: _matt_: We're working on improving reporting. The issue is that most of the problems you refer to are local, affecting only a few apps, so it's hard to measure and present that on a concise status page. [09:16am] jeverling: afaik it wasn't possible the last time i checked. I will file a feature request, I also think this might be useful. [09:16am] eneekmot: Hey, I can't figure out how to get chatbots to do stuff in color, you use CSS, right? Why not standard HTML tags? [09:16am] nickjohnson: _matt_: You can map any domain you own to your app engine app already [09:17am] nickjohnson: You'll always have to pick an App ID, but if you don't intend to serve off appspot, it's not as important that it's recognizable [09:18am] Jason_Google: jeverling: You can configure wildcard subdomains, but it's not automatically mapped, aside from the version mappings that Ikai mentioned. It's possible to process the 'Host' header to create your own mappings though: http://googleappengine.blogspot.com/2009/08/new-features-in-124.html [09:19am] lent: A query question. When you want to query on a field, where its value is not null, it has been suggested to use field is greater than null. I have a problem where I need to do an inequality on another field and I get an error saying that you can do comparisons on two separate fields. Is there any other way to do "is not null" other than greater than null? [09:19am] jeverling: Jason_Google: cool, that looks very useful! [09:20am] Jason_Google: lent: != counts as an inequality filter, and as you mentioned, you can only do inequality on one property. [09:20am] nickjohnson: lent: No, because != is implemented as > and < filters. You can modify your schema, and use a calculated property instead, though [09:20am] ikai_google: lent: No, "null" values are not indexed, and != filter is really just a merge of < and > [09:20am] nickjohnson: Or if few elements will have null, filter them out by hand. [09:20am] ikai_google: GAH [09:20am] jeverling: [09:21am] _matt_: nickjohnson: re:mapping domains, yeah that's what I'm doing for any serious projects. But for small, discardable things, or for someone just getting started on appengine I think it might be frustrating to have to come up with a unique name (given how crowded the namespace is) just to try it out . . . [09:21am] fmarcoz: what's that 20GB for 5$ per year? [09:21am] Jason_Google: Clarification: <null> is indexed, <missing> is not. If you leave a certain property value <missing>, then you don't have to worry about it since they won't ever be included in query results involving that property. [09:22am] _matt_: nickjohnson: github has an interesting approach where the base namespace is the owner name.....that could be interesting.... [09:22am] eneekmot: I had a few issues with the Google Wave chatbot API, did you know that it can't find your classes (Java) if you put the Profile class in anything other than the default package? [09:22am] ikai_google: Right, I misspoke. You can look for a "null" value, but not an empty value. This is because at the datastore level, we don't enforce a hard schema [09:22am] nickjohnson: _matt_: That's not going to change any time soon. You could always prefix your name to the app ID. [09:23am] lent: When you add a new field to an kind and there are existing entities, is the field value for the new fields null (as retrieve through java apis) or is it some "other" value? [09:23am] Jason_Google: lent: It's empty or <missing> but not NULL which is an actual value. [09:23am] ikai_google: so when entities are stored they are stored as serialized Protobufs. Analogy here is storing values as JSON (though protobufs have a bit more type information) [09:24am] Jason_Google: Although if you were to fetch the entity using JDO/JPA, it will appear as null to your app. [09:24am] nickjohnson: lent: It's not changed in the datastore, but it will show up as the default value if the field has one, or None if it doesn't (or an exception if there's no default and the field is required) [09:24am] lent: Jason_Google: is there any way to query for field value being empty or missing through java apis [09:24am] Jason_Google: lent: No, you can't query for empty values at this time. [09:24am] Jason_Google: I'm ducking out for a few minutes but I'll be right back. [09:24am] ikai_google: lent: I'm going to simplify to JSON for explanations. Suppose you create a class Animal and define fields "name" and "age". Your entity would be stored as something like this: { entity_kind: Animal, name: "Lion", age: 3 } [09:25am] dunk_: you only can have int float string list and dict [09:25am] ikai_google: lent: That structure is stored in the datastore. When you add new fields, say, "favorite food", you're just adding new fields to each new entity you put in { kind: Animal, name: "Lion", age: 3, favorite_food: "meat" } [09:25am] dunk_: so no Animal i guess [09:25am] ikai_google: dunk_: It's a simplification [09:25am] dunk_: ah ok [09:26am] eneekmot: I also had a weird issue with the Wave chatbots, in some waves the bots will start repeating themselves, as though there are two bots in the conversation. It even would make two "hello" messages if I removed it from the conversation and added it again. [09:26am] ikai_google: so when you insert a new entity, we update the indexes [09:26am] daskalou_: Q: Any ETA on if and when the 1000 entity offset/ limit on queries will be increased (or abandoned altogether)? [09:26am] dunk_: i store json in a database with my bot [09:26am] dunk_: memcache in between [09:26am] ikai_google: and as you can see, entities with fields that exists before the index was created aren't indexed [09:26am] _matt_: Q: any plans to allow provisioning of a new appid (up to the 10 limit of course) via an API/cmdline..... ? [09:27am] ikai_google: daskalou_: Cursor support is on the roadmap [09:27am] ikai_google: _matt_: Not that I know of [09:27am] nickjohnson: daskalou_: That's not really a high priority. Generally if you're running into that limit, you need to reconsider what you're doing and what impact doing that much processing in a user request will have on latencies. Cursors will make handling many results easier for batch operations. [09:27am] lent: ikai_google: thanks for the clarification. [09:28am] daskalou_: Q: Can someone explain what exactly a "Cursor" is? [09:29am] Wesley_Google: daskalou: it let's you iterate over your results set [09:29am] nickjohnson: daskalou_: An object that points to a particular point in a query, which can be serialized and reconstituted later to let you pick up from where you left off [09:29am] eneekmot: Can I get some pointers on how to get wave robots to color text? [09:30am] daskalou_: ok, i understand now [09:30am] ikai_google: eneekmot: This is a bit outside of our expertise, though I am looking at building some wave robots in the near future. Have you tried the Google Wave groups? [09:30am] dunk_: annotations ? [09:31am] Jason_Google: eneekmot: I believe there are Wave office hours, and they might be taking place later today... [09:31am] dunk_: problem is that the wave hours are in the sandbox only ? [09:31am] Jason_Google: See http://code.google.com/events/calendar/ -- it looks like just another 90 minutes or so. [09:32am] dunk_: well this is #appengine not #wave i guess [09:32am] Jason_Google: dunk_: Yes, it looks like it's in the Wave sandbox. [09:32am] eneekmot: Ah, I thought that this was the developers chat, thanks Jason. [09:32am] eneekmot: I couldn't find the link for that one. [09:32am] dunk_: guess i need to fix a developers account [09:34am] daskalou_: Q: 1. When is the next scheduled downtime? 2. Why is scheduled downtime required (what do you guys do during it)? 3. How often do you foresee scheduled downtime happening in 2010? [09:35am] practicalint: Jason_Google: Hi - I'm back with my question on whether the TaskEngine demo app should run as delivered - still haven't been able to get it to. [09:35am] ikai_google: daskalou_: 1. I have these in a list somewhere. It's sometime in February or March. We're working on a calendar that you can subscribe to 2. Most of our updates do NOT require downtime, but there are instances where the cost of keeping everything up for maintenance as we update something are absolutely astronomical [09:36am] Jason_Google: practicalint: Can you send me a private email? apijason at google dot com and I'll look at it today. [09:36am] ikai_google: daskalou_: 2 (cont). These are the instances in which we schedule downtimes, we try to avoid these as much as possible, like if we are shifting datastores, we try to solve this by putting the datastore into Read Only mode [09:37am] ikai_google: daskalou_: This way applications can run in at least a degraded mode [09:37am] Jason_Google: daskalou: Most of the time, scheduled "downtime" really means that the datastore is in read-only mode. Your apps still continue to serve, and you can still read from the datastore, but you're not able to write data for around 15-60 minutes depending on the maintenance. [09:37am] ikai_google: daskalou_: I don't have any insight as to how much downtime there will be, but I know there are 2 planned in the first half of the year [09:38am] eneekmot: Well, I don't have a sandbox account, I probably won't be able to get into that chat today... [09:38am] practicalint: Jason_Google: will do. thanks. [09:38am] ikai_google: eneekmot: The chat should be IRC, so you should be able to go in and ask [09:38am] daskalou_: Q: do you guys think that, as GAE evolves, scheduled downtime/maintenance will be less and less frequent? [09:38am] Jason_Google: practicalint: Thanks, and sorry I haven't looked into it yet. I'll do it today after I get your email. [09:39am] Jason_Google: daskalou: Yes, that's the plan. We're working on a lot of improvements to make downtimes shorter and less frequent. [09:39am] ikai_google: daskalou_: Definitely. At the very least we will give developers the tools to deal with degraded modes [09:39am] nickjohnson: The recent blog post on datastore changes is worth a read with regards to that [09:40am] lent: does java HttpSession work when database is in readOnly mode? I know it is stored in memcache backed by database. [09:40am] daskalou_: nickjohnson: got a link handy? [09:41am] eneekmot: _ikai What IRC channel will it be on? [09:41am] daskalou_: Q: In your opinion(s), what is the "best" way to run Django on GAE? [09:41am] nickjohnson: daskalou_: http://googleappengine.blogspot.com/2009/09/migration-to-better-datastore.html [09:41am] daskalou_: nickjohnson: thanks [09:41am] Jason_Google: lent: No, I don't believe so, unfortunately. [09:41am] ikai_google: eneekmot: Oops, looks like I read it wrong, it is in Wave [09:41am] ikai_google: eneekmot: I guess you'll have to look at the Wave Google Groups [09:42am] eneekmot: That's what I'm doing right now, no luck yet. [09:42am] eneekmot: I put in a request for the Sandbox group, but Google is taking accounts manually right now, so I think I won't be able to get in within the next hour. [09:42am] Wesley_Google: daskalou: what do you mean by "best"? which ways are you familiar with? [09:43am] lent: Jason_Google: it would be helpful if it that HttpSession could gracefully degrade to just using memcache [09:43am] daskalou_: Wesley_Google: To be honest I've never used Django (or Python much) at all, but I've read the popular way is to use the app-engine-patch [09:44am] dunk_: eneekmot look at this http://code.google.com/p/wave-protocol/issues/detail?id=16 [09:44am] nickjohnson: lent: I actually tackled this a while ago. The issue is that if you do that, you could get reversions, which may be very bad for some apps. [09:44am] dunk_: there is something about coloring in there [09:44am] jeverling: There is no limit on the number of Blobs you can store other than your credit card, right? [09:44am] dunk_: not sure if it is what you need though [09:44am] daskalou_: I was wondering how you guys would "do it" [09:44am] ikai_google: daskalou_: I don't use app-engine-patch [09:44am] Jason_Google: lent: I agree. I've brought this up with the larger team before and the current reasoning is that if memcache were to ever be flushed (for whatever reason) and the datastore was also not available, that could potentially leave your app in a very weird state if it's dependent on sessions. Probably best to design your app to function as stateless as possible, although I realize it's sometimes easier said than done. [09:45am] nickjohnson: jeverling: correcty [09:45am] daskalou_: ikai_google: do you still run Django? [09:45am] nickjohnson: er, correct [09:45am] ikai_google: daskalou_: I use the native datastore API [09:45am] jeverling: ok, thanks [09:45am] ikai_google: daskalou_: I actually write Java and JVM languages more, but when I do Python, my personal preference is Django because I like the way it handles certain aspects [09:45am] nickjohnson: Well, there may be other limits, but you'll exhaust your credit card first [09:45am] ikai_google: daskalou_: I like forms, url mappings. Nick, for instance, prefers the Webapp framework [09:46am] eneekmot: _dunk Ah, thanks, that looks like it will work. [09:46am] dangerdave: daskalou_: This wasn't written by a GOOG employee, but you might find this interesting re GAE+Django: http://bit.ly/17md0R [09:46am] jeverling: nickjohnson: heh [09:47am] daskalou_: ikai_google: When you use Django on GAE, which of the methods do you use (if not app-engine-patch)? And why not app- engine-patch? [09:47am] daskalou_: dangerdave: thanks [09:47am] eneekmot: _dunk Looks like this is a feature request... Is this actually implemented in the API? [09:47am] chickamade: nickjohnson: when will your OpenID module be finished? [09:47am] Wesley_Google: daskalou: yup, i believe that helper is the route that most people go. i thought i heard of another helper out there but haven't found it yet. [09:47am] ikai_google: daskalou_: A lot of people do use app-engine- patch, I just don't find it necessary [09:47am] nickjohnson: chickamade: If all goes well, I'll release a first look at the end of this week [09:47am] nickjohnson: If not, it'll have to wait until the new year [09:47am] dangerdave: ikai_google: Thanks for making the change to 'citygoround' app -- no DeadlineExceededExceptions this morning. I'll continue to keep an eye on it and let you know if things change. [09:48am] eneekmot: _dunk Ah, a developer gives an example, I'll try that... [09:48am] ikai_google: dangerdave: Awesome! This is kind of a bandaid. Your particular app will not be affected again (not likely) [09:48am] ikai_google: dangerdave: But there is a long term solution in the works to address the underlying problem [09:48am] ikai_google: dangerdave: In the meantime we have to roll up our sleeves and bust out the duct tape [09:49am] dunk_: i just filled in the sandbox form [09:49am] dangerdave: ikai_google: got it. thanks for your quick response on this. I'd ask what the underlying problem is but it would probably require peering too deep into the black box [09:49am] daskalou_: ikai_google: What is "bad" about using app-engine- patch? Eg. I've read performance is bad on a cold request [09:49am] dunk_: want to join the office hour to talk about waveapi stuff [09:50am] Wesley_Google: daskalou: it's slightly outdated (based on Django 0.96), but i wrote an appendix section on how to get your Django apps to run on App Engine in our book "Python Web Development with Django" http://withdjango.com [09:50am] Wesley_Google: it features using the patch and how it integrates more of a Django dev feel to App Engine development [09:50am] ikai_google: daskalou_: Bad is relative ... from what I remember it eager loads [09:50am] daskalou_: Wesley_Google: thanks [09:51am] dangerdave: daskalou_: App Engine Patch uses zipimport over a pretty big zipfile... it can be slow. On cold request you basically have to unzip all of Django, in order to get at the code that knows how to respond to the request. [09:51am] daskalou_: ikai_google: ah ok (the anti-lazy loading im assuming) [09:51am] ikai_google: dangerdave: I'm oversimplifying greatly ... but the underlying problem is that when areas have hotspots, other apps can get caught in the crossfire, and our system should be smart enough to move apps out of resource depleted instances [09:51am] nickjohnson: daskalou_: That's not an issue of unzipping specifically, just that Django transitively imports basically the whole framework to serve anything. [09:52am] dunk_: ok i got a appengine question .. is the 1 gig a month quote on storage also for the free accounts ? [09:52am] daskalou_: gotcha [09:52am] dunk_: i mean is it per month or is it just fixed [09:52am] dangerdave: ikai_google: cool, I'll take that as good enough for now. Thanks again for the help. [09:52am] nickjohnson: dunk_: It's not 1 gig a month, it's 1 gigabyte stored data [09:52am] ikai_google: daskalou_: Again, it's relative. I just think Django's models map better to RDBMS persistence, whereas App Engine's db.Model maps better to our backend [09:52am] dunk_: ok nick thnx [09:52am] dunk_: and are the logfiles included in storage ? [09:53am] daskalou_: ikai_google: thanks for the insights [09:53am] nickjohnson: dunk_: no. [09:53am] ikai_google: daskalou_: I'm surprised Wesley doesn't have more input here, he looooves Django [09:53am] daskalou_: lol [09:53am] dunk_: ok [09:54am] daskalou_: well im looking for someone to sell it to me [09:54am] dunk_: then storing all the rss data in db might not be what i want [09:54am] Jason_Google: dunk_: Indexes are, however. [09:54am] daskalou_: I come from a PHP/Symfony world [09:54am] ikai_google: daskalou_: If you aren't sold, don't use it yet [09:54am] ikai_google: daskalou_: Oh dear, let's get you off that as quickly as possible! [09:54am] daskalou_: lolll [09:54am] dunk_: aha [09:54am] dunk_: what do i actually have to do if i cant afford to let me app grow big ? [09:55am] Wesley_Google: it would be a long conversation, and we would take over the entire chat LOL [09:55am] daskalou_: haha [09:55am] nickjohnson: dunk_: that's too vague a question to answer sensibly. [09:55am] dunk_: i was thinking of have user run a bot of mine on there own [09:55am] nickjohnson: There are scores of articles on this on code.google.com. [09:55am] dunk_: no really i have a state full wave bot [09:55am] ikai_google: dunk_: Open source it [09:55am] dunk_: so i can only query a few users not that much [09:55am] ikai_google: dunk_: Then other developers can use your application [09:55am] dunk_: i already have ikai [09:55am] ikai_google: dunk_: Ah! [09:56am] dangerdave: So: I've been working on integrating Django with Blobstore -- mostly so that we can move screenshots on citygoround.org out of the datastore. I had to file one bug against dev_appserver.py (issue 2515) but there is a relatively straightforward workaround. It looks like clean integration (so you can handle blobs in forms) requires [1] a pretty complex uploadhandler that implements handle_raw_input (because Django doesn't hold on [09:56am] dangerdave: the 'blob-key' when calling receive_data_chunk); [2] a relatively straightforward form field, and [3] some amount of nastiness to deal with the required post/redirect/get architecture for blob posts. (The PRG stuff really leads to less clean Django form handling code.) [09:56am] dunk_: http://cmndbot.appspot.com [09:56am] dunk_: but i was wondering if google would allow such a scheme [09:56am] jeverling: any thoughts on this post? http://groups.google.com/group/google-appengine/browse_thread/thread/43f1b6b44cf6e877 (Java more cost effective than Python due to less CPU time) [09:56am] Wesley_Google: i think that the biggest wins of Django are that: 1) it is a single large stack created by one team, 2) it's very simple to make minor tweaks to the app (very few lines of code to make largeish changes, and 3) the admin is killer [09:56am] ikai_google: jeverling: There are too many factors there [09:56am] dunk_: as in 1 million users all running bots on free services [09:57am] _matt_: Q: any tools out there easier querying/generating reports off data in the datastore? [09:57am] jeverling: ikai_google: ok, I will stick with Python anyway, just asking out of interest [09:57am] ikai_google: jeverling: I don't see Java as necessarily being most cost effective. As soon as you do any kind of off server IO, the speed difference in the language you run is too small [09:57am] nickjohnson: dunk_: You're welcome to make your code open source and encourage people to deploy it, if that's what you're asking [09:57am] dunk_: ah ok nick thats cool [09:57am] ikai_google: jeverling: My *personal* opinion is that Java scales to more developers better, but Nick and Wesley will disagree with me [09:58am] eneekmot: _dunk I tried what the developer mentioned in the link you showed me, but it doesn't work. [09:58am] jeverling: heh [09:58am] ikai_google: jeverling: Code is hard to scale to a bigger team no matter what [09:58am] Wesley_Google: one alternative to Django is TurboGears2/ Pylons: it has a slightly different model that's "best of breed:" take multiple well-known components written by their own teams, i.e., CherryPy, Genshi, SQLAlchemy, etc., and glue them all together for a web framework wiwth interchangeable parts [09:58am] daskalou_: dangerdave: Does your Django session code use a similar approach as cached_db (Datastore+Memcache)? [09:58am] dunk_: eneekmot im used to years of IRC and colors are not done so i dont have any experience with them [09:58am] jeverling: ikai_google: that's true [09:58am] daskalou_: Wesley_Google: And it's supported 100% on GAE? [09:59am] Wesley_Google: ikai is wrong about Java vs. Python [09:59am] dw: non java user here. i was wondering, does the jvm supported by appengine have jitting enabled? [09:59am] ikai_google: yes [09:59am] ikai_google: dw: yes [09:59am] dangerdave: daskalou_: No. My session code was the most braindead simple thing I could do. It basically uses a secure cookie (as in Werkzeug's secure cookie design.) You can't store too much stuff in it, but for a handful of values it works well and can't be tampered with. [09:59am] dw: cool. i wondered if there was any benefit in doing so [09:59am] ikai_google: dw: Yeah, there are some [09:59am] nickjohnson: dw: We also just deployed performance improvements for the JAva runtime [09:59am] ikai_google: dw: It comes for free with the JVM, so why not [09:59am] nickjohnson: dw: see http://googleappengine.blogspot.com/2009/12/request-performance-in-java.html [10:00am] daskalou_: dangerdave: oh ok, interesting [10:00am] dw: nickjohnson: cool, thanks [10:00am] ikai_google: dw: Realistically I don't think you will notice [10:00am] dw: ikai_google: memory usage, executeable heap, ... also the relatively short process lifetimes [10:00am] ikai_google: dw: You really get the benefits of JIT in large data processing scenarios [10:00am] Wesley_Google: daskalou: nope, it isn't (i don't believe), because you *must* use App Engine's datastore API (and not the DJango ORM). also, you don't have access to full Python either (sockets, file, OS access, etc.) [10:00am] dangerdave: nickjohnson: I didn't realize the issue with App- Engine-Patch + zipimport was more about Django's eager loading. That's good to know. [10:01am] daskalou_: Wesley_Google: Which GAE+Django method do you use? [10:01am] nickjohnson: dangerdave: The zip format permits efficient access to individual files, fortunately. [10:01am] ikai_google: dw: The issue is that you applications can be stopped and started at any time [10:01am] ikai_google: dw: So if your app is migrated to a new instance, the JVM will need to warm up again [10:01am] dangerdave: nickjohnson: Makes sense. Thanks. [10:02am] daskalou_: Wesley_Google: my "And it's supported 100% on GAE" was referring to TurboGear2/Pylons [10:02am] ikai_google: dw: Yeah, in theory it's more efficient, but again, anytime you are making off box or off network calls, the benefits of something like JIT become relatively smaller and impactful [10:02am] dw: precompilation sounds interesting [10:02am] ikai_google: dw: Like, for instance, method inlining (one of the best benefits of JIT) might save you a few clock cycles, but a datastore call will take 20ms [10:02am] daskalou_: if we're still taking Q's... drumroll ... When will PHP be supported? [10:03am] dw: ikai_google: i remember reading an old slide deck regarding app engine's security architecture. just thought jit might not be the sort of thing that would be supported in such an environment [10:03am] nickjohnson: daskalou_: It already is, via Quercus. [10:03am] ikai_google: dw: Yeah, well, I'm thinking of another way of saying, it *should* be enabled, but it doesn't matter [10:03am] dw: hehe, ok [10:03am] ikai_google: dw: Much more important is our work to improve cold startup time [10:04am] Jason_Google: OK Everyone, today's chat time is officially over, but a few of us will continue to hang out in the channel. Thanks for the excellent discussions! The next office hour session should take place on the first Wednesday in January, Jan. 6th 2010 from 7-8 p.m. PST. [10:04am] ikai_google: because that REALLY matters [10:04am] daskalou_: nickjohnson: yeah I've seen that... do you know of any problems people are having with doing PHP via Quercus? [10:04am] dunk_: thnx for joining here [10:04am] dw: thanks all [10:04am] jeverling: yes, thanks all [10:04am] nickjohnson: daskalou_: I really haven't looked into it. After switching to Python, I've never looked back [10:05am] daskalou_: thanks Google guys/gals! [10:05am] ikai_google: thanks everybody! We'll see you all soon [10:05am] dangerdave: I notice that the image API now lets you start with a blob key. In the future, do you plan on allowing us to execute_transforms() back _to_ a blob? [10:05am] dangerdave: oops [10:05am] dangerdave: too late [10:05am] Wesley_Google: daskalou: no, sorry, i meant pure Django vs. TG/Pylons... as far as i know, they haven't had the resources to port it. but you can search their group to see if there have been any updates. the last i heard about this, it was about a year and a hal ago http://groups.google.com/group/turbogears [10:06am] daskalou_: Wesley_Google: thanks [10:06am] Wesley_Google: np [10:06am] Wesley_Google: have a great day/night all! [10:06am] daskalou_: Thanks you too -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
