Last Wednesday, the App Engine team held its latest "Chat Time" IRC office hour session. The next session will take place next Wednesday, June 3rd from 7:00-8:00 p.m. PDT in the #appengine channel on irc.freenode.net. A transcript of last week's session and a summary of the topics covered is provided below.
------------------------------------------------------------------------------------------ - There are plans to add Java support for increasing the time limit for URLFetch requests before timeout to mirror the Python SDK [9:08] - Does it make sense to add a 95% metered billing option so applications aren't shut down if they hit a peak? Our current billing setup allows developers to easily increase their daily budgets when they expect higher traffic and then re-adjust, but there is no "metered" option. (File an issue if this is important to you.) [9:10-9:16] - XMPP API progress "going good" [9:11] - Local data viewer for Java coming with next SDK release [9:17] - Discussion on issue 1378, request for ability to specify content- encoding header [9:20, 9:25, 9:27] - There are no current plans to introduce a UUIDProperty, but developers can roll their own if it's important to them [9:29] - There's no way to delete applications currently, but if you need more applications, just post your request in the discussion group [9:32] - There is no way to retrieve more than 1,000 results with a single query but if you're trying to export your data from the datastore, you can use the built-in support (http://code.google.com/appengine/docs/ python/tools/uploadingdata.html#Downloading_Data_from_App_Engine) or build your own solution using remote_api [9:33, 9:37] - Q: Is having a single app that is used by 3-4 other "services" allowed by the TOS? A: It depends on the implementation. Building a service (e.g. stats) on App Engine that is then used by various other applications is fine, but using multiple application IDs to implement the functionality of a single application will violate TOS section 4.4. [9:36, 9:38] - Entities are memcached but queries are not, so it's best to optimize your application where possible to fetch entities directly by key [9:41, 9:44] - HTTPS is not supported for non-appspot.com domains because of a lack of support for SNI. With TLS 3.1, which is already available in the newest versions of Safari, Firefox, Internet Explorer, and Opera, widespread support for SNI is an eventuality, but until then it's unlikely to get HTTPS support for non-appspot.com domains. [9:48-9:49] ------------------------------------------------------------------------------------------ [09:02am] scudder_google: Hi all, I'd like to start another installment of App Engine chat time, where Google engineers are on hand to answer your questions and listen to concerns [09:03am] nickjohnson: *tumbleweed* [09:03am] knoonan: Hi guys, for Ruby on App Engine, I think it would be very useful to have a page with a whitelist of gems (and plugins for Rails) which have been found to work on GAE/J. [09:05am] maxr_google: That sounds useful. I think I see ribrdb logged in here. [09:05am] scudder_google: knoonan: interesting idea, the best way to start might be to create your own list somewhere and allow others to collaborate with you on it [09:05am] maxr_google: He's done a lot of the GAE Ruby work. [09:05am] nickjohnson: knoonan: That seems like a good idea, but probably best on the appengine-jruby wiki? [09:05am] Wooble: Is there any truth to the rumor that henceforth people spamming the main GAE google group with requests to activate their Java accounts will be redirected to the Java group and moved to the back of the queue to be activated? [09:06am] bmckinlay: I have an issue with URLFetch on Java. My application contacts a web service that sometimes needs more time to respond to my queries than the ~4.5 seconds that app engine allows. In the Python SDK, there is an option to increase this limit. Any possibility of a similar option being added in Java? [09:06am] nickjohnson: Wooble: If there is, nobody's told me. ;) [09:06am] knoonan: Thanks guys, the App Engine JRuby wiki sounds like the place. [09:06am] schwardo_google: bmckinlay: it's definitely something that we plan to support. do you know if it's been filed in the issue tracker yet? [09:07am] ribrdb: only a couple of people can edit the wiki right now though. [09:07am] bmckinlay: schwardo_google: yes... http://code.google.com/p/googleappengine/issues/detail?id=1559 [09:07am] ribrdb: knoonan: do you know of a list of gems anywhere that do or do not work? [09:08am] knoonan: @ribrdb No, I don't know of an existing list. That's why I think it's time to start one...! [09:08am] schwardo_google: bmckinlay: ok great, i'll accept it to indicate that it's something we plan to support, and we'll rely on the # of stars to help prioritize it [09:09am] knoonan: ((I did a lightning talk at the Euruko conference in Barcelona, Spain, on May 10th, about App Engine. I hope I got some Rubyists interested.) [09:09am] bmckinlay: schwardo_google: thanks! This would be very useful to me. [09:10am] drawlabs: any plan for monthly pricing? Sometimes I wish I could over pay on slow months with the insurance of not breaking too high into billing on large months. or is this just a dev chat? [09:10am] brett-google: fine to talk about billing too, of course [09:11am] brett-google: we've heard that request from other people, too; definitely considering that option going forward [09:11am] mcxx: how's the XMPP API going? for me, that's the feature I'm looking forward to most [09:11am] brett-google: drawlabs: one question is, do you mean you want your budget to carry over, or your actual payment? [09:12am] brett-google: mcxx: going good. =) nothing to announce but it should be soon according to the roadmap [09:12am] drawlabs: thanks, yes more like an insurance cloud. Every indie developer I know is scared of the cloud on that high end one day or process that ends up costing a few grand. [09:12am] drawlabs: monthly, no carry over to make it economical [09:12am] brett-google: hrm [09:12am] brett-google: yeah because right now when you set your budget that's a maximum spend, it's not an actual paid amount [09:12am] mcxx: brett-google: superb :) [09:12am] brett-google: so you can at any time raise your budget to fit a big day [09:13am] brett-google: but then the next day drop your budget again in case you're worried about costs [09:13am] brett-google: does that make sense? i'm confused about how they're scared about one day costing a few grand [09:13am] drawlabs: true, but what I am saying is some part of big days would be shared cost because people pay like an insurance for that. [09:13am] brett-google: ah okay so this is more getting into 95%-like billing [09:13am] brett-google: where you don't have ot pay for extreme peaks [09:14am] drawlabs: yep [09:14am] nickjohnson: It sounds like you're asking for "slashdot insurance"? :) [09:14am] drawlabs: hehe yes [09:14am] brett-google: well traditional hosting providers effectively have that, nick, with 95% metered billing [09:14am] nickjohnson: right [09:15am] drawlabs: I know it would be useful for some people that have had fears, so they stay at trad hosting for that. [09:15am] brett-google: in my personal opinion, though 95% can help with peaks in some cases, it's actually bad for customers because it overcharges the rest of the time [09:15am] brett-google: if you have a single 95% peak, then you're billed at that level for the whole month [09:15am] brett-google: even if your average is zero [09:15am] brett-google: so our current billing model lets you change your budget on a daily basis depending on your current needs [09:15am] drawlabs: true, it might be one way to get people though, I know it would help me moving people onto it. [09:15am] Wooble: For a service where the cost varies so much depending on what your app does, I can't imagine how you'd begin to even calculate the monthly costs to charge... [09:16am] brett-google: yeah that's a fair point [09:16am] drawlabs: I think what you have is good or better than most, sry to overtake with billing [09:16am] rogerh: are there any plans to allow access to the timestamped fields inside bigtable? [09:16am] brett-google: wooble: indeed it's a complex idea [09:16am] brett-google: drawlabs: well i worked on billing so i'm happy to talk about it =) [09:16am] brett-google: rogerh: of previous cell version snapshots? [09:16am] nickjohnson: rogerh: What do you mean exactly? [09:17am] brett-google: nick: i think he means the version axis [09:17am] nickjohnson: ah [09:17am] Ashar__: did i missed the "session" ?! [09:17am] rogerh: brett/nick: yeah, trying to figure out how best to be able to keep track of previous verisons of data we store [09:17am] nickjohnson: Ashar__: No, you're in it. [09:17am] Ashar__: nice [09:17am] oizo: hi, can we expect java local data viewer in the next SKD? Really, its very necessary for many developers.. [09:17am] maxr_google: ozio: Yes, you can expect a java local data viewer in the next SDK. [09:18am] nickjohnson: rogerh: Even if it were available, Bigtable's model is such that you can't rely on it sticking around. If you need to be able to reliably access old versions of data, you're best to implement it in user code. [09:18am] maxr_google: It was supposed to have made the update that went out last week but we missed the cutoff by just a little bit. [09:18am] drawlabs: python question, any way in the future plans to specify a python version you want to target? Or Google supporting versions of python that you can specify per app? [09:18am] AsharLohmar: i had a problem these days building a data structure in java [09:18am] AsharLohmar: using JDO [09:18am] brett-google: rogerh: indeed nick's right; especially considering the sugar we put on top of bigtable to support multi-row transactions, it gets more complex [09:19am] brett-google: but it would be cool to have built-in support for Datastore snapshotting [09:19am] brett-google: which we coudl do under the covers more efficiently than just user code [09:19am] maxr_google: yes, built-in support for Datastore snapshotting would be super cool [09:19am] AsharLohmar: i had a situation like classroom -> students-> grades, built with 1-n bidirectional relations [09:19am] maxr_google: AsharLohmar: Tell me more about your JDO problem. [09:20am] AsharLohmar: on the parrent classes i've used the "mabedBy" [09:20am] rogerhu: max/brett/nick: great, thanks. indeed that would be a great feature to have [09:20am] scudder_google: drawlabs: if we were to add additional versions of python, you would likely specify a different runtime version. However different Python versions are not on our roadmap at this point [09:20am] dieterk: I submitted this problem a while ago: http://code.google.com/p/googleappengine/issues/detail?id=1378 [09:20am] brett-google: drawlabs: we've heard requests for python3k; it would have to be treated as a different runtime version (so in app.yaml you'd specify py3k instead of python) [09:20am] AsharLohmar: th problem was in this situation ... i had the classrom and the students defined [09:20am] oizo: maxr_google: good, thanks! And when it will be released if not secret? [09:20am] AsharLohmar: i could add grades to students [09:20am] brett-google: so yes there's support for doing something liek this, but there's no py3k right now [09:21am] drawlabs: brett: thanks good to know there is support for it, hoping for 3000 soon [09:21am] AsharLohmar: but i didn't manage to query the grades for a student [09:21am] knoonan: With GAE/J and Ruby, does anyone have a neat solution for zipping files to overcome the limit of 1000 *static* files? (Ola Bini and others showed how to tackle the code-file limit.) [09:21am] dieterk: I have two questions: 1. can somebody state if there is at least a will to fix this? 2. why did somebody move this issue to Java. My appengine is running on python. [09:21am] maxr_google: oizo: The next release isn't a secret but we don't have a good estimate of when it will be available. [09:22am] Wooble: knoonan: I don't believe that's possible; you almost certainly need to serve them out of the datastore. [09:22am] scudder_google: dieterk: looks like there was some confusion, as you are serving Java applet code, I'll recategorize [09:22am] maxr_google: AhsarLohmar: Did your query return 0 results? [09:22am] maxr_google: Or was there some other problem with it? [09:22am] AsharLohmar: i recived an message saying that the student field from tha grade that i tryed to query is maped to Student, and tha paramater i've sent has key from tha classroom [09:22am] knoonan: @wobble, there already are well-documented solutions for Python and for Java itself, using Zip libraries. [09:22am] dieterk: scudder_google: yes. but Content-Encoding is all about the HTTP Headers and has nothing to do with the appengine at all [09:23am] AsharLohmar: my real classes war Country->city->taxiCompany [09:23am] maxr_google: AsharLohmar: You're passing a Key in as a parameter to the query? [09:23am] Wooble: the files aren't static anymore if your code can unzip them. [09:23am] AsharLohmar: no, i use strings [09:23am] AsharLohmar: gae.encoded-pk or something [09:23am] maxr_google: Right. [09:23am] dieterk: scudder_google: also...may I ask somebody from the appengine team to comment on the issue - so I get to know the mind of the appengine team about this issue. [09:23am] maxr_google: So, there's a good chance that error message is accurate. [09:24am] mib_5nyoz6qq: Any comments why I'm getting ApiDeadlineExceededException for querying one entity? [09:24am] schwardo_google: dieterk: yeah, it's actually not relevant whether it's marked as python or java -- it's the core infrastructure that would need to be changed [09:24am] bmckinlay: [A[B[B[B[B[B[B[B[B[B/quit [09:24am] maxr_google: The way to check is to do System.out.println (KeyFactory.stringToKey(theKeyString)) [09:24am] AsharLohmar: i know that the message was telling the truth, i didn't find anty workarounds thought [09:24am] nickjohnson: mib_5nyoz6qq: Occasional timeouts will happen, even for simple queries. They should only account for a very small fraction of your total requests, though [09:24am] maxr_google: That will tell you if the String you're passing in identifies a Country or a city. [09:25am] schwardo_google: dieterk: I believe we were only aware of gzip and compress being used as Content-Encodings, -- I wasn't aware that custom encodings were being used in this way [09:25am] AsharLohmar: exactly, i know that he is right, my real problem was how can i query then [09:25am] knoonan: @wobble, sure: the code is unzipping the content from zip-files stored in the file-system: it works for Python and Java, I wonder if someone's done it successfully for JRuby... [09:25am] schwardo_google: will need to read up to see if that's even allowed in the spec [09:25am] mib_5nyoz6qq: nickjohnson: no, I'm getting this error for more than 15 hours constantly [09:25am] dieterk: schwardo_google: core change? So it is unlikely to happen in the near future or ever? Please comment on the issue if you do not want to allow it since it is a security risk or else? [09:25am] nickjohnson: mib_5nyoz6qq: For every fetch of a particular entity, or of all entities? [09:25am] maxr_google: If you want to filter by city you'll need a String that identifies a city, not a country. [09:25am] scudder_google: dieterk: and schwardo, we do not currently allow the content-encoding header to be set/modified [09:26am] AsharLohmar: because i save the city as chil of country ofcource it has country in it [09:26am] AsharLohmar: also inside the taxi object [09:26am] AsharLohmar: the city has the same key [09:26am] nickjohnson: And have you determined what your failure rate is? The logs page only shows errors by default, so it can give the impression everything is failing when only a very small proportion are. [09:26am] ribrdb: knoonan: you can jar your code or your gem files, and put it in WEB-INF/lib [09:26am] mib_5nyoz6qq: nickjohnson: for simple query with range set for 1 entity, I described my problem here: http://groups.google.com/group/google-appengine-java/browse_thread/thread/9720cb2b5cf45947 [09:26am] maxr_google: City and country should different keys since they are different objects. [09:27am] TheBashar: Quick question. If I create a new entity with a key_name MyEntity(key_name="abc") and I put() it, if there is already an entity of the same kind with the same key_name in the datastore, will it get overwritten? [09:27am] schwardo_google: dieterk: i don't think there are any insurmountable security implications, but yes, it's a lower level change than most and would probably take longer to roll out [09:27am] maxr_google: This might be easier to debug if you post your model classes and your query code to ghe group. [09:27am] nickjohnson: mib_5nyoz6qq: As I said - what proportion of the time does this happen? And is it only for a specific entity/query? [09:27am] nickjohnson: TheBashar: Yes. [09:27am] maxr_google: Then I can drop it in to my test framework and see exactly what is going on. [09:27am] TheBashar: Thank you Nick! [09:27am] Aaron_10: I need some help, I uploaded a app to google which has django. I need to use the admin area but I can't for some reason. [09:28am] Aaron_10: the site is : dev.mobileconsultinginc.com/admin [09:28am] AsharLohmar: thanks maxr_google , i'll add them on the appengine group [09:28am] dieterk: schwardo_google: scudder_google: also I don't understand the argument about core change since you allow to set some http headers? I really would need Content-Encoding for only a single file. I didn't think this to be an issue since I do set my Content- Encoding in my many other hosting setups on a daily basis. Seems to be no security risk for them. [09:28am] drawlabs: datatype question: any plans for a UUID type or UUIDProperty for datamodels or as the key maybe? Currently we store them in strings but lots of apps are distributed and linked via UUID. Or do you recommend using the name property of the db.Key field? What is the best way to use UUID when ints as keys aren't needed? [09:29am] knoonan: @ribrdb I know you can JAR your code and gem files with JRuby, but what about zipping web-content files to exceed the static-file limit of 1000...? [09:29am] gvanrossum: Hey all, Guido van Rossum here [09:29am] nickjohnson: drawlabs: There's no plans to introduce a UUIDProperty, but you can write your own. If you're using UUIDs as the primary key for your entity, certainly, using it in the key name makes sense [09:29am] nickjohnson: As long as you make sure the first character isn't numeric [09:30am] nickjohnson: Hi Guido. Do you have an App Engine question to ask? [09:30am] moraes: hehehe. yes, this is the place. [09:30am] rmalik: hello Guido! [09:30am] schwardo_google: dieterk: i'll respond with more detail on the issue itself -- sorry about mislabelling it Language-Java [09:30am] drawlabs: nickjohnson: thanks [09:30am] dieterk: gvanrossum: hi...I am Python fan for ever. Thanks! [09:30am] gvanrossum: Hi Nick, no, I thought I'd just make some waves [09:30am] nickjohnson: You seem to be doing that. [09:30am] ribrdb: knoonan:ah. well you could write some ruby code to serve the files out of a jar or zip. [09:30am] rmalik: is there a way to delete an existing application that you are no longer developing? [09:31am] rogerhu: right now a major limitation is the 1 inequality property, which makes it hard to do date range scan types. noticed in google finance you can graphically view stock performances http://www.google.com/finance?q=INDEXDJX:.DJI,INDEXSP:.INX,INDEXNASDAQ:.IXIC does this mean that multiple table scans are required to implement this type of feature? [09:31am] nickjohnson: rmalik: Not currently, no. [09:31am] drawlabs: guido came to support python 3 on GAE [09:31am] nickjohnson: drawlabs: Single-handedly, while blindfolded, in his sleep! [09:31am] yowgi: Hello. Are there any plans to provide support for the delivery of static files that have been uploaded or created by the app? Putting images in the database is always quite inefficient... [09:31am] nickjohnson: rmalik: If you just want it off the list of apps, you can create a dummy account and make it the sole administrator of the app, but it'll still count towards your 10 app limit [09:31am] gvanrossum: sorry, no py3k support planned yet [09:32am] maxr_google: rogerhu: You can do a 'between' query for a single property though. [09:32am] TheBashar: Are there any plans to support 2.6 on GAE in the near future? [09:32am] maxr_google: rogerhu: What sort of query are you looking to write? [09:32am] dieterk: schwardo_google: no problem. I really like appengine in every way but this really is somehow a big issue. With my other hosting services if a feature is missing I just pay more and they fix it. I guess we cannot settle this with cash? [09:32am] dieterk: [09:32am] nickjohnson: yowgi: Static files are already served directly, not from the datastore. [09:32am] rmalik: nickjohnson: yeah i'm just trying to take it off from my 10 app limit [09:32am] drawlabs: gvanrossum: yeh I asked, the mechanism is there but maybe soon. [09:32am] nickjohnson: rmalik: If you run out of apps, you can ask on the group for more. [09:33am] rogerhu: max: like trying to pull up a list of more than 1000 people who have registered to vote between certain dates or signed up online between a few months. [09:33am] yowgi: nickjohnson, even user-uploaded images? i mean HTTP upload, think blog post attachments. [09:33am] rmalik: nickjohnson: the problem with one app is that i originally set authentication against all google accounts versus a google apps domain i created later [09:33am] nickjohnson: rmalik: What do you want to do with >1000 records in a single query? [09:33am] AsharLohmar: (when )will we be available to delete apps from our account ?! [09:33am] rmalik: nickjohnson: and now i want to switch that authentication [09:33am] gvanrossum: TheBashar: no 2.6 support planned either, though it will be a lot easier to pull off. I'm waiting for the rest of Google to adopt 2.6 first though. [09:33am] knoonan: @ribrdb I believe it should be feasible with JRuby to serve web-content out of jars / zip files, I'm just wondering if anyone has done it. I'm especially concerned about how it might be implemented with Rails (or Merb)... [09:34am] TheBashar: @GvR: Thanks! [09:34am] gvanrossum: drawlabs: I don't understand your remark [09:34am] nickjohnson: yowgi: Oh, right. That is coming - see the roadmap 'service for storing and serving large files' [09:34am] maxr_google: select from Person where registrationDate < x and registrationDate > y [09:34am] rmalik: nickjohnson: I think you mistaken the >1000 records concern with someone else's question [09:34am] yowgi: nickjohnson, thanks. [09:34am] maxr_google: that should work fine since you're only filtering on a single property. [09:34am] brett-google: rmalik: indeed we've heard people say they want this before; right now it's not that easy for a variety of reasons, but it's on our radar [09:34am] nickjohnson: rmalik: Unfortunately, the only way to change the authentication mode is by creating a new app, currently [09:34am] nickjohnson: rmalik: my apologies [09:34am] dieterk: schwardo_google: scudder_google: thank you for our help! please also contact me by email if you have any questions - I could provide much more information on why not havin Content-Encoding is such a big problem for me. [09:35am] nickjohnson: I meant to address that at rogerhu [09:35am] DocSavage: Hello [09:35am] drawlabs: gvanrossum: i asked about py3k on gae. dev responded with a reply that said they had the ability to do versioning but no plans on adding python 3 yet. [09:35am] rmalik: nickjohnson: i'm glad it's on the radar and i understand it might not be a high priority at the moment. p.s. i'm REALLY glad you guys added cron jobs [09:35am] rogerhu: max: what if there are more than 1000 records that get pulled out? then i need to pull out the next 1000 with a __key__ query [09:36am] nickjohnson: rogerhu: What do you want to do with >1000 records in a single query, though? Surely not display them all to the user at once? [09:36am] drawlabs: architecture question: say you have an app that runs on GAE. Then you have 3-4 services that it uses (say a user, stat and content service). Is that allowed under the TOS to have the site, and each service on a different app or do all services also have to be hosted under the same app? the services would be shared among many apps not just one. But is that seen as getting out of billing? [09:37am] gvanrossum: drawlabs: the versioning capability is for the library, not for the language version; py3k language support (or even 2.6) requires much more work and would probably require you to say "runtime: python3.0" or "runtime: python2.6". [09:37am] rogerhu: nope, but what if we want to export it to a .csv file? [09:37am] nickjohnson: rogerhu: http://code.google.com/appengine/docs/python/tools/uploadingdata.html#Downloading_Data_from_App_Engine [09:37am] brett-google: drawlabs: that's a difficult question because it really depends on how it's implemented [09:37am] nickjohnson: That's the built-in support; you can also write your own using remote_api and __key__ queries if you want. [09:37am] brett-google: for example, if you have a generic stats service that's used by your app and many other people's apps, then yeah, that's fine [09:37am] drawlabs: gvanrossum: thanks, yes I imagine it is quit a bit of work to support on this scale. I hope for it in the future. [09:37am] mun: does bdbdatastore understand ORDER BY <attr> DESC? [09:38am] nickjohnson: mun: It certainly ought to [09:38am] brett-google: but if you have one app for storing data and another app that just does URLfetch calls to retrieve that data, then that's more of a single use; that would fall under section 4.4 [09:38am] mun: somehow i can get it to print by 'ORDER BY...' by not 'ORDER BY ... DESC' [09:38am] nickjohnson: If it doesn't, that's a bug [09:38am] rogerhu: nickjohnson: thanks . just trying to figure out a good way to provide tiered-level permission based access to the data set. the remote_api makes it hard to do that [09:38am] nickjohnson: mun: Might want to take this offline or wait until the end of the dev chat to avoid too much clutter. [09:39am] DocSavage: I had a stub-level question on how a cookbook recipe works -- http://tinyurl.com/rbgfq9 [09:39am] picalolabu: good day. python question: do i need to install PyCrypto separately for my dev server? [09:39am] mun: nickjohnson, yeah sure [09:39am] nickjohnson: rogerhu: Yes, you definitely don't want to use remote_api for anything other than administrators. If you want your users to be able to download CSV files, I would suggest using __key__ queries, with multiple requests if necessary. [09:39am] drawlabs: brett: yes that is the item I am talking about 4.4, we have multiple games for instance that report data. Each in different places. Some use one or more of the services. But it isn't a single app. This is where differnet billing would help maybe, to do stuff like this without seeing as getting out of billing? [09:39am] gvanrossum: picalolabu: yes [09:39am] oizo: bdbdatastore question: if this db emulate production db, may i estimate datastore size with indexes, (for example 10000 entites), based on the size of bdfile bdbdatastore on the local disk (with this 10k entities)? [09:39am] nickjohnson: picalolabu: If you want to use PyCrypto, yes. [09:40am] gvanrossum: DocSavage: what's the question about that page? [09:40am] nickjohnson: oizo: I would not rely on BDBDatastore's index size reflecting space consumption on production App Engine [09:40am] brett-google: drawlabs: we'd like to provide better native support for 'multitenant' applications, so you can run multiple siloed datastores and applications under a single app_id [09:40am] rogerhu: nickjohnson/maxr: then does it become difficult in app engine to do query date ranges & dumping out more than 1000 people? (i guess rather than using the __key__ > last key name i could retrieve the next 1000 entries by using an offset)? [09:40am] picalolabu: nickjohnson: gvanrossum: thanks - but don't upload the PyCrypto libraries with my app. Correct? [09:40am] nickjohnson: It may - it's probably representative - but don't rely on it. [09:40am] nickjohnson: picalolabu: Correct. [09:40am] brett-google: this would help a lot with cases like this where you have lots of code and DB sharing but some fundamental differences [09:41am] DocSavage: The cookbook recipe gives "seamless memcaching" of puts/gets/queries. But I'm trying to figure out how it expires query caches after a put on an entity. [09:41am] nickjohnson: rogerhu: You can filter by date range instead of key range, as long as you make sense to exclude duplicates with the exact same timestamp. [09:41am] picalolabu: nickjohnson: thx [09:41am] nickjohnson: DocSavage: It doesn't need to expire caches; memcache does that [09:42am] drawlabs: brett: that would be good. However as long as it doesn't add overhead. Let's say for architectural reasons you'd want them under different locations or virtual endpoints. Like right now we have services hosted for user, content, game, and stats all in differnet places and can get data async/parallel. [09:42am] DocSavage: Oh, you mean there's some amount of time the cache is stale after I do an entity put? [09:42am] brett-google: sure well, you can host an app_id on many domains and then use the 'host' header to differentiate the task to do [09:42am] brett-google: i've seen people do this [09:42am] nickjohnson: drawlabs, brett: It seems to me that drawlabs is talking about horizontal separation; what we see abuse around mostly is vertical separation, with sharding to different backends [09:43am] brett-google: well nick i think he's talking about both [09:43am] nickjohnson: DocSavage: No, it does the memcache put when you do the datastore put [09:43am] brett-google: like each game would be horizontally separated [09:43am] brett-google: but then the stats and user services would be vertical [09:43am] rogerhu: nickj: but ultimately the number of queries that get dumped back is 1000. so i'm trying to figure out how to retrieve the next batch and still doing a date range query (inequality property) + getting the next 1000 batch [09:43am] drawlabs: brett: I hope to see more on this soon maybe, because since it is unclear. Yes talkign more about horizontal not just breaking it up for getting out of billing. [09:43am] picalolabu: python and gdata: i added a function to read blogger via gdata (news on my home page). I cache the results but I have started to see very inconsistent processing times (i.e. some requests take 3000ms others 60) - is this normal? [09:43am] rogerhu: number of results that get dumped back that is [09:44am] nickjohnson: rogerhu: If you do a "SELECT * FROM MyEntity ORDER BY date", you can take the last date returned and do "SELECT * FROM MyEntity WHERE date > :1 ORDER BY DATE" [09:44am] brett-google: drawlabs: i guess the way i see it is it makes sense to have one app_id for al the common stuff [09:44am] brett-google: content, users, stats [09:44am] brett-google: and then one app_Id per game [09:44am] DocSavage: If I do a Model put, then ask in a query for all entities in a Model that meet some criteria, how is that query memcached? Do I have a misunderstanding in how queries are handled at the stub-level? [09:44am] brett-google: but i think it makes less sense to have one app_Id for each individual common service [09:44am] scudder_google: picalolabu: are those times for the urlfetch request to the Blogger API? [09:44am] brett-google: does that distinction make sense? [09:44am] nickjohnson: DocSavage: Queries aren't memcached, only the entities themselves. So you'll see the most benefit from puts and gets. [09:45am] jcgregorio: rogerhu: http://google-appengine.googlegroups.com/web/efficient_paging_using_key_instead_of_a_dedicated_unique_property.txt [09:45am] lidaobing: [java] if I want to count the rows with JDO, is this also limited to 1000? [09:45am] nickjohnson: lidaobing: Yes [09:46am] jcgregorio: rogerhu: That writeup from Ryan should cover the use case you are talking about, using a bookmark to resume the query for the next 1000 items [09:46am] rogerhu: jcgregorio: great thanks i remember seeing that link, thanks for the reminder [09:47am] drawlabs: brett: thanks that helps but it would be great if this was more clear on the ethics there in the future. We are actually discussing what to do now and some are suggesting mt grid service at $20 a pop for each. The limit is 1TB but the drop off is a scary 2.30 per GB. So you can see how billing is kindof dictating how things are architected in many cloud projects probably. ok enough billing [09:47am] picalolabu: scudder_google: i don't know, they are the times that appear in the log files with the "... high CPU ..." warning - when i test from my computer it is always under 200ms [09:47am] DocSavage: nick: Oh, jeez. Queries *aren't* memcached. OK, wasn't clear because the recipe says "the cache is updated through - queries", but I see this meant the individual entities are memcached on query. [09:47am] nickjohnson: right [09:47am] knoonan: Has there been any progress on using certs other than Google's, or domains other than XYZ.appspot.com, for HTTPS connections? (I remember talk of a workaround using the latest browsers...) [09:48am] nickjohnson: It could probably be improved now that key-only query support is available to do a key-only query and retrieve entities from memcache and the datastore as appropriate, but I'm not sure how much that would improve things. [09:48am] brett-google: drawlabs: yeah makes sense [09:48am] brett-google: knoonan: the issue here is still HTTPS's support for SNI [09:48am] brett-google: "server name identification" [09:48am] oizo: if app_id is the part of all primary keys, is more shortly app_id for very big db (billions entities) saves my budget for data store? [09:49am] lidaobing: is there any progress (or any possibility) on issue 1269: unique static ip per app/account http://code.google.com/p/googleappengine/issues/detail?id=1269 [09:49am] brett-google: the issue is that HTTPS requires the reverse IP address of the host to match the certificate hostname [09:49am] brett-google: so each website with HTTPS must have a unique IP address [09:49am] nickjohnson: oizo: In theory, yes. In practice, I think you'd be hard pressed to tell the difference. [09:50am] brett-google: SNI fixes this by putting the hostname in the SSL handshake; so then virtual hosting providers can use a single IP range to serve multiple hosts [09:50am] nickjohnson: brett-google: I thought the real problem was that you have to establish the SSL session _before_ specifying the hostname, but you can't tell what hostname they want until _after_, in the virtual hosting scenario. [09:50am] brett-google: that's the technical problem [09:50am] brett-google: but the only way to get around that is to assign a separate IP for each hostname you want to serve HTTPS on [09:50am] nickjohnson: right [09:50am] brett-google: the good news is that TLS 3.1 (which is in safari, firefox, ie8, and opera) supports SNI [09:50am] scudder_google: picalolabu: it sounds like the CPU usage warnings are from a longer than expected wait time for the response to get back from the Blogger API, which occasionally happens [09:50am] nickjohnson: I didn't know SSL certs encoded the reverse-ip, though. [09:50am] brett-google: that's how they work [09:51am] brett-google: the certs odn't have it, but the policy of browsers is to do the reverrse lookup [09:51am] nickjohnson: Oh, right. [09:51am] brett-google: the recent ie6 forced upgrade to ie8 has helped a lot [09:52am] brett-google: so maybe we'll have more wide-spread SNI support [09:52am] brett-google: but until then it's likely unrealistic to get HTTPS support beyond https://app_id.appspot.com [09:52am] brett-google: the good news is that now you can use any Google Apps domain to log in on appspot.com [09:52am] brett-google: so all app engine apps can use it [09:52am] picalolabu: scudder_google: could the inconsistency be that some users see the cache and others don't, hence the need to hit blogger again? or would the cache be seen by everybody until it is destroyed? [09:53am] brett-google: lidaobing: nothing to report now; this bug falls into a similar realm as HTTPS support actually [09:53am] brett-google: we'd need a separate IP per app_id [09:53am] rogerhu: am i correct to assume also that there's no current way to have multiple versions of a datastore for one application? (i know you can have different app versions) [09:53am] knoonan: @brett-google and @nickjohnson: thanks! I guess we just have to wait for IE8 to become the norm... [09:53am] brett-google: that's very costly and significantly increases the complexity of our network infrastructure [09:53am] nickjohnson: rogerhu: That's correct. [09:53am] nickjohnson: Don't thank me, I'm learning from brett-google too [09:53am] mun: nickjohnson, i'm now using bdb and it contains about 150k entities. but when i run a .count() on it, it still times out. is it typical? [09:54am] nickjohnson: mun: Yes, because BDBDatastore takes the same approach to counting queries as App Engine does, and thus has the same limitation [09:55am] lidaobing: brett-google, bad news [09:55am] picalolabu: brett-google: could you please expand on "... any Google Apps domain to log in on appspot" - im not sure I understand the implication [09:55am] DocSavage: nick: After installing bdbdatastore, I'm getting the following error in socket_apiproxy_stub.py, line 63, in closeSession: self._sock.close() ... was wondering if there's a Google group or someplace bdbdatastore users could exchange info or ask questions. [09:55am] mun: nickjohnson, you mean they take the same approach as the app engine on the dev server or on the production server as well? [09:55am] rogerhu: nick: great thanks. this has all been very helpful [09:55am] nickjohnson: mun: On the production server. [09:55am] scudder_google: picalolabu: I'm not sure what your design looks like, are you memcaching the results of the Blogger API query? Do you set an expiration, etc? An infrequent long request could also be a new instance of your app being spun up, imports need to be done again, etc. [09:55am] DocSavage: nick: The error was 'NoneType' object has no attribute 'close' [09:56am] brett-google: picalolabu: if your app has auth restricted to a single google apps domain (i.e., users @example.com), then you can serve content to them over HTTPS on app_id.appspot.com [09:56am] nickjohnson: Counting the results of a query requires iterating over them all. This is actually the case for the vast majority of queries in an RDBMS, too, they just hide the work. [09:56am] brett-google: and they can log in thorugh appspot.com [09:56am] brett-google: before it was the case that these users could not log in through appspot.com [09:56am] mun: nickjohnson, so does that mean if my queries time out on bdb, then they'll time out on the production server as well? [09:56am] brett-google: they could only log in on domains that were under example.com [09:56am] nickjohnson: DocSavage: Known bug that I'm fixing this evening. It happens when there were no datastore requests in a request. You can fix it by adding an "if self._socket:" guard clause. [09:56am] mun: nickjohnson, well, i've set the limit of the count to a small value-- like 10. [09:56am] DocSavage: nick: thanks! [09:57am] nickjohnson: mun: Not for certain, but they're definitely correlated [09:57am] nickjohnson: mun: And it times out when the limit is set to 10? [09:57am] mun: nickjohnson, yeah that's right [09:57am] mun: i'm doing .count(10) [09:57am] nickjohnson: mun: I'll look into it. [09:57am] mun: nickjohnson, i see. thanks [09:59am] picalolabu: scudder_google: yes, i was referring to the blogger api results being memcached - but i had not thought about the new creating a new instance. thx [09:59am] picalolabu: brett-google: got it - thx [10:00am] DocSavage: Regarding different apps providing services -- does each app_id get its own allocation of memcache and memory because each app has its own instances? [10:00am] nickjohnson: mun: Just checked the source - stupidly, I'm not giving up if it reaches the count specified. Easy fix. [10:01am] Aaron_10: I went to the django channel and the told me that the django admin area uses django models which google apps dosen't allow, so they said to ask here for a work around the problem. [10:01am] Aaron_10: cause I am having trouble making a admin user for the website django admin area. [10:01am] Aaron_10: which is called superuser. [10:02am] mun: nickjohnson, you mean it keeps searching even beyond n many have found? [10:02am] gvanrossum: Aaron_10: have you tried the Google app engine helper for Django? [10:02am] nickjohnson: mun: Yes. [10:02am] mun: nickjohnson, does that mean fetch has that behaviour too? [10:02am] Aaron_10: ya, it didn't do much. [10:02am] gvanrossum: Aaron_10: http://code.google.com/appengine/articles/appengine_helper_for_django.html [10:02am] Aaron_10: are you saying that should be the work around? [10:02am] gvanrossum: Aaron_10: IIRC it supports the Django admin interface [10:03am] gvanrossum: Aaron_10: yes [10:03am] knoonan: @Aaron_10 try AEP (App Engine Patch): the sample application includes an admin interface out-of-the-box. [10:03am] jago1: I have a problem with wordle.net hosted on the Google appengine [10:03am] nickjohnson: mun: No, fetch only returns the requested number of entities [10:03am] Aaron_10: ok, I will look more into it. I tried it but I might of done something wrong. [10:03am] nickjohnson: gvanrossum: Aaron is trying to use the Django admin interface, which apparrently uses Django models [10:04am] mun: nickjohnson, okay [10:04am] nickjohnson: I'm not aware of anything that lets you use unmodified Django models in GAE, though [10:04am] jago1: visiting the site works...but when I open the applet it blocks for over 30 sec. before the applet starts. It seems it is confused finding the appengine: http://img166.imagevenue.com/img.php?image=38755_http_traffic_122_43lo.jpg [10:04am] scudder_google: I'd like to close the official chat hour, but some of us will still be here after, so don't hold back on questions. Thanks all! [10:04am] gvanrossum: Aaron_10: nickjohnson: I take it back, the article says it does not use the Django admin site [10:04am] gvanrossum: So, try app-engine-patch [10:04am] rmalik: is there a way to use my appspot domain as a staging environment and then push to a custom domain later? [10:05am] Aaron_10: yep nickjohnson: the problem I hear that google app engine won't allow it to use django models. [10:05am] Aaron_10: so I need a work around with that. [10:05am] moraes: Aaron_10: django admin never worked on app engine. [10:05am] Aaron_10: so then now what do I do? [10:05am] mib_6yy52j: hi [10:05am] Aaron_10: haha [10:05am] nickjohnson: rmalik: yes, certainly. You can use multiple major versions - the main one will serve on your domain, any others will serve on theirname.latest.yourapp.appspot.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
