Re: [appengine-java] Re: Improved query planner in SDK 1.6.0
What is Siena? zig zag is built into the datastore and always on, if it is not working it is a miss-match between the indexes available and the query executed. The index error will tell you what index is needed. Can you give me your app id and the index suggested? On Wed, Nov 16, 2011 at 5:59 PM, Emanuele Ziglioli theb...@emanueleziglioli.it wrote: zig zag doesn't work for me, using the Siena API What's worse, an 'index error' was created and I had to cleanup the index error on the production server, a real PITA On Nov 17, 5:23 am, Alfred Fuller arfuller+appeng...@google.com wrote: That is referring only to the transient 'NeedIndex' error that popped up sometimes when a query was executed using zigzag in production: NeedIndexError: The built-in indices are not efficient enough for this query and your data. Please add a composite index for this query. This will no longer ever be thrown. On Wed, Nov 16, 2011 at 4:54 AM, Mos mosa...@googlemail.com wrote: Hello Alfred, thanks for clarification! I will report the issue below if it happens again. My confusion started when reading the presentation: http://whiteship.me/wp-content/uploads/2011/11/getting_the_most_out_o. .. On page 51, regarding the advanced query planner you can read No more 'needs index' errors (although some may timeout). I thought and hoped no more need for managing indexes manually .. Cheers Mos On Tue, Nov 15, 2011 at 6:46 PM, Alfred Fuller arfuller+appeng...@google.com wrote: On Wed, Nov 9, 2011 at 1:42 AM, Mos mosa...@googlemail.com wrote: Does the improved query planner reduce the need of managing indexes manually in the datastore-index.xml file? It increases the need to manage indexes manually I'm a bit confused. The documentation is not obvious for me. In local development mode I set autoGenerate=false and delete the datastore-index-auto.xml. My application runs fine. Deploying on GAE I've got the annoying com.google.appengine.api.datastore.DatastoreNeedIndexException: no matching index found. exception. This should not happen Hence, how does the improved query planner change the workflow of a GAE developer? Basically you have to turn off auto generate and manage the indexes manually. Cheers Mos -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Improved query planner in SDK 1.6.0
That is referring only to the transient 'NeedIndex' error that popped up sometimes when a query was executed using zigzag in production: NeedIndexError: The built-in indices are not efficient enough for this query and your data. Please add a composite index for this query. This will no longer ever be thrown. On Wed, Nov 16, 2011 at 4:54 AM, Mos mosa...@googlemail.com wrote: Hello Alfred, thanks for clarification! I will report the issue below if it happens again. My confusion started when reading the presentation: http://whiteship.me/wp-content/uploads/2011/11/getting_the_most_out_of_spring_and_app_engine_springone_2011.pdf On page 51, regarding the advanced query planner you can read No more 'needs index' errors (although some may timeout). I thought and hoped no more need for managing indexes manually .. Cheers Mos On Tue, Nov 15, 2011 at 6:46 PM, Alfred Fuller arfuller+appeng...@google.com wrote: On Wed, Nov 9, 2011 at 1:42 AM, Mos mosa...@googlemail.com wrote: Does the improved query planner reduce the need of managing indexes manually in the datastore-index.xml file? It increases the need to manage indexes manually I'm a bit confused. The documentation is not obvious for me. In local development mode I set autoGenerate=false and delete the datastore-index-auto.xml. My application runs fine. Deploying on GAE I've got the annoying com.google.appengine.api.datastore.DatastoreNeedIndexException: no matching index found. exception. This should not happen Hence, how does the improved query planner change the workflow of a GAE developer? Basically you have to turn off auto generate and manage the indexes manually. Cheers Mos -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Improved query planner in SDK 1.6.0
On Wed, Nov 9, 2011 at 1:42 AM, Mos mosa...@googlemail.com wrote: Does the improved query planner reduce the need of managing indexes manually in the datastore-index.xml file? It increases the need to manage indexes manually I'm a bit confused. The documentation is not obvious for me. In local development mode I set autoGenerate=false and delete the datastore-index-auto.xml. My application runs fine. Deploying on GAE I've got the annoying com.google.appengine.api.datastore.DatastoreNeedIndexException: no matching index found. exception. This should not happen Hence, how does the improved query planner change the workflow of a GAE developer? Basically you have to turn off auto generate and manage the indexes manually. Cheers Mos -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Improved query planner in SDK 1.6.0
This really shouldn't happen, something strange is going on. I need to know the queries seen by the local development server that are failing and the indexes you have to debug it. On Wed, Nov 9, 2011 at 3:02 PM, Emanuele Ziglioli theb...@emanueleziglioli.it wrote: So, from what I see, my query (that uses two simple indexes istead of one complex one) fails 4 or 5 times then succeeds. I'm testing on the local server with the latest SDK 1.6.0. The funny thing is that if I change the number of rows I fetch, that query also fails 5 times (or a number of seconds) before it starts working!! I don't know if I can log the GQL that goes out to show you On Nov 10, 10:54 am, Emanuele Ziglioli theb...@emanueleziglioli.it wrote: Just tried again, now it seems to be working! On Nov 10, 10:49 am, Emanuele Ziglioli theb...@emanueleziglioli.it wrote: It's not working for me either, using Siena: http://groups.google.com/group/siena-discuss/browse_thread/thread/b41. .. On Nov 10, 2:14 am, Matthew Jaggard matt...@jaggard.org.uk wrote: Hi VIKASH, It looks like you've replied to an unrelated message. If this is not the case, please clarify. Otherwise, please repost a new thread and give more detail. What are you trying to do? What have you tried? Are you using the low level API, JDO or JPA? Post the relevant bit of your code too if possible. Thanks, Mat. On 9 November 2011 12:47, VIKASH PATEL vickyexpert...@gmail.com wrote: Hello Friends, Can anyone have idea for working with datastore. I am able to create key and entity in data store but i am not able to edit or delete the created data.. so if any one has then pls help me On Wed, Nov 9, 2011 at 3:12 PM, Mos mosa...@googlemail.com wrote: Does the improved query planner reduce the need of managing indexes manually in the datastore-index.xml file? I'm a bit confused. The documentation is not obvious for me. In local development mode I set autoGenerate=false and delete the datastore-index-auto.xml. My application runs fine. Deploying on GAE I've got the annoying com.google.appengine.api.datastore.DatastoreNeedIndexException: no matching index found. exception. Hence, how does the improved query planner change the workflow of a GAE developer? Cheers Mos -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Improved query planner in SDK 1.6.0
The new query planner was actually released in 1.5.4 (the article was the only thing launched in 1.6.0) On Thu, Nov 10, 2011 at 11:46 AM, Emanuele Ziglioli theb...@emanueleziglioli.it wrote: Bump 2 :-) I'd be curious to see other people's experience with the new Query Planner. But 1.6.0 is still not part of a new release of the Google Plugin. That's frustrating, when a new SDK is released but one has to install it manually because it's not immediately available from the Eclipse update site. On Nov 10, 11:38 pm, Mos mosa...@googlemail.com wrote: anyone? Bumping this thread because it was hijacked On Wed, Nov 9, 2011 at 10:42 AM, Mos mosa...@googlemail.com wrote: Does the improved query planner reduce the need of managing indexes manually in the datastore-index.xml file? I'm a bit confused. The documentation is not obvious for me. In local development mode I set autoGenerate=false and delete the datastore-index-auto.xml. My application runs fine. Deploying on GAE I've got the annoying com.google.appengine.api.datastore.DatastoreNeedIndexException: no matching index found. exception. Hence, how does the improved query planner change the workflow of a GAE developer? Cheers Mos -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] 1.6.0 Prerelease SDKs are out
The order of the filters does not have an effect on the performance (the zigzag algorithm we use does not give preference to any index scans it is performing). On Tue, Nov 1, 2011 at 2:45 AM, Matthew Jaggard matt...@jaggard.org.ukwrote: Regarding the index selection shown here: http://code.google.com/appengine/articles/indexselection.html is it possible / required to put the query in the right order to get the best performance? For example, if I have a lot of black and white photos and just a handful of panoramic ones, does the first query below behave differently compared to the second? SELECT * FROM Photo WHERE colouration=black white AND aspect=panoramic ORDER BY date_added DESC; SELECT * FROM Photo WHERE aspect=panoramic AND colouration=black white ORDER BY date_added DESC; On 1 November 2011 01:13, Marzia Niccolai ma...@google.com wrote: Hi, We've just uploaded the 1.6.0 Prerelease SDKs: http://code.google.com/p/googleappengine/downloads/list Please note that new features are not available in production until the final release, and documentation for these features will be available at this time as well. Let us know if there are any issues found. Below are the release notes for this release. Thanks, Marzia App Engine Python SDK - Release Notes Version 1.6.0 === - On November 7th, App Engine will be out of Preview. The new Terms of Service and previously announced pricing changes will be in effect. Additionally, all paid apps are now covered by our SLA. http://www.google.com/enterprise/cloud/appengine/pricing.html - Paid apps can now specify the maximum pending latency for instances and the minimum number of idle instances for your application in the Admin Console. - We have released an experimental utility, available in the Admin Console, to assist in migrating your application to the High Replication datastore. This utility allows you to copy the bulk of your data in the background, while the source application is still serving. You then need a brief read-only period to migrate your application data while you copy the data that has changed from the time the original copy started. - Blobstore, which was previously limited to apps with billing enabled, is now available for all apps. - We have published a new article on Datastore Index Selection and Advanced Search which explains our recent improvements to the query planner that make exploding indexes unnecessary. http://code.google.com/appengine/articles/indexselection.html - Applications can now receive xmpp error stanzas at /_ah/xmpp/error. - In the Admin Console data viewer, you can now filter by namespace from a drop down menu, if applicable. - In the Admin Console's Datastore Statistics, we now offer namespace suggest for filtering stats. - We have released as experimental the full MapReduce framework. - The mail_stub.get_sent_messages() call now returns EmailMessage instances. - Fixed an issue when setting an initial_value in memcache.incr unexpectedly returned a string. http://code.google.com/p/googleappengine/issues/detail?id=2012 - Fixed an issue where DoS stats in the Admin Console didn't work for High Replication apps. http://code.google.com/p/googleappengine/issues/detail?id=5237 App Engine Java SDK - Release Notes Version 1.6.0 = - On November 7th, App Engine will be out of Preview. The new Terms of Service and previously announced pricing changes will be in effect. Additionally, all paid apps are now covered by our SLA. http://www.google.com/enterprise/cloud/appengine/pricing.html - Paid apps can now specify the maximum pending latency for instances and the minimum number of idle instances for your application in the Admin Console. - We have released an experimental utility, available in the Admin Console, to assist in migrating your application to the High Replication datastore. This utility allows you to copy the bulk of your data in the background, while the source application is still serving. You then need a brief read-only period to migrate your application data while you copy the data that has changed from the time the original copy started. - Blobstore, which was previously limited to apps with billing enabled, is now available for all apps. - We have published a new article on Datastore Index Selection and Advanced Search which explains our recent improvements to the query planner that make exploding indexes unnecessary. http://code.google.com/appengine/articles/indexselection.html - Applications can now receive xmpp error stanzas at /_ah/xmpp/error. - In the Admin Console data viewer, you can now filter by namespace from a drop down menu, if applicable. - In the Admin Console's Datastore
Re: [appengine-java] Migration to HRD
The copy MR code is available here: http://code.google.com/p/googleappengine/source/browse/trunk/python/google/appengine/ext/datastore_admin/copy_handler.py It is not doing anything special, it can be adapted to do what ever you need. Though it looks like you are a Java programmer (which might make it harder to adapt). On Fri, Oct 21, 2011 at 5:56 AM, Johan Euphrosine pro...@google.com wrote: You could also apply for testing the new HRD migration tool here: http://goo.gl/3jrXu Hope that helps. On Fri, Oct 21, 2011 at 1:49 PM, Aswath Satrasala aswath.satras...@gmail.com wrote: Hi, I have my application in M/S, and I have lots of namespaces. The Datastore Admin utility does not display entities in namespaces. How do I copy the data to new HRD app-id. Some of the other options I considered, and does not work very well for me - bulkloader to transfer data to the local machine and then uploading to the new app-id. It is too sloYw and I did not have luck dealing while dealing with the namespace data. - Remote api option does not work yet on the webapplication. Regards, -Aswath www.AccountingGuru.in -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- Johan Euphrosine (proppy) Developer Programs Engineer Google Developer Relations -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: JDO and XG transactions - Performance
This will tell you what type of datastore you are running against, but I don't think it will help you in the development env: http://code.google.com/p/googleappengine/source/browse/trunk/java/src/main/com/google/appengine/api/datastore/DatastoreService.java#388 On Tue, Oct 18, 2011 at 2:02 PM, Jeff Schnitzer j...@infohazard.org wrote: Dunno about production, but the developer mode takes issue with it. Even a simple single-entity transaction commit fails in M/S mode with the test harness - ie, without calling this method: LocalDatastoreServiceTestConfig.setDefaultHighRepJobPolicyUnappliedJobPercentage(100) commit() always produces this: java.lang.IllegalArgumentException: transactions on multiple entity groups only allowed in High Replication applications :-( Jeff On Tue, Oct 18, 2011 at 1:44 PM, Alfred Fuller arfuller+appeng...@google.com wrote: I believe (you should test this) that it won't do any harm to set XG=true in master slave (it will just ignore you). On Tue, Oct 18, 2011 at 1:28 PM, Jeff Schnitzer j...@infohazard.org wrote: On Tue, Oct 18, 2011 at 9:07 AM, David Gay (Google) d...@google.com wrote: One other consideration: XG transactions do not work on master/slave. While the default could be different depending on whether HRD is used, that definitely has drawbacks. How can I detect if running on HRD vs M/S? I don't see anything on SystemProperty. I could try starting an XG transaction and catching the exception, but that would add an extra RPC to every Objectify app's startup time :-( The XG issue seems pretty easily solved by documentation: 1) If you're running on M/S, you can only use one entity group in a txn. 2) If you're running on HRD, you can use up to 5 entity groups in a txn. 2a) If you use multiple entity groups in a txn, this is implemented with 2pc and may produce surprising effects like ConcurrentModificationException on read. This doesn't really matter in practice if you use the RepeatInTransaction pattern. Put it this way: Non-ancestor queries work funky on the HRD too. Yet I don't need to set a special flag just to issue non-ancestor queries. Understanding eventual consistency and exotic transaction behavior is just part of working with GAE. Magic flags that enable otherwise expected behavior don't really help. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: [appengine-python] Re: Prerelease SDK 1.5.5 available for download!
XG transactions will have no effect on global queries, as the fundamental problems still remains (namely that it is impossible to know what entity groups will/should appear in a global query). Additionally you should not use this read_policy, it has little effect in M/S and no effect in HRD. On Wed, Oct 5, 2011 at 11:04 PM, Beech Horn beechh...@gmail.com wrote: With the advent of XG, will APPLY_ALL_JOBS_CONSISTENCY now work with non-ancestor groups? APPLY_ALL_JOBS_CONSISTENCY = 2 A read consistency that aggressively tries to find write jobs to apply. Use of this read policy is strongly discouraged. This read_policy tends to be more costly and is only useful in a few specific cases. It is equivalent to splitting a request by entity group and wrapping each batch in a separate transaction. Cannot be used with non-ancestor queries. -- You received this message because you are subscribed to the Google Groups google-appengine-python group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-python/-/wAh2LQWixZQJ. To post to this group, send email to google-appengine-pyt...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-python+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-python?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: [google-appengine] Re: 1.5.2 SDK Prerelease
On Fri, Jul 15, 2011 at 12:25 AM, Alexandru Farcaş alex.far...@expert-group.biz wrote: Hi Alfred, Thanks for clarification. I also have 2 questions: 1. For this query: SELECT * FROM Model WHERE list = :1 AND list =:2 AND list=:3 AND string :=4 ORDER BY date DESC will be enough this index? - kind: Model properties: - name: list - name: string - name: date direction: desc Yes. This is the index the SDK will now suggest. 2. After I create this index (or indexes) I will still receive this exceptions? com.google.appengine.api.datastore.DatastoreNeedIndexException The built-in indices are not efficient enough for this query and your data. Please add a composite index for this query.. An index is missing but we are unable to tell you which one due to a bug in the App Engine SDK. If your query only contains equality filters you most likely need a composite index on all the properties referenced in those filters. It is possible. This means that there are lots of results that match each filter and no results that match all filters (in the first 10k results). If you see this, adding the following exploding index should help a great deal: - kind: Model properties: - name: list - name: list - name: string - name: date We plan on removing this exception in the future, but this won't improve the efficiency of the query (the only thing that will do that is adding indexes like this). --Alex -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/cHCKy8QEXw0J. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-python] Re: [appengine-java] Re: [google-appengine] Re: 1.5.2 SDK Prerelease
Yes, you are correct and those indexes will work. It's a trade of, composite indexes 'pre-intersect' (at write time) properties while zigzag merge join 'post-intersects' properties (at read time). I left the ancestor in because it is probably very 'selective' which has the potential to greatly reduce the amount of data that needs intersected at read time (though this is very data dependent). On Thu, Jul 14, 2011 at 11:35 AM, PK p...@gae123.com wrote: Alfred thanks for the clarification. However, isn't ancestor a list too that could contribute to an explosion (albeit minor assuming shallow hierarchies). If this is the case, would these indexes help/work? - kind: Model ancestor: yes properties: - name: int - name: date direction: desc - kind: Model properties: - name: list1 - name: int - name: date direction: desc - kind: Model properties: - name: list2 - name: int - name: date direction: desc Thanks -- You received this message because you are subscribed to the Google Groups google-appengine-python group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-python/-/aQh0Xx49xlsJ. To post to this group, send email to google-appengine-pyt...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-python+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-python?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-python] Re: [appengine-java] Re: [google-appengine] Re: 1.5.2 SDK Prerelease
:-) performance is data dependent. Here is convoluted explanation of performance: Sx = set of entities where list = :x smallest_set = min(S1.size(), S2.size(), ...) It works best when the intersection(S1, S2, S3,...) is large compared to the smallest_set. The pathological case is intersection(S1, S2, S3, ...) = 0 and smallest_set = |all data| / 2 On Fri, Jul 15, 2011 at 12:26 PM, Matija matija.jerko...@gmail.com wrote: OMG... finally... zigzag merge join... Any info on query performance SELECT * FROM Model WHERE list = :1 AND list = :2 AND list = :3 ORDER BY date DESC with index - kind: Model properties: - name: list - name: date or is it highly dependent on data distribution? Matija -- You received this message because you are subscribed to the Google Groups google-appengine-python group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-python/-/w_WoWIWmEncJ. To post to this group, send email to google-appengine-pyt...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-python+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-python?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: [google-appengine] Re: 1.5.2 SDK Prerelease
I should point out the SDK is currently very insistent about it's suggestions and believes that everything else is wrong (will throw an NeedIndexError), even though it may not be. We are working making the SDK smarter in this regards, but until then you will have to test in production (you can test in production now). Here is a short blurb about how you can remove exploding indexes (official docs coming soon): Consider: SELECT * FROM Model WHERE list1 = :1 AND list2 =:2 AND ancestor is :3 AND int = :4 ORDER BY date DESC Which 'needs' this index (from the perspective of the SDK): - kind: Model ancestor: yes properties: - name: list1 - name: list2 - name: int - name: date direction: desc Here a easy way to figure out what non-exploding indexes can be used instead: 1. Group all properties with equality/ancestor filters 1. in this case [ancestor, list1, list2, int] 2. Split this grouping such that no multi-valued properties are in the same group 1. one example is: [ancestor, list1], [list2, int], but the most efficient split actually repeats the single value properties: [ancestor, list1, int], [ancestor, list2, int] 3. Create the following indexes For each group: add index([grouped values] + [query orders/inequality]) in this case: index(ancestor, list1, int, -date) index(ancestor, list2, int, -date) or - kind: Model ancestor: yes properties: - name: list1 - name: int - name: date direction: desc - kind: Model ancestor: yes properties: - name: list2 - name: int - name: date direction: desc - Alfred On Tue, Jul 12, 2011 at 4:34 PM, Alfred Fuller arfuller+appeng...@google.com wrote: Hi, It means that there are alternatives to using exploding indexes (i.e. they are no longer required to execute a given query). You can still have them (there are cases where they are useful, namely to optimize query speed over write cost) and the SDK will still suggest them in many cases (as it is hard to detect them by looking at the query). However, the SDK will no long suggest indexes with the same property repeated multiple times (as these are obviously an exploding index). One example where this is very important is SearchableModelhttp://code.google.com/p/googleappengine/source/browse/trunk/python/google/appengine/ext/search/__init__.py, which previously was crippled by exploding indexes if you tried to sort results (and now works as expected). We are working on an article that goes through how to decided when to remove or keep them, but for now the Google IO talk from 2010, Next Gen Querieshttp://www.google.com/events/io/2010/sessions/next-gen-queries-appengine.html (The Zigzag Merge Join += Sort part), is a good resource if you want a really deep dive on how this works. - Alfred On Tue, Jul 12, 2011 at 2:24 AM, Pascal Voitot Dev pascal.voitot@gmail.com wrote: exactly the same question ;) On Tue, Jul 12, 2011 at 11:21 AM, Max thebb...@gmail.com wrote: Great job! May I know more about *t**he datastore now never requires an exploding index*? Does that mean we don't need to build exploding index or simply can't build exploding index? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/KyL9f70-VtkJ. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: [google-appengine] Re: 1.5.2 SDK Prerelease
Hi, It means that there are alternatives to using exploding indexes (i.e. they are no longer required to execute a given query). You can still have them (there are cases where they are useful, namely to optimize query speed over write cost) and the SDK will still suggest them in many cases (as it is hard to detect them by looking at the query). However, the SDK will no long suggest indexes with the same property repeated multiple times (as these are obviously an exploding index). One example where this is very important is SearchableModelhttp://code.google.com/p/googleappengine/source/browse/trunk/python/google/appengine/ext/search/__init__.py, which previously was crippled by exploding indexes if you tried to sort results (and now works as expected). We are working on an article that goes through how to decided when to remove or keep them, but for now the Google IO talk from 2010, Next Gen Querieshttp://www.google.com/events/io/2010/sessions/next-gen-queries-appengine.html (The Zigzag Merge Join += Sort part), is a good resource if you want a really deep dive on how this works. - Alfred On Tue, Jul 12, 2011 at 2:24 AM, Pascal Voitot Dev pascal.voitot@gmail.com wrote: exactly the same question ;) On Tue, Jul 12, 2011 at 11:21 AM, Max thebb...@gmail.com wrote: Great job! May I know more about *t**he datastore now never requires an exploding index*? Does that mean we don't need to build exploding index or simply can't build exploding index? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/KyL9f70-VtkJ. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] no async queries on AsyncDatastoreService for 1.4.0?
Currently hasNext() will block on the first batch of results. There is
currently no way to do what you are asking in Java.
On Tue, Dec 7, 2010 at 2:40 AM, Maxim Veksler ma...@vekslers.org wrote:
Hi Max,
Could you please provide input on the following design:
We design for DataSource redundancy. For this we've built a flow which
obtains data from 2 sources: First from GAE DataSource using PreparedQuery
and if that fails (0.5sec timeout) from our backup service feed which we
access using URLFetch (auto scaled EC2 nodes).
We strive for maximum performance, So we plan to issue async call to each
of the data sources and later when the data becomes required poll the 2
sources, using the first one available.
As for the URLFetch service, we will use this patten:
FutureHTTPResponse f = fetchService.fetchAsync(request);
...
if(f.isDone()) { ... }
But for PreparedQuery.asIterator(), I'm not sure how this is implement on
your side. I would love an answer confirming that this code will not block
on hasNext() (While datasource is asynchronously getting results)?
IteratorEntity iterator =
getAsyncDatastoreService().prepare(findGeoIP).asIterator();
if(iterator.hasNext()) { ... }
Thanks for the insights.
Maxim.
On Mon, Nov 29, 2010 at 9:08 PM, Max Ross (Google)
maxr+appeng...@google.com maxr%2bappeng...@google.com wrote:
Hi Luke,
First the awesome news:
As of 1.4.0, many queries are implicitly asynchronous. When you call
PreparedQuery.asIterable() or PreparedQuery.asIterator(), we initiate the
query in the background and then immediately return. This lets you do work
while the first batch of results is being fetched. And, when the first
batch has been consumed we immediately request the next batch. If you're
performing a significant amount of work with each Entity as you iterate you
will probably see a latency win as a result of this.
Now the less awesome news:
We didn't get around to making the List returned by PreparedQuery.asList()
work this same magic, but you can expect this in a future release.
Some deeper thoughts:
The underlying RPCs between your app and the datastore fetch results in
batches. We fetch an initial batch of results, and once that batch has been
consumed we fetch the next batch. But, there's nothing in the API that maps
to these batches - it's either a List containing the entire result set or an
Iterable/Iterator that returns Entities one at a time. An API that provides
async access to the individual results returned by an Iterable/Iterator
(IteratorFutureEntity) doesn't really make sense since you don't know
which call to hasNext() is going to require a new batch to be fetched, and
without that knowledge, the knowledge of what is going to trigger something
expensive, you can't really make appropriate use of an asynchronous API.
Going forward, we're definitely interested in exposing these batches
directly, and an explicitly async API for these batches makes a lot of sense
since fetching these batches would map directly to something expensive on
the server side.
Hope this helps,
Max
On Fri, Nov 26, 2010 at 4:41 PM, Luke lvale...@gmail.com wrote:
i was taking a look at the 1.4.0 javadoc for AsyncDatastoreService. i
see the get, put and delete operations return a Future, but the
prepare methods return a naked PreparedQuery object, and it doesn't
look like PreparedQuery has any async get methods.
does the AsyncDatastoreService not support asynchronous queries, or is
there something i'm missing?
glad to see at lets the get and put methods are async, hoping to get
async queries too (as well as async interfaces to more services).
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to
google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to
google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to
google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
Re: [appengine-java] built-in indices are not efficient enough for this query and your data
On Sat, Aug 7, 2010 at 6:18 PM, Philip Tucker ptuc...@gmail.com wrote: I've just started getting the following error on my server. This query had been working fine for months, and I haven't changed the indexes lately. Are there some indexes that just buckle under heavy load or larger data sets? This particular table contains only 242,604 entries. The built-in indices are not efficient enough for this query and your data. Please add a composite index for this query.. An index is missing but we are unable to tell you which one due to a bug in the App Engine SDK. If your query only contains equality filters you most likely need a composite index on all the properties referenced in those filters. (full stack dump included below) My query source code: Query query = pm.newQuery(GameDataV2.class); Object[] params; query.declareParameters( com.google.appengine.api.datastore.Key userKeyParam + , com.honkentuber.wordwise.GameState stateParam1 + , com.honkentuber.wordwise.GameState stateParam2 + , com.honkentuber.wordwise.GameState stateParam3); params = new Object[4]; params[0] = userKey; params[1] = GameState.PENDING; params[2] = GameState.PLAYER_TURN; params[3] = GameState.OPPONENT_TURN; query.setFilter((memberKeys == userKeyParam) + ((state == stateParam1) + || (state == stateParam2) + || (state == stateParam3))); ListGameDataV2 gamesData = (ListGameDataV2) query.executeWithArray(params); I have the following 2 indexes defined: memberKeys ▲ , state ▲ , memberLastAccessMs ▲ memberKeys ▲ , state ▲ , memberScores ▼ I'm not sure why I don't have an index with only memberKeys and state - does datastore-indexes autoGenerate not generate a new index if it already has a superset index? I've changed the query to the following in my local development environment: Query query = pm.newQuery(GameDataV2.class); query.setFilter((memberKeys == :userKeyParam) :statesParam.contains(state)); ListGameDataV2 gamesData = (ListGameDataV2) query.execute( userKey, Arrays.asList( GameState.PENDING.name(), GameState.PLAYER_TURN.name(), GameState.OPPONENT_TURN.name())); Some questions about this: 1) Running locally with autoGenerate=true, I'm not getting a new GameDataV2 index on memberKeys and states. Does contains() not require an index on that field? Is this just generating 2 separate queries internally? It's using zigzag merge join to solve your query. We do not generate an index for this query because it is not strictly needed. The error message you are getting will appear if you pull a lot of data or the specific data you are querying is especially inefficient for zigzag merge join to solve. I am removing this particular error soon, but that doesn't mean zigzag will become more efficient when working with your data (if this is in fact the problem). And example when efficiency is not the problem is having X equality filters and pulling 10,000/X results (this will always produce this error even when zigzag merge join is perfectly efficient). 2) Is the contains() query more efficient than the nested ORs I had in my original query? Or will I still get the not efficient enough for this query and your data error? 3) Why do enumerated types not work with contains()? I had to add .name() to each entry in the list to get this to work. 4) Why do declared parameters not work with collections? I tried declaring statesParam as a both ListGameState and List, and I got errors resolving the types. com.google.appengine.api.datastore.DatastoreNeedIndexException: The built-in indices are not efficient enough for this query and your data. Please add a composite index for this query.. An index is missing but we are unable to tell you which one due to a bug in the App Engine SDK. If your query only contains equality filters you most likely need a composite index on all the properties referenced in those filters. at com.google.appengine.api.datastore.DatastoreApiHelper.translateError(DatastoreApiHelper.java: 40) at com.google.appengine.api.datastore.DatastoreApiHelper.makeSyncCall(DatastoreApiHelper.java: 67) at com.google.appengine.api.datastore.PreparedQueryImpl.runQuery(PreparedQueryImpl.java: 127) at com.google.appengine.api.datastore.PreparedQueryImpl.asIterator(PreparedQueryImpl.java: 87) at com.google.appengine.api.datastore.PreparedMultiQuery $DedupingMultiQueryIterator.getNextIterator(PreparedMultiQuery.java: 154) at com.google.appengine.api.datastore.PreparedMultiQuery $DedupingMultiQueryIterator.computeNext(PreparedMultiQuery.java:173) at com.google.appengine.api.datastore.PreparedMultiQuery $DedupingMultiQueryIterator.computeNext(PreparedMultiQuery.java:98) at
Re: [appengine-java] problem in auto-generate index after deploy application
What query are you running that produces this exception? On Sun, Aug 1, 2010 at 8:06 AM, Erencie xiaoni@gmail.com wrote: I think someone has posted this before but the post is not written in English so I don't understand. I tested thoroughly in my local machine and there is no problem in persisting one of my user objects. After I upload it to Google App Engine, and do the same thing, the following errors occurs: Uncaught exception from servlet com.google.appengine.api.datastore.DatastoreNeedIndexException: no matching index found.. datastore-index kind=Course ancestor=true source=manual property name=courses_INTEGER_IDX direction=asc/ /datastore-index at com.google.appengine.runtime.Request.process-496d0292b582fcd8(Request.java) at com.google.appengine.api.datastore.DatastoreApiHelper.translateError(DatastoreApiHelper.java: 40) at com.google.appengine.api.datastore.DatastoreApiHelper.makeSyncCall(DatastoreApiHelper.java: 67) at com.google.appengine.api.datastore.PreparedQueryImpl.runQuery(PreparedQueryImpl.java: 127) at com.google.appengine.api.datastore.PreparedQueryImpl.asIterator(PreparedQueryImpl.java: 87) at com.google.appengine.api.datastore.BasePreparedQuery $1.iterator(BasePreparedQuery.java:26) at org.datanucleus.store.appengine.DatastoreElementContainerStoreSpecialization.getChildren(DatastoreElementContainerStoreSpecialization.java: 105) at org.datanucleus.store.appengine.DatastoreFKListStore.listIterator(DatastoreFKListStore.java: 48) at org.datanucleus.store.mapped.scostore.AbstractListStore.listIterator(AbstractListStore.java: 84) at org.datanucleus.store.mapped.scostore.AbstractListStore.iterator(AbstractListStore.java: 74) at org.datanucleus.sco.backed.List.loadFromStore(List.java:241) at org.datanucleus.sco.backed.List.iterator(List.java:507) at org.apache.jsp.coordinator.classes_jsp._jspService(classes_jsp.java: 59) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:94) at javax.servlet.http.HttpServlet.service(HttpServlet.java:806) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java: 511) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1166) at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java: 97) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java: 35) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java: 43) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java: 388) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java: 216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java: 182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java: 765) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java: 418) at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java: 238) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java: 152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java: 542) at org.mortbay.jetty.HttpConnection $RequestHandler.headerComplete(HttpConnection.java:923) at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java: 76) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java: 135) at com.google.apphosting.runtime.JavaRuntime.handleRequest(JavaRuntime.java: 250) at com.google.apphosting.base.RuntimePb$EvaluationRuntime $6.handleBlockingRequest(RuntimePb.java:7115) at com.google.apphosting.base.RuntimePb$EvaluationRuntime $6.handleBlockingRequest(RuntimePb.java:7113) at com.google.net.rpc.impl.BlockingApplicationHandler.handleRequest(BlockingApplicationHandler.java: 24) at com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java: 398) at com.google.net.rpc.impl.Server$2.run(Server.java:852) at com.google.tracing.LocalTraceSpanRunnable.run(LocalTraceSpanRunnable.java: 56) at com.google.tracing.LocalTraceSpanBuilder.internalContinueSpan(LocalTraceSpanBuilder.java: 576) at
Re: [appengine-java] DataStore, ApplicationError: 3: Unexpected error, please try again
This can happen when you add the same entity to the put(or delete) request more than once (same by .equals). If this is the case, your put actually succeeded. This should be fixed soon, however I would suggest checking your code to figure out why you are putting the same entity multiple times (if in fact this is what is happening). On Fri, Jun 18, 2010 at 10:19 AM, Toby toby.ro...@gmail.com wrote: Hello, I am lately getting errors of this kind: Response: com.google.apphosting.api.ApiProxy$ApplicationException: ApplicationError: 3: Unexpected error, please try again. at com.google.apphosting.runtime.ApiProxyImpl $AsyncApiFuture.rpcFinished(ApiProxyImpl.java:293) at com.google.net.rpc.RpcStub$RpcCallbackDispatcher $1.runInContext(RpcStub.java:1025) at com.google.tracing.TraceContext $TraceContextRunnable$1.run(TraceContext.java:444) at com.google.tracing.TraceContext.runInContext(TraceContext.java:684) at com.google.tracing.TraceContext $AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java: 322) at com.google.tracing.TraceContext $AbstractTraceContextCallback.runInInheritedContext(TraceContext.java: 314) at com.google.tracing.TraceContext $TraceContextRunnable.run(TraceContext.java:442) at com.google.net.rpc.RpcStub $RpcCallbackDispatcher.rpcFinished(RpcStub.java:1046) at com.google.net.rpc.RPC.internalFinish(RPC.java:2047) at com.google.net.rpc.impl.RpcNetChannel.finishRpc(RpcNetChannel.java: 2338) at com.google.net.rpc.impl.RpcNetChannel.messageReceived(RpcNetChannel.java: 1265) at com.google.net.rpc.impl.RpcConnection.parseMessages(RpcConnection.java: 319) at com.google.net.rpc.impl.RpcConnection.dataReceived(RpcConnection.java: 290) at com.google.net.async.Connection.handleReadEvent(Connection.java:474) at com.google.net.async.EventDispatcher.processNetworkEvents(EventDispatcher.java: 831) at com.google.net.async.EventDispatcher.internalLoop(EventDispatcher.java: 207) at com.google.net.async.EventDispatcher.loop(EventDispatcher.java: 103) at com.google.net.async.GlobalEventRegistry $2.runLoop(GlobalEventRegistry.java:95) at com.google.net.async.LoopingEventDispatcher $EventDispatcherThread.run(LoopingEventDispatcher.java:378) It is during batch insert of multiple objects. The request took 2904ms 2753cpu_ms 1703api_cpu_ms. On system status I can not see any irregularities of the datastore. Maybe it is in my code? Thank you, Tobias -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Database cursor for back cursor?
For most queries the process goes something like this:
Index Scan - Fetch Entity - Send to user
For any query with a sort order or inequality filter on any property
other than __key__ we are still actually fetching the entity in the
datastore, so the only savings is the serialization and transmission
cost of sending the result to the user. This means that offset is
still typically quite costly.
On Apr 6, 11:42 am, Ikai L (Google) ika...@google.com wrote:
Offset doesn't fetch data - it does an index scan. Going to the 10,000th
result using offset will require us to pass 10,000 results first in our
index, but we won't be retrieving those objects. Here's a bit of a
simplification of indexes, entities and how these queries work. As you know,
Bigtable is a key-value store with the ability to perform range scans.
Suppose you have a Person entity. Your entries may look like this:
Key
Value
--
APPID:Person:1 { name:
Ikai, favorite_food: donuts }
APPID:Person:2 { name:
Wesley, favorite_food: tacos }
Index:APPID:Person:favorite_food:ASC:donuts:1 [ empty ]
Index:APPID:Person:favorite_food:ASC:tacos:2 [ empty ]
*order of values in keys may not match how we actually do it, it's just to
illustrate my point
** argh, non-monospaced fonts. Sorry for the hideous table
When we look for Person entities sorted by favorite_food, we essentially ask
Bigtable to return all Keys that match:
Index:APPID:Person:favorite_food:ASC*
We get a list of Keys back and figure out keys. Using offsets, we would just
pass the first N entries per the supplied offset.
Cursors work by serializing the query position via the last index, allow us
to go straight to the index in constant time and continue the range query.
We've got a lot of content about how this works, though we probably haven't
done the best job organizing it. I'd start
here:http://code.google.com/appengine/articles/storage_breakdown.html
I've been meaning to collect all of our talks and place them into a set of
YouTube playlists. Why don't I do that right now ...
On Sat, Apr 3, 2010 at 8:16 AM, John Patterson jdpatter...@gmail.comwrote:
How many pages do your users really want to see? Even Google search sets a
limit - I think 1000 results. Although the number of results can exceed
1000 the offset is still limited so you would need to filter out results to
continue past 1K. That would be slow.
If you really need to return more than 1000 results and access them in
forward and backward directions you could sort them by __key__ or some other
unique combination and use that as you own cursor. You could define both
ascending and descending indexes on your chosen property(s) to let you
iterate in both directions.
On 3 Apr 2010, at 19:32, Arny wrote:
But isn't it getting slower and slower on higher pages, since it
fetches ALL data (according to docs) and discards the offset value?
So Range(0,10) is faster than Range(1,10) ?
Anyone did some performance tests?
Regards
On Apr 3, 7:49 am, John Patterson jdpatter...@gmail.com wrote:
Probably you should set an offset and limit instead for your paging.
I believe cursors are really intended for processing a lot of data off-
line. Remember there is no longer a 1000 result limit on fetches.
On 3 Apr 2010, at 02:57, Arny wrote:
Hi,
Is there a way to get a cursor to page back to a previous page?
If not, whats the point of paging forward only? I'm not building an
ajax page.
Regards
--
You received this message because you are subscribed to the Google
Groups Google App Engine for Java group.
To post to this group, send email to
google-appengine-java@googlegroups.com
.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com
.
For more options, visit this group athttp://
groups.google.com/group/google-appengine-java?hl=en
.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to
google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to
google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
[appengine-java] Re: Database cursor for back cursor?
This isn't a bad solution, but it has several problems: - It doesn't scale - Cache could have holes. I guess then you could grab a query and scroll though the first 20 pages to fill out the cache. You could actually start at the highest cursor before the requested page and page forward to fill out any gapes in the cache. - What happens when the data changes. Say some one adds a full page of elements in the middle then the cache would miss these results when scrolling back I am planning to implement a new type of Plannable cursor that will be smaller and allow for backwards scrolling. It will however require an extra index for every index you want to page over. It is roughly identical to John's suggestion, so in the mean time I would recommend that (just remember to include __key__ desc in the descending index). On Apr 23, 4:02 pm, boustanihani boustanih...@googlemail.com wrote: To achieve this I am saving all corsors (in an ArrayListString) on the Client and reusing them for getting bachward pages :) ... this works! What do u think? On 2 Apr., 21:57, Arny arny...@googlemail.com wrote: Hi, Is there a way to get acursorto pagebackto a previous page? If not, whats the point of paging forward only? I'm not building an ajax page. Regards -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group athttp://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
