[appengine-java] Transactions on entities in different entity groups ?
Hi, I want to know how to create/update the entities which are in different entity groups ? for ex: I want to update the *apple* and *carrot* entities in a transaction but those parents are not same. (Here i am in such a situation) Thanks in advance :) *S*antosh *K*umar *K* http://www.accountingguru.in/** -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: GAE - Vote counting system
Many writes to the same object will lead to db failures. You really should consider sharding: http://code.google.com/appengine/articles/sharding_counters.html On Sep 26, 12:41 am, Peter Dev dev133...@gmail.com wrote: We are developing an application, where users can vote for many objects. (for example, voting the best music video of the week) - This means, we have millions of possible objects to vote for, and millions of users To our best knowledge, after taking in consideration different options, the best (or the only) voting system is: Memcache+Bulk DB write in DB - If number of objects in Memcache will achieve a specified limit (for example 3000) then write in DB. The writing speed into DB is about 100/sec. This also means, if we would set the above mentioned 3000 objects, the writing would last for 30 sec... The problem; during save in DB, voting must be blocked. In other words, if from many millions of objects, there are 3000 achieved voted objects, we need to write it into DB and it can happen too many times, and blocking the whole voting mechanism. If we do not block the voting whilst writing in DB, the result could be wrong number of votes from cache (see Workflow: step 3). Workflow: 1. vote received 2. find object in memcache 3. if not found in memcache get from DB and put it in 4. increment the number of votes of the object in memcache 5. check object number in memcache 6. if necessary, save in DB and empty memcache ... -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] GAE - Vote counting system
Have you looked into using pull queues combined with sharded counters? If you lease 1,000 tasks at a time you can probably condense writes considerably. There are several different sharding strategies you could take, even just one master set of shards each of which covers all items. 100 shards * 1,000 tasks should give you a minimum write rate limit of 100,000/sec, magnified by the amount of overlap (ie if those 1,000 tasks only updated 100 counts, your minimum write rate limit is 1 million/sec). I haven't tried this. It presumes you can write 1k entities to a single entity group in a single transaction - maybe a googler can chime in. It would probably be expensive to store. Jeff On Mon, Sep 26, 2011 at 12:41 AM, Peter Dev dev133...@gmail.com wrote: We are developing an application, where users can vote for many objects. (for example, voting the best music video of the week) - This means, we have millions of possible objects to vote for, and millions of users To our best knowledge, after taking in consideration different options, the best (or the only) voting system is: Memcache+Bulk DB write in DB - If number of objects in Memcache will achieve a specified limit (for example 3000) then write in DB. The writing speed into DB is about 100/sec. This also means, if we would set the above mentioned 3000 objects, the writing would last for 30 sec... The problem; during save in DB, voting must be blocked. In other words, if from many millions of objects, there are 3000 achieved voted objects, we need to write it into DB and it can happen too many times, and blocking the whole voting mechanism. If we do not block the voting whilst writing in DB, the result could be wrong number of votes from cache (see Workflow: step 3). Workflow: 1. vote received 2. find object in memcache 3. if not found in memcache get from DB and put it in 4. increment the number of votes of the object in memcache 5. check object number in memcache 6. if necessary, save in DB and empty memcache ... -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] File Upload
Why don't you use blobstore service?? http://code.google.com/appengine/docs/java/blobstore/overview.html On Mon, Sep 26, 2011 at 5:57 PM, Guilherme Souza souza.guilherm...@gmail.com wrote: Hey guys, i have a problem with uploading files at my project, i tried to use the library org.apache.commons.fileupload, but i couldn't import to eclipse, now i'm looking for a new way to upload files to my JSP page. Thanks in advance. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Google App Engine Internet Crawler
Good day Developers! Any idea on how to crawl the internet in Google App Engine - Java? My thesis requires me to search a book in the internet just using the book's ISBN. Thank you very much. :D -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Disable 'refresh' in a page
I agree with Mat. Just tell your users not to refresh or give them no reason to. They aren't stupid. It's like when a site prevents me right-clicking - I hate that site and never go back. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/uuaynCZW5rUJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Cannot deploy for past several hours?
I haven't been able to update one of my applications for the past several hours. I keep getting the error: Unable to update app: Error posting to URL: https://appengine.google.com/api/appversion/create?app_id=xversion=22; 400 Bad Request Client Error (400) The request is invalid for an unspecified reason. See the deployment console for more details Unable to update app: Error posting to URL: https://appengine.google.com/api/appversion/create?app_id=xversion=22; 400 Bad Request Client Error (400) The request is invalid for an unspecified reason. What is causing this? -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/RbC4fmTgTOUJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: GAE - Vote counting system
Shared counter is cool and I use it... but if you have millions of objects I cannot imagine how to manage them. 1 000 000 obj x 100 shards = 10 000 000 counters 1. How to reset them to 0 in specified periods? 2. How to set the shared sum for each object to show top 100 objects? 3. Too much DB API Calls (each vote makes write in DB) Any ideas...? Thx -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: GAE - Vote counting system
Sorry, 100 000 000 counters On Sep 27, 4:53 pm, Peter Dev dev133...@gmail.com wrote: Shared counter is cool and I use it... but if you have millions of objects I cannot imagine how to manage them.1 000 000obj x 100 shards =10 000 000counters 1. How to reset them to 0 in specified periods? 2. How to set the shared sum for each object to show top 100 objects? 3. Too much DB API Calls (each vote makes write in DB) Any ideas...? Thx -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] File Upload
Thanks Stefanos, i'm reading the documentation now. It will do resolve my problems. Question done! 2011/9/27 Stefanos Antaris ssanta...@gmail.com Why don't you use blobstore service?? http://code.google.com/appengine/docs/java/blobstore/overview.html On Mon, Sep 26, 2011 at 5:57 PM, Guilherme Souza souza.guilherm...@gmail.com wrote: Hey guys, i have a problem with uploading files at my project, i tried to use the library org.apache.commons.fileupload, but i couldn't import to eclipse, now i'm looking for a new way to upload files to my JSP page. Thanks in advance. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- Atenciosamente, Guilherme Santos Souza Sistemas de Informação - PUCRS Fone: (51) 9695-1070 -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Memcache objects remain only 5 minutes
My objects remain in the memcache only for about 5 minutes. I know that there is no guarantee about the caching period, but this seems abnormal. I've read that lack of memory could be the reason. The size of one object is about 50K, and I typically cache less as 50 objects. I have no flush statements in my code don't remove objects. What can be the reason for this, or how can I look for the reason? -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/cn5YIIUS5AYJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] SOAP-Samples and the Development Server
Is it possible to run the soap-samples from within the development server? I changed the soap address location in the wsdl filt to soap:address location=http://localhost:/codelabex5part1/ but it doesn't work... -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/dKbYLnJ6glMJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Allowing the user to Change accounts
I would like to allow the user of my app engine application to switch accounts, for users that have multiple accounts and use the google multiple sign-ins turned on. I have tried using the UserServiceFactory.getUserService().createLogoutURL() and send them there, but that only seems to let them log back into the domain they were already logged in on. I have also tried not logging out and simply UserServiceFactory.getUserService().createLoginURL() and sending them to that, but it seems to detect that they are already logged in and just brings them right back to my app with the same account. Any help would be greatly appreciated. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: GAE - Vote counting system
Yeah, messy. I'd use a backend for this. Possibly a set of backends if you need to shard the data for write volume. I'd use Memcache only to cache the count reads. The basic entity is just an id and a count. An increment request goes to a backend, which simply tracks the change. A batch process goes through and writes any changed counts to both datastore and memcache (as an update; read increment write in a txn) every 5 minutes and clears the memory count. If write volume is too high for a single backend to handle, shard it by thingId % number of shards. You can change the shard count on the fly this way. All reads should be read from the memcache, read-through to the datastore as necessary. This should be able to handle any volume you want. If a backend crashes you'll lose its accumulated counts but I presume that's not a big deal. Jeff On Tue, Sep 27, 2011 at 10:38 AM, Peter Dev dev133...@gmail.com wrote: Sorry, 100 000 000 counters On Sep 27, 4:53 pm, Peter Dev dev133...@gmail.com wrote: Shared counter is cool and I use it... but if you have millions of objects I cannot imagine how to manage them.1 000 000obj x 100 shards =10 000 000counters 1. How to reset them to 0 in specified periods? 2. How to set the shared sum for each object to show top 100 objects? 3. Too much DB API Calls (each vote makes write in DB) Any ideas...? Thx -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Snapshots...
Hi all, What's the best way to take snapshots of the GAE's database in a way it's possible to backup/restore it if needed? Thanks, Daniel V. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Snapshots...
Hi, I do personally scan each entity type via an ad hoc request and store each instance as entry in an xml file stored in the blobstore. You can then externalize it via download from another machine. Another possible way is to store you data on Amazon S3: you can then also store them 1 by 1 if you wish so. regards didier On Sep 27, 11:45 pm, Daniel Viveiros vivei...@ciandt.com wrote: Hi all, What's the best way to take snapshots of the GAE's database in a way it's possible to backup/restore it if needed? Thanks, Daniel V. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Snapshots...
Hi Daniel, Take a look at our Bulkloader Tool: http://code.google.com/appengine/docs/python/tools/uploadingdata.html Here is a great post: http://ikaisays.com/2010/06/10/using-the-bulkloader-with-java-app-engine/ Hope this helps your needs! Best, Jose Montes de Oca -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/mqMA-JEDmFQJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
