Re: [appengine-java] How do I query by key?
Let's say you have a Profile entity with an id of 123. The easiest way to retrieve it is to use DatastoreService.get(), not a query. Construct a Key and pass that to get(): DatastoreService ds = DatastoreServiceFactory.getDatastoreService(); Key key = KeyFactory.createKey(Profile, 123L); Entity ent = ds.get(key); Jeff On Thu, Mar 22, 2012 at 5:34 PM, Aribar Hunter aribarhun...@gmail.com wrote: I feel stupid. I've looked over lots of documentation, I've tried Google searches, I've searched on other sites like Stackoverflow, and I just don't understand this. I have a site that allows the user to access profiles of imaginary people containing information like names and descriptions. I have links dynamically put on the side bar for each profile in my datastore. The links look like profile.jsp?id=1, replacing 1 with whatever id that profile is. I want to try and display the information in profile.jsp that relates to the id, or key, that's in the url. The thing is, I don't know how to structure the query. I've seen things about JDO/JDA and queries that look a lot like SQL on other forms, but I have no idea what those people are actually doing. This is the closest I've come to making this work, but it just returns nothing: % Query query = new Query(Entry); query.addFilter(Key, Query.FilterOperator.EQUAL, request.getParameter(id)); PreparedQuery pq = datastore.prepare(query); % End User: %= pq.asSingleEntity().getProperty(name) % What I think I'm doing is making a Query for the Kind with all the information, then filtering that id should equal whatever is in the Key spot... But that doesn't seem to be the case. Any help would be appreciated. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/t5OcxJUjLAgJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Quick setup and deploy for university class lab time?
Can students install Eclipse and run Hello, World on GAE in 3.5 hours? Almost certainly. How much more sophisticated can the sample project get? This is a hard thing to judge without knowing your students. Back when I was a CSc student, data structures was a 200-level class and few (if any) students would have been ready to build a web application (especially since there was no web... but that's beside the point). Maybe students are more savvy today, but my guess is that learning how to develop a GAE app would simply distract from learning data structures. Jeff On Wed, Mar 21, 2012 at 9:43 PM, bsddorin bsddo...@gmail.com wrote: In 3 weeks I have scheduled some lab time for my data structures class. We have been studying data structures since January and I would really like for them to have a little application they can 'show the world'... We only have 3.5 hours in the lab. The PC's have java installed and can do java builds. Is there something that can be accomplished in 3.5 hours ... with no app engine toolkits setup? Any super easy templates out there? I just don't want to get into a quagmire where we spend the whole time on tools...so perhaps this isn't a good idea. Thoughts? -Mike -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] How to handle class version issues in DataStore
That's not quite correct - in the low-level datastore, this isn't an issue because absent properties will simply be missing from the Entity. In Objectify, absent properties are ignored. You can add fields of primitive types without ill effect - they will simply have their default values. This is very useful when you add fields, or flag fields as not-saved when they have certain values (ie the default). I believe JPA/JDO will throw exceptions if you add a primitive type field to your entity and there is no corresponding property in the underlying loaded Entity. This is very poor behavior, IMHO. It creates a very brittle structure. Jeff On Mon, Mar 12, 2012 at 4:12 AM, Matthew Jaggard matt...@jaggard.org.uk wrote: This answer also applies to the low level datastore and Objectify. I can only assume JPA works the same way. On Mar 12, 2012 7:40 AM, andrew andrew.macken...@bcntouch.com wrote: Nulls (Java), if the object can have null as a value. E.g. Integer can be null, int cannot. I think this is explained in GAE/JDO documentation. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/GBfSKyvTztcJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] GAE performance, the log, and a missing 500ms
Have you tried using appstats? (isn't this group supposed to be readonly?) Jeff On Sun, Mar 11, 2012 at 12:28 PM, n3phele n3ph...@gmail.com wrote: Hi Im trying to understand the performance of my GAE app, which appears to be declining over time. Here is a log from my Java app .. please note that my instance is running so this isnt the first time the code is being executed in the instance.. 2012-03-11 10:16:12.841 /resources/command? summary=truestart=0end=16preferred=true 200 892ms 7kb Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/ 17.0.963.78 Safari/535.11 67.176.95.243 - - [11/Mar/2012:09:16:12 -0700] GET /resources/command? summary=truestart=0end=16preferred=true HTTP/1.1 200 7282 - Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.78 Safari/535.11 n3phele.appspot.com ms=893 cpu_ms=376 api_cpu_ms=82 cpm_usd=0.011300 instance=00c61b117cca20def4d597294b9929c08f6983 I 2012-03-11 10:16:12.467 n3phele.service.rest.impl.BasicSecurityFilter$Authorizer isUserInRole: Checking role authenticated with user defined I 2012-03-11 10:16:12.469 n3phele.service.rest.impl.CommandResource$CommandManager getPreferredCollection: Starting owner fetch W 2012-03-11 10:16:12.469 n3phele.service.rest.impl.CommandResource list: list entered with summary true start 0 end 16 preferred true I 2012-03-11 10:16:12.486 n3phele.service.rest.impl.CommandResource$CommandManager getPreferredCollection: Finished owner key fetch I 2012-03-11 10:16:12.527 n3phele.service.rest.impl.CommandResource$CommandManager getPreferredCollection: Finished shared key fetch I 2012-03-11 10:16:12.814 n3phele.service.rest.impl.CommandResource$CommandManager getPreferredCollection: Finished collection fetch I 2012-03-11 10:16:12.832 n3phele.service.rest.impl.CommandResource list: partition commands start I 2012-03-11 10:16:12.832 n3phele.service.rest.impl.CommandResource list: partition commands end Sun Mar 11 16:16:12.832 UTC 2012 The first line at 10:16:12.467 represents some processing in the security filter associated with the request, and the last line 10:16:12.832 is in my applet code just before I return my reply. The overall log line itself ends at 10:16:12.841, and I am assuming that the 10ms difference is some final processing turning the reply into JSON. My real question here is about the elapsed time ms=893. If I look at the processing time spent in my code it is 365ms. Can any one suggest where the other 893-365=528 ms is being spent? (its this 500ms which is killing me from a performance perspective). Thx in advance.. \n -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Reminder: Google App Engine community support is moving to Stack Overflow
I find that you're either one of those people who lives on StackOverflow or you're one of those people who only visits when they're searching for something specific. I'm the later myself; I can't follow SO the way I follow an email list. Truth be told, I have the google-appengine list and the google-appengine-java list both dumped to a common label in gmail, so combining the mailing lists really makes no difference to me. Actually it improves the situation, since most discussions really end up being GAE-general. But yeah, SO is tough to follow. I'm not sure it's really all that useful to try. Jeff On Thu, Mar 8, 2012 at 7:30 PM, Shawn Brown big.coffee.lo...@gmail.com wrote: When you're searching, you can specify a tag by putting it in square brackets. E.g. if you search for: [google-app-engine] [java] spring this will retrieve questions tagged with [google-app-engine] and [java], and containing 'spring'. Thanks Amy, It's weird. Here's what worked. [google-app-engine] +java then sort by newest Here's what didn't work [google-app-engine] java because the site changed it to [google-app-engine] [java] BUT many posts are not tagged as java though they contained java in the post which is what I was looking for but not showing up Thanks for your help. Shawn -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Logging exceptions all over!
I also see a few entries like this in my logs. My site is still up, however. (this mailing list is about to go read-only - further discussion should go on the regular google-appeng...@googlegroups.com mailing list) Jeff On Tue, Mar 6, 2012 at 2:38 PM, Zappa kbhdk1...@gmail.com wrote: Suddenly, my application stopped working. Have I missed something? I didn't change a thing. This is the first exception, the website responds now with a 500. 1. 2012-03-05 18:10:24.941 /page.htm?name=articles.htm 500 2027ms 0kb Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) 66.249.67.79 - - [05/Mar/2012:18:10:24 -0800] GET /page.htm?name=articles.htm HTTP/1.1 500 0 - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) www.vegetarlight.dk ms=2027 cpu_ms=957 api_cpu_ms=0 cpm_usd=0.026618 loading_request=1 instance=00c61b117c54322c5564374377430cd358ffe857 https://appengine.google.com/instances?app_id=vegetarlightversion_id=12.339884832661696825key=00c61b117c54322c5564374377430cd358ffe857#00c61b117c54322c5564374377430cd358ffe857 2. W2012-03-05 18:10:24.375 Failed to read file: /base/data/home/apps/vegetarlight/12.339884832661696825/WEB-INF/lib/commons-logging-1.1.1.jar java.util.zip.ZipException: error in opening zip file at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.init(ZipFile.java:143) at java.util.jar.JarFile.init(JarFile.java:150) at java.util.jar.JarFile.init(JarFile.java:114) at org.mortbay.jetty.webapp.TagLibConfiguration.configureWebApp(TagLibConfiguration.java:174) at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1247) at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517) at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:467) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:202) at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:171) at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:123) at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:422) at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:449) at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:455) at com.google.tracing.TraceContext.runInContext(TraceContext.java:695) at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:333) at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:325) at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:453) at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251) at java.lang.Thread.run(Thread.java:679) 3. C2012-03-05 18:10:24.939 Uncaught exception from servlet java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory at net.sourceforge.stripes.util.Log.getInstance(Log.java:43) at net.sourceforge.stripes.controller.StripesFilter.clinit(StripesFilter.java:57) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:532) at java.lang.Class.newInstance0(Class.java:372) at java.lang.Class.newInstance(Class.java:325) at org.mortbay.jetty.servlet.Holder.newInstance(Holder.java:153) at org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:92) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:662) at org.mortbay.jetty.servlet.Context.startContext(Context.java:140) at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1250) at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517) at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:467) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:202) at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:171) at
Re: [appengine-java] Best way to dynamic scheduled task
It depends on your data volume and timeline... lots of alarms set out 12 months mean you have a huge churn just to keep state. And you'll have to use the task-name trick to prevent tasks which execute twice (they do sometimes) from re-enlisting a second future task, otherwise you will end up with people getting multiple alarms. Also, what happens when you want to kill an alarm or reset its date? Keeping state in the task queue is a PITA. What if you want to ask has the Alarm been fired or how many alarms are pending? Life is much simpler if you just store an Alarm object. With the minimum indexes, it ends up being maybe 10 datastore write ops per alarm lifetime. So 100,000 alarms (not counting actual work like email delivery) costs you $1. This is not something to lose sleep over. Jeff On Mon, Mar 5, 2012 at 10:54 AM, ale aleee...@gmail.com wrote: This was like my first idea, and I think that will work. My doubt was about the use of the resource... I think that using task queue (as suggested by Ian Marshall) is less expansive than a cron job... Are you agree? Thanks Alessandro On Monday, March 5, 2012 4:19:04 PM UTC+1, Jeff Schnitzer wrote: When you set the alarm you want to create an Alarm object in the datastore somewhere. Have a cron that queries this table by date every minute. When it finds an entity, send the email and delete the Alarm. You need to record some way of recognizing when specific Alarms have been sent; you can't always guarantee that cron executes on schedule or that clocks are not skewed by some random number of minutes. The natural way to do this is by creating/deleting an object. Jeff On Sun, Mar 4, 2012 at 9:10 AM, ale ale.@gmail.com wrote: Hi everybody,Hello I'm writing an application that works as a calendar online (like google calendar). My question is: if a user sets an event to an hour, and I wanted to put an alarm to alert him (eg an email), how can I do ? I save the date of the event in a table, then I put up a task that checks every minute, and if the corret hour and send him the mail? There are better ways? thanks a lot! Alessandro -- -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/k0jLG8FQL8QJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Eclipse shortcut key to launch/debug app engine web application
This. Please. This has been killing me for years. GWT adds a little restart button on the console. I want that button for non-GWT apps - and specifically a hotkey that will press it. For some apps you can effectively get this by executing touch war/WEB-INF/appengine-web.xml. However, appstats doesn't survive this and generates ClassCastExceptions: http://code.google.com/p/googleappengine/issues/detail?id=7052 Note: I don't just want start, I want restart. The red kill button is usually buried: Console is not always the forwardmost tab, and even on the Console tab there are layers of consoles. It takes a lot of clicking to stop the app. Jeff On Sat, Mar 3, 2012 at 6:04 AM, vraa veera.k.sun...@gmail.com wrote: In Eclipse, I find it unproductive to navigate to Run / web application every time when I want to launch a web application in app engine. Eclipse does provide a shortcut *Alt+Shift+X *and then launch a application, but this doesn't list App Engine web application. So, I have to use the mouse every time. In Eclipse, Is there a way that I can setup a shortcut key to launch a App engine web application either in Run/Debug mode? -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/aEci0TlYLmMJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Problem with exception on server startup when GAE sdk is in svn
Time to migrate to Git! :-) Jeff On Thu, Mar 1, 2012 at 9:25 AM, Rick Mangi r...@broadcastr.com wrote: Dilbert, see my reply on the ticket. Max - that comment about svn being writable is nonsense. I wish you guys would talk to your own support engineers. I tried that solution with my premier support contact and it doesn't even make sense. svn directories are always writable. This is a bug, it's reproducible and it's going to be fixed in 1.6.4 (supposedly). The workaround is posted in my comment on the ticket. It took me weeks of back and forth to get the google engineers to see the problem. If someone had just tried it out I'm sure they would have seen the problem right away. On Thursday, February 2, 2012 4:05:43 AM UTC-5, dilbert wrote: We keep the GAE sdk (together with the project) in svn. Yesterday we upgraded to GAE sdk 1.6.2. It worked fine until we checked it into the svn repo. After that when the server is started it dies with the following message: Exception in thread main java.lang.ExceptionInInitializ**erError at com.google.appengine.tools.uti**l.Logging.initializeLogging(Lo* *gging.java:35) at com.google.appengine.tools.dev**elopment.gwt.AppEngineLauncher* *.start(AppEngineLauncher.java:**77) at com.google.gwt.dev.DevMode.doS**tartUpServer(DevMode.java:509) at com.google.gwt.dev.DevModeBase**.startUp(DevModeBase.java:1068* *) at com.google.gwt.dev.DevModeBase**.run(DevModeBase.java:811) at com.google.gwt.dev.DevMode.mai**n(DevMode.java:311) at sun.reflect.NativeMethodAccess**orImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccess**orImpl.invoke(NativeMethodAcce* *ssorImpl.java:39) at sun.reflect.DelegatingMethodAc**cessorImpl.invoke(DelegatingMe* *thodAccessorImpl.java:25) at java.lang.reflect.Method.invok**e(Method.java:597) at com.intellij.rt.execution.appl**ication.AppMain.main(AppMain.j* *ava:120) Caused by: java.lang.NullPointerException at com.google.appengine.tools.inf**o.SdkInfo.getLibs(SdkInfo.java* *:77) at com.google.appengine.tools.inf**o.SdkInfo.determineOptionalLib* *s(SdkInfo.java:284) at com.google.appengine.tools.inf**o.SdkInfo.determineOptionalUse* *rLibs(SdkInfo.java:254) at com.google.appengine.tools.inf**o.SdkInfo.init(SdkInfo.java:23* *3) at com.google.appengine.tools.inf**o.SdkInfo.getSdkRoot(SdkInfo.j* *ava:182) at com.google.appengine.tools.inf**o.SdkImplInfo.clinit(SdkImpl* *Info.java:19) ... 11 more We keep the sdk in svn since version 1.3.1 and this has never happened before. Any ideas on why this is happening? Any solution? Thanks. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/dJrB5UEdA7kJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Javaland scheduler behavior
There's been a lot of discussion of the scheduler behavior in Pythonland, but not much about it's eccentricities in Javaland. I have a threadsafe=true Java app. Let's say every request completes in exactly 1s. Settings are: idle instances min 1 max 1, latency auto/auto. Here is what I expect: * Instance1 starts up and becomes permanently resident * Instance1 serves concurrent requests up to some arbitrary CPU capcity * When Instance1 exceeds capacity: * Instance2 starts warming up * All requests remain in the pending queue for Instance1, getting processed at 1/s * concurrency * Instance2 is ready and starts processing new requests, sharing the load with Instance1 What I actually see (as far as I can determine): * Instance1 starts up and becomes permanently resident * Instance1 supports almost no concurrency. At most it's 2. (no, my app is not particularly compute intensive) * A new request comes in which for some reason can't be handled by Instance1: * Instance2 starts warming up * The new request is blocked on Instance2's pending queue, waiting 10-20s for Instance2 to be ready * In the mean time, Instance1 is actually idle * Another new request comes in and starts up Instance3 * Possibly this is while Instance2 is warming up * AFAICT, Instance1 is taking a coffee break The net result is that I have an idle website with 1 user (me) clicking around and I've already gotten multiple 20s pauses and three instances. Something is seriously wrong here. Whether or not it's rational to have so many instances started, pending requests shouldn't be shunted to non-warmed-up servers, right? I've tried upping the min latency to a high value to see if this improves the situation. If this works... shouldn't min latency *always* be as high as the startup time for an instance? I know it's been said before, but it needs to be said again... the guidance for scheduler configuration is really, really inadequate. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Javaland scheduler behavior
Thanks for the link. This doesn't directly address the main point I'm worried about though: * Why does any user-facing request ever sit in the pending queue for an instance that is warming-up when there is a perfectly good instance sitting there? My problem is not that GAE is spinning up new instances. My problem is that users are waiting for it. There's an instance sitting there with a low wait time in the pending queue - why is GAE shunting new requests to instances with an effective 10-20s wait time instead? Jeff On Tue, Feb 28, 2012 at 12:00 PM, Francois Masurel f.masu...@gmail.comwrote: Hi Jeff, Check this post : https://groups.google.com/forum/#!msg/google-appengine/3t5muVhXajg/SSFU-udjIhwJ Setting Idle instances to automatic, as Johan Euphrosine suggested, seems to have solved the problem, at least temporarily (I was at 1 minimum and 1 maximum before). I'm just a bit scared now to get over my daily quotas. François On Tuesday, February 28, 2012 5:34:36 PM UTC+1, Jeff Schnitzer wrote: There's been a lot of discussion of the scheduler behavior in Pythonland, but not much about it's eccentricities in Javaland. I have a threadsafe=true Java app. Let's say every request completes in exactly 1s. Settings are: idle instances min 1 max 1, latency auto/auto. Here is what I expect: * Instance1 starts up and becomes permanently resident * Instance1 serves concurrent requests up to some arbitrary CPU capcity * When Instance1 exceeds capacity: * Instance2 starts warming up * All requests remain in the pending queue for Instance1, getting processed at 1/s * concurrency * Instance2 is ready and starts processing new requests, sharing the load with Instance1 What I actually see (as far as I can determine): * Instance1 starts up and becomes permanently resident * Instance1 supports almost no concurrency. At most it's 2. (no, my app is not particularly compute intensive) * A new request comes in which for some reason can't be handled by Instance1: * Instance2 starts warming up * The new request is blocked on Instance2's pending queue, waiting 10-20s for Instance2 to be ready * In the mean time, Instance1 is actually idle * Another new request comes in and starts up Instance3 * Possibly this is while Instance2 is warming up * AFAICT, Instance1 is taking a coffee break The net result is that I have an idle website with 1 user (me) clicking around and I've already gotten multiple 20s pauses and three instances. Something is seriously wrong here. Whether or not it's rational to have so many instances started, pending requests shouldn't be shunted to non-warmed-up servers, right? I've tried upping the min latency to a high value to see if this improves the situation. If this works... shouldn't min latency *always* be as high as the startup time for an instance? I know it's been said before, but it needs to be said again... the guidance for scheduler configuration is really, really inadequate. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/zULJyO6VtHwJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Javaland scheduler behavior
I just realized I sent this to the wrong (java-specific, deprecated) Google Group. For those of you on both the google-appengine list and this one, I apologize in advance for the duplicate thread I'm about to start :-( Jeff On Tue, Feb 28, 2012 at 12:25 PM, Jeff Schnitzer j...@infohazard.orgwrote: Thanks for the link. This doesn't directly address the main point I'm worried about though: * Why does any user-facing request ever sit in the pending queue for an instance that is warming-up when there is a perfectly good instance sitting there? My problem is not that GAE is spinning up new instances. My problem is that users are waiting for it. There's an instance sitting there with a low wait time in the pending queue - why is GAE shunting new requests to instances with an effective 10-20s wait time instead? Jeff On Tue, Feb 28, 2012 at 12:00 PM, Francois Masurel f.masu...@gmail.comwrote: Hi Jeff, Check this post : https://groups.google.com/forum/#!msg/google-appengine/3t5muVhXajg/SSFU-udjIhwJ Setting Idle instances to automatic, as Johan Euphrosine suggested, seems to have solved the problem, at least temporarily (I was at 1 minimum and 1 maximum before). I'm just a bit scared now to get over my daily quotas. François On Tuesday, February 28, 2012 5:34:36 PM UTC+1, Jeff Schnitzer wrote: There's been a lot of discussion of the scheduler behavior in Pythonland, but not much about it's eccentricities in Javaland. I have a threadsafe=true Java app. Let's say every request completes in exactly 1s. Settings are: idle instances min 1 max 1, latency auto/auto. Here is what I expect: * Instance1 starts up and becomes permanently resident * Instance1 serves concurrent requests up to some arbitrary CPU capcity * When Instance1 exceeds capacity: * Instance2 starts warming up * All requests remain in the pending queue for Instance1, getting processed at 1/s * concurrency * Instance2 is ready and starts processing new requests, sharing the load with Instance1 What I actually see (as far as I can determine): * Instance1 starts up and becomes permanently resident * Instance1 supports almost no concurrency. At most it's 2. (no, my app is not particularly compute intensive) * A new request comes in which for some reason can't be handled by Instance1: * Instance2 starts warming up * The new request is blocked on Instance2's pending queue, waiting 10-20s for Instance2 to be ready * In the mean time, Instance1 is actually idle * Another new request comes in and starts up Instance3 * Possibly this is while Instance2 is warming up * AFAICT, Instance1 is taking a coffee break The net result is that I have an idle website with 1 user (me) clicking around and I've already gotten multiple 20s pauses and three instances. Something is seriously wrong here. Whether or not it's rational to have so many instances started, pending requests shouldn't be shunted to non-warmed-up servers, right? I've tried upping the min latency to a high value to see if this improves the situation. If this works... shouldn't min latency *always* be as high as the startup time for an instance? I know it's been said before, but it needs to be said again... the guidance for scheduler configuration is really, really inadequate. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/zULJyO6VtHwJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] max keys for memcache
You are overthinking this. To the point of lunacy. Jeff On Wed, Feb 22, 2012 at 5:07 PM, Kesava Neeli nke...@gmail.com wrote: We would like to have a map of all users in datastore in mem cache so that we can look up for any username/email/number to verify if he is already an user of our app. Ex: A user wants to send some message to an email address from our mobile app. We want to find if there is any user in our database with that email id and do something different. We could do a costly query all the time or do a quick lookup in MemCache if user exists. For each of our user account, we have a primaryKey but we cannot query with that field all the time. We would like to update the mem cache like once a day with all user accounts and tag them with multiple fields. So interested in the size of the mem cache for an app with 4-5 million records which are small key/value pairs. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/ssbcFx9IxmAJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] max keys for memcache
Your requirement appears to be able to look up user by email. The datastore does exactly this for you. The idea of loading your entire datastore into memcache just so you can do lookups is totally absurd. Jeff On Wed, Feb 22, 2012 at 6:05 PM, Kesava Neeli nke...@gmail.com wrote: Jeff, If you understood my requirement, I would love to hear any suggestions you have if you think MemCache is not a solution. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/blK7tIyiUhMJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] max keys for memcache
Are all of these 1-2 million users going to be active at the same time? If not, why do you care if it all fits in memcache? On the other hand, if they *are* all going to be active at once, I would be very curious to know what happens... Jeff On Tue, Feb 21, 2012 at 2:07 PM, Kesava Neeli nke...@gmail.com wrote: Hi IKai, Could you point me to the docs about the max memory for the app for MemoryCache? We have a mobile app with many user accounts and each account has a primary key. We expect to have 1-2million accounts in the datastore. We need to store many entries in mem cache to find if account exists based on other fields in our user account data. i.e a map of 3 or 4 fields per account pointing to the primary key. So we will have 3-4 entries in mem cache for each user account. Each entry will be simple key/value string objects. Do you have any recommendations on how to use MemCache here? How do we need to handle the memory for 5 million entries in cache? Thanks Neeli -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/HmjI7KSoAz4J. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Is it safe for one ancestor to have millions of children in one List object?
The biggest question is why would you want to do this? From the low-level api perspective, you can't have a list property with more than 5k items. But maybe JDO models this list as a proxy to a query. When you try to load that proxy it's going to load all 5m items out of the datastore and you are going to wait a long time. It will not complete in reasonable time scales - it might take over an hour, and unless you're running in a backend you will certainly blow the request time limit. And even there you'll blow the amount of time for a single query to run, which is something like one minute. So no, you can't do this if you actually plan to load the list directly. But maybe you're just using this to model the data and plan to use filtering operations for queries, in which case you can do that as long as you never load the whole set. The real question is not whether you can store the data (presumably there's a way to make it possible) but whether you can execute whatever queries you have min mind. You should post examples of those. FWIW, it sounds like a terrible idea. Store them all in separate EG's and use normal queries to access them. Jeff On Fri, Feb 17, 2012 at 10:07 AM, hendrix.jason hendrix.ja...@gmail.comwrote: Well, I plan to write in batches of 1,000. So a million items would take 1,000 batch writes. In the worst case scenario of 1 write per second that's about 17 minutes. That's acceptable for a once nightly job. On Feb 17, 2:31 am, Cyrille Vincey cvin...@qunb.com wrote: Forget about this way : even if this having millions of children entities for one ancestor is theoretically feasible, you'll hit a write performance issue when you generate those entities : only 1 write / second in one given entity group (ancestor). On 17 fév, 03:16, hendrix.jason hendrix.ja...@gmail.com wrote: If I have a @PersistenceCapable class that contains a List of another @PersistenceCapable class, and that List is millions of items in size, is that OK? Does that exceed a maximum for Java's List size? What is the best way for a parent to have millions of children and still have it easy to add new ones without loading tons of data in to memory? -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] scheduling 1000 emails
You can schedule tasks to execute in the future. It doesn't guarantee that the task will be executed then, however. You might find the queue backed up, in which case you'll get a long delay then all the tasks executing rapidly. Alternatively, you can throttle the task queue to execute once per second. But that will affect the whole queue. Jeff On Mon, Feb 20, 2012 at 5:39 PM, marco marco.m.al...@gmail.com wrote: Goal: - Transform a request to send ~1000 emails by sending one each X seconds. Problems found: - Cron doesn't allow scheduling for every second (and has to be one all the time; not just when requested;) - Using recursive deferred tasks (the first launching the second after a Thread.sleep(X * 1000) ) seem to break the 30 seconds rule (1000 * X seconds 30 seconds) Did I miss something? Do you recommend any alternative? Thank you. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/nAEat-B1qZQJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Remote API security ?
On Fri, Feb 17, 2012 at 10:22 PM, meric mivi...@gmail.com wrote: Can i use the remote api on standalone applications that users, other than the admin, can download and use ? Can the users see the username/password of my GAE account that i have to put in my app to access the api ? Just making sure :) PS: Tried to post the same message on stackoverflow first, and i got: Oops! Your question couldn't be submitted because: It does not meet our quality standards. I very nearly have the same complaint. You should be more verbose when asking questions; I'm still not sure I know exactly what you are asking. The remote API grants API level access to any administrator in your app. There's no granularity. If you plan on exposing an API to non-admin users, you need to craft the API and secure it yourself. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Batch puts
From my experiments, writing large #s of entities to a single EG in a batch put is quite fast and does not cause the throughput problem. Jeff On Tue, Feb 7, 2012 at 4:43 PM, Andrew Myers am2...@gmail.com wrote: I'm sure this is a FAQ but I haven't been able to find the answer so far. I have read there's a limit of 2-3 writes per second per entry group. If you batch out multiple entities that all have the same parent is this ok? Or does it count as multiple writes? I understand it is multiple writes in terms of write quotas but am not sure if it's a problem in terms of the writes per group per second? Many thanks, Andrew -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: How do I run DeferredTasks in a Unit Test
This is a blast from the past. There are now official tools for unit
testing deferred tasks:
http://code.google.com/appengine/docs/java/tools/localunittesting.html#Writing_Deferred_Task_Tests
However, ignore the latch stuff - it works erratically, and if you
have an even slightly complicated system it's hard to predict how many
tasks you expect ahead of time. I just add a 1s delay for the tasks
to settle.
If you care about unit testing with task queues, please star these two issues:
http://code.google.com/p/googleappengine/issues/detail?id=6818
http://code.google.com/p/googleappengine/issues/detail?id=6819
Jeff
On Thu, Feb 2, 2012 at 3:20 PM, Riley Eynon-Lynch rileyl...@gmail.com wrote:
private QueueStateInfo getQueue(String queueName) {
return
LocalTaskQueueTestConfig.getLocalTaskQueue().getQueueStateInfo().get(queueName);
}
will do. Thanks again!
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine-java/-/r3vJS5G9gkAJ.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Logging when using local unit testing
A appreciate the out-of-the-box suggestion, but this strikes me as an extraordinarily invasive change just to get logging to render the way it is supposed to. Presumably there is something in the GAE stack that monkeys with java.util.logging. I just want to know what it is and how to beat it into submission when running in the local test mode. Or, if somehow I'm the only one this happens to, I'd like to know that as well :-) Thanks, Jeff On Sun, Jan 29, 2012 at 11:18 PM, Nichole nichole.k...@gmail.com wrote: B.T.W. eclipse may use other arguments. see the bottom of the jdpa document for examples: http://docs.oracle.com/javase/6/docs/technotes/guides/jpda/conninv.html On Jan 29, 4:09 pm, Nichole nichole.k...@gmail.com wrote: To simplify the problem, can you use your IDE purely as jdwp agent? For maven (which for multi-module projects w/ profiles, current netbeans and eclipse plugins can have difficulty resolving a class, but are well worth that trouble for everything they do do right :) I often use this: mvn -Dmaven.surefire.debug -Dtest=$unittest test which internally is using -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000 - Xnoagent -Djava.compiler=NONE On Jan 28, 2:28 am, Simon Knott knott.si...@gmail.com wrote: When I get near a computer I'll see what settings I've got. I swear I see log output for my unit tests. Cheers, Simon -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Logging when using local unit testing
Does anyone have logging working when using the local unit testing system (ie LocalServiceTestHelper)? For the life of me I cannot get any log output. System.out works, but the jdk logger does not. I'm running my (TestNG) tests through the eclipse plugin. I'm pretty certain this isn't a TestNG issue because my TestNG tests in non-GAE projects log fine. FWIW, when in normal dev mode my logging works great... just when running tests, no love. It would help me just to know that someone has it working. Thanks, Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] how to store java Date type in specific timezone?
You must store the TZ as a separate field. Alternatively, if you don't need the date to be indexed, you can convert it back and forth to a String that preserves TZ info (eg, the ISO-8601 format). Jeff On Wed, Jan 25, 2012 at 12:53 PM, Kesava Neeli nke...@gmail.com wrote: Hi, I have a java.util.Date data type in my object. Date now = Calendar.getInstance(TimeZone.getTimeZone(America/Los_Angeles), Locale.US).getTime(); When I persist the object and look at the entity, I see the type stored as gd:when as the kind of element. I see the value as 2012-01-25 01:01:37.938000 which is UTC time. How can I save the time in my desired timezone? The java Date type has no timezone setting explicitly and appengine only allows java.util.Date as a valid date type from java. Thanks Neeli -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/__Q5MEGh6jwJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] how to store java Date type in specific timezone?
Taking a step back, I should say that if you're using java.util.Date with timezone info, you're doing it wrong. java.util.Date is a trainwreck, and any methods that affect an internal notion of a TZ have been deprecated. If you care about timezones, using Date is a mistake - it's best to think of Date as purely a wrapper for millis-since-epoch, nothing more. I strongly recommend purging java.util.Date from your code and using Joda Time instead. You can use DateTime to represent an instant (and set TZ if you want to format it for a locale) or you can use LocalDateTime to store an abstract notion of a time without a TZ - which, if you're taking a date as user input, is often what you really want to persist. Jeff On Wed, Jan 25, 2012 at 2:17 PM, Matthew Jaggard matt...@jaggard.org.uk wrote: You could create a sortable date format string. For example, convert the date to GMT and store it -mm-dd hh:mm:ss.nnn then store the timezone afterwards so you can convert back to the date/time in the timezone. On 25 Jan 2012 19:05, Kesava Neeli nke...@gmail.com wrote: Thanks. That's what I was doing for some date fields now. Store the date in a well formatted string and then do conversions. But it becomes tough when you want to build a query get me all records in last day and the datastore for that object contains thousands of records. With the strings, you need to iterate through all records, get a date and then do a comparison. With date data type, you can rely on appengine to return correct data. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/7sg4mVI55SMJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] how to store java Date type in specific timezone?
I store DateTime, LocalDate, LocalDateTime, and DateTimeZone quite a lot, but I use Objectify (disclosure: I'm the lead author) which handles the translation for you (if you enable it). ReadableInstant objects (eg DateTime) are stored natively as java.util.Date and Local objects (LocalDate, LocalDateTime, etc) are stored as the ISO-8601 string. Jeff On Wed, Jan 25, 2012 at 4:44 PM, Kesava Neeli nke...@gmail.com wrote: Thanks Jeff. I used JodaTime before but I was stuck with java.util.Date since appengine allows only standard java data types. I don't think Joda Time object will work on appeninge. Will give a try. Have you stored JodaTime objects in datastore? -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/vHFYiyYJGTkJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] lost trying to understand JDO on BigTable: please help coding example
On Tue, Jan 24, 2012 at 4:21 PM, John Goche johngoch...@googlemail.com wrote: The main reason I ask is that loading two entities is about twice as expensive as loading one twice the size. Tough. I cannot sacrifice code readability here. This is an intriguing attitude. You have arrived with a preconceived notion of exactly what you want your Java data structure to look like and you are now trying to force GAE into that mold. If you stay this path, you will very likely end up abandoning appengine in frustration. At best, you will produce software that works poorly. You have a long learning curve ahead of you, both in terms of learning how the datastore works and learning how to create a data model that is both performant and maintainable. I have two pieces of advice: * Read the Storing Data section of the GAE manual *completely*. The nature of keys and ancestry is only one of many pieces of knowledge you must master to develop applications here. Even though it looks vaguely like one on the surface, this is not an RDBMS and your learned instincts are likely wrong. Do not believe you can just understand JDO. * When you ask for help, try be somewhat more general describing what you want to do. You're asking about how to specifically arrange primary keys in Java classes when it may be the case that what you want to do is embed one class in the other. There is not one canonical way to model entities on GAE, so if you want help, start from the perspective of this is roughly what I'm trying to do... is it the right way? As to your specific problem, I think you'll get a lot more help if you rephrase the question. Everyone here is just taking guesses at what you're trying to ask. Suerte, Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] json communication with google app engine
I use Resteasy with Jackson. It works great. Uses the standard JAX-RS annotations. http://docs.jboss.org/resteasy/docs/2.3.0.GA/userguide/html_single/index.html Jeff On Mon, Jan 16, 2012 at 11:51 PM, angstrom348 linux.mod...@gmail.com wrote: I am trying to allow for app engine interaction through JSON calls and return JSON back to the client (a mobile phone running a client app). A good example exists for python users at http://code.google.com/appengine/articles/rpc.html. How would you set up a basic ability to listen for JSON requests, parse them, and convert any results back to JSON and push them back as a JSON response? Any examples I have been able to find appear either vague or outdated. Thank you. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/aCUsTbjjtVEJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] json communication with google app engine
As a bonus, you can also use Resteasy for webpage rendering using this thin framework on top: http://code.google.com/p/htmleasy/ (I wrote it originally, although another guy maintains it now. I still use it, and - if I may say so - it's great) Jeff On Wed, Jan 18, 2012 at 12:38 AM, Jeff Schnitzer j...@infohazard.org wrote: I use Resteasy with Jackson. It works great. Uses the standard JAX-RS annotations. http://docs.jboss.org/resteasy/docs/2.3.0.GA/userguide/html_single/index.html Jeff On Mon, Jan 16, 2012 at 11:51 PM, angstrom348 linux.mod...@gmail.com wrote: I am trying to allow for app engine interaction through JSON calls and return JSON back to the client (a mobile phone running a client app). A good example exists for python users at http://code.google.com/appengine/articles/rpc.html. How would you set up a basic ability to listen for JSON requests, parse them, and convert any results back to JSON and push them back as a JSON response? Any examples I have been able to find appear either vague or outdated. Thank you. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/aCUsTbjjtVEJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] json communication with google app engine
I have a long history with JBoss, plus I have friends who still work there - although I don't know Bill Burke well and don't particularly love the way he runs Resteasy (insufficiently self-critical for my tastes, and not enough internal code comments). That said, Resteasy works well, the product continues to evolve, and I almost always find hooks for the deep integration I want - even if it involves reading uncommented source code to find them. To be honest, I haven't tried any other JAX-RS provider. I've looked, but the grass didn't look greener. Jeff On Wed, Jan 18, 2012 at 12:47 AM, Wilson MacGyver wmacgy...@gmail.com wrote: I'm curious if you could share why you picked resteasy over jersey of other jax-rs implementations? On Jan 17, 2012, at 7:38 PM, Jeff Schnitzer j...@infohazard.org wrote: I use Resteasy with Jackson. It works great. Uses the standard JAX-RS annotations. http://docs.jboss.org/resteasy/docs/2.3.0.GA/userguide/html_single/index.html Jeff On Mon, Jan 16, 2012 at 11:51 PM, angstrom348 linux.mod...@gmail.com wrote: I am trying to allow for app engine interaction through JSON calls and return JSON back to the client (a mobile phone running a client app). A good example exists for python users at http://code.google.com/appengine/articles/rpc.html. How would you set up a basic ability to listen for JSON requests, parse them, and convert any results back to JSON and push them back as a JSON response? Any examples I have been able to find appear either vague or outdated. Thank you. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/aCUsTbjjtVEJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: problem with Content-type: null with css files
One thing that would make this helpful would be the ability to automatically delete old versions. The current limit of 10 versions means I have to manually go through and delete old ones before I can deploy new ones. It would be nice to automate a scrolling window of versions. Of course, not all versions can be scrolled off; our maintenance mode version (never used, but there for emergencies) should always remain. appcfg.sh needs a way to list versions and delete individuals. Jeff On Sat, Jan 14, 2012 at 9:52 AM, andrew andrew.macken...@bcntouch.com wrote: A FYI about alternating versions for deployment to production... Our build script names the version according data from SVN (GIT would be similar), as a working copy (name = data and time), revision (name=r123), or tag (name = tag name = our release version number). Then the uploaded versions in appengine Dashboard all show either with their release number (or revision, or date). You can have up to 10 of them uploaded in production, working against production data - but just one as the 'default'... On doing a new release/deploy to production (after previously testing on another appid in integration and qa), we test the basics via the specific version urlif everything looks fine we then make the new version the default. If something goes wrong, you can quickly sqitch back to any of the previous 9 releases you have made, usually the penultimate one off course. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Abridged summary of google-appengine-java@googlegroups.com - 8 Messages in 5 Topics
Google has said in the past that there is no negative performance effect from enabling XG transactions if you only use a single entity group. The rational thing to do is always enable XG transactions all the time, but be aware when you span multiple entity groups in a transaction. I think it was silly of them to add this flag. Jeff On Sat, Jan 14, 2012 at 8:23 AM, Mike Lawrence m...@systemsplanet.com wrote: I guess I didn't expect to see mention of requiring entity groups for transactions now that GAE supports cross-entity-group transactions. Does GAE really need both modes? Is there a performance benefit to not enabling global transactions? On Jan 13, 3:25 m, Ikai Lan (Google) ika...@google.com wrote: Actually, I took a look at these docs and it seems like we do talk about cross group transactions. There are a few areas where we talk about exceptions being thrown for transactions on entities in different entity groups, but that's expected behavior. You must specify that you want a cross-group transaction. We'd definitely like to improve these docs, however. Do you have any suggestions or areas where this documentation seems confusing? -- Ikai Lan Developer Programs Engineer, Google App Engine plus.ikailan.com | twitter.com/ikai On Fri, Jan 13, 2012 at 12:23 PM, Ikai Lan (Google) ika...@google.comwrote: Thanks! I'll file a bug for this since it's not merely a one line fix. We'll probably want to better explain cross-entity group transactions a bit and link to the documentation on that subject. -- Ikai Lan Developer Programs Engineer, Google App Engine plus.ikailan.com | twitter.com/ikai On Fri, Jan 13, 2012 at 11:12 AM, Mike Lawrence m...@systemsplanet.comwrote: Ikai, Docs still say transactions wont span entity groups: http://code.google.com/appengine/docs/java/datastore/transactions.html http://code.google.com/p/googleappengine/wiki/SdkForJavaReleaseNotes Version 1.5.5 - Oct 11, 2011 App Engine now supports Cross Group (XG) transactions with the High Replication Datastore, which allow you to perform transactions across multiple entity groups. http://code.google.com/appengine/docs/java/datastore/transactions.html Sincerely, Mike Lawrence On Thu, Jan 12, 2012 at 12:23 AM, google-appengine-java@googlegroups.com wrote: Today's Topic Summary Group:http://groups.google.com/group/google-appengine-java/topics problem with Content-type: null with css files [1 Update] Minor update to Java SDK: 1.6.1.1 [1 Update] [No Subject] [1 Update] proxying datastore request [3 Updates] how to get the correct time for germany? [2 Updates] problem with Content-type: null with css files Ikai Lan (Google) ika...@google.com Jan 11 05:22PM -0800 We have an SDK update that resolves this issue on upload (I've also posted in a separate thread about this): ...more Minor update to Java SDK: 1.6.1.1 Ikai Lan (Google) ika...@google.com Jan 11 05:21PM -0800 Hey everyone, There's a new version of the Java SDK available for download. You can get it here: http://code.google.com/p/googleappengine/downloads/detail?name=appeng... ...more [No Subject] Aviv Keren aviv...@gmail.com Jan 11 11:08PM +0200 Ok tnx! ...more proxying datastore request meiaestro jmalbre...@gmx.de Jan 11 07:18AM -0800 Hi all! I was thinking about making my RPC calls to the server (datastore commands) more secure against java script or data stream modifications on client side (when user is already signed ...more Ikai Lan (Google) ika...@google.com Jan 11 10:44AM -0800 Hi there! - a logged in user could manipulate the RPC call (he could for example exchange his userID by the ID of someone else and access the data of this user) - to avoid that I need to ...more meiaestro jmalbre...@gmx.de Jan 11 12:31PM -0800 Thanks for the reply. You're right. I did not use the Users API, as I do not want to force the users to have a google account (or any other existing account). This will be optional at a ...more how to get the correct time for germany? vega _v...@vr-web.de Jan 11 07:24AM -0800 hi everyone. since the days are getting longer, i noticed that in 2 months ill have an error with my getTime() Method, which gives me the current time in a human friendly way. /** ...more Ian Marshall ianmarshall...@gmail.com Jan 11 08:44AM -0800 The code below is how I do this. It works for summer and winter times automajically. public static final Locale G_LOCALE_UK = Locale.UK; public static final String G_S_TIME_ZONE_ID = ...more You received this message because you are subscribed to the Google Group google-appengine-java. You can post via email. To unsubscribe from this group, send an empty message. For more options, visit this group. -- You received this message because you
Re: [appengine-java] An O/R mapper to use with cloud sql
On Mon, Jan 2, 2012 at 12:22 PM, Arjan arjan.br...@gmail.com wrote: I keep getting errors of the hibernate configuration that can not read 'public static' properties. It's due to some reflection limitation on the GAE. So there may be a more fitting OR mapper to use in GAE. Are you absolutely certain of the cause? It sounds odd. Perhaps consult the wisdom of this group with stacktraces? Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] An O/R mapper to use with cloud sql
If you are using cloud sql, what's wrong with hibernate? Jeff On Sat, Dec 31, 2011 at 4:24 AM, Arjan arjan.br...@gmail.com wrote: Hi all, When google announced cloud sql i was realy happy to port my web application from the nosql datastore to the MySQL instance. I'm a bit used to hibernate, kbut support for hibernate is terible in GAE. Does anyone know of good O/R mappers to use in GAE? Regards, Arjan -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/f1H_GPpjFYUJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- We are the 20% -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Any inputs for How to use the DataStore for Java?
It won't be a problem with the Low-Level API. It works just fine with exotic unicode strings. Your problem is somewhere else in your app, probably somewhere that you are generating the string or rendering it. Jeff On Wed, Dec 28, 2011 at 10:59 AM, Doug douganderson...@gmail.com wrote: Those are all great references indeed. However, I'm still stumped on this one which should be basic. Using the Low Level API, I still cannot why String Properties break on special characters. thisEntity.setProperty(stringDescription ,Save 20% Off anything in store); A little help on how to (convert?) and store (then get...) Strings containing non-alphanumeric characters with LLAPI? Many Thanks Doug -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/Whb8noU7rZMJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- We are the 20% -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Problem starting development after adding backends.xml
I don't think very many people are using backends... and since a lot of the support here comes from the community, your question is a little esoteric. Try disabling appstats on the backend. Maybe appstats is incompatible with backends. Jeff On Thu, Dec 22, 2011 at 3:53 AM, Rick Mangi r...@broadcastr.com wrote: Apparently they do not. It's amazing that a product with such horrid customer support has so many users. That's not likely to continue now that the paas market has competition. On Dec 19, 6:41 pm, Rick Mangi r...@broadcastr.com wrote: Do the google app engine developers even read these messages? It's amazing how few questions in this group actually seem to get answered compared to open source projects... On Dec 16, 5:02 pm, Rick Mangi r...@broadcastr.com wrote: After adding a backends.xml file to my application I'm no longer able to startup my development environment. Has anybody seen this before? Removing the file clears everything up, but I really need to get a backend running :-) [ERROR] Unable to start App Engine server java.lang.ClassCastException: com.google.appengine.tools.appstats.Recorder cannot be cast to com.google.appengine.tools.development.ApiProxyLocal Unable to start embedded HTTP server com.google.gwt.core.ext.UnableToCompleteException: (see previous log entries) at com.google.appengine.tools.development.gwt.AppEngineLauncher.start(AppEngin eLauncher.java: 102) at com.google.gwt.dev.DevMode.doStartUpServer(DevMode.java:509) at com.google.gwt.dev.DevModeBase.startUp(DevModeBase.java:1068) at com.google.gwt.dev.DevModeBase.run(DevModeBase.java:811) at com.google.gwt.dev.DevMode.main(DevMode.java:311) at com.google.appengine.tools.development.AbstractContainerService.startup(Abs tractContainerService.java: 235) at com.google.appengine.tools.development.BackendServers $ServerWrapper.startup(BackendServers.java:816) at com.google.appengine.tools.development.BackendServers.startupAll(BackendSer vers.java: 304) at com.google.appengine.tools.development.DevAppServerImpl.start(DevAppServerI mpl.java: 155) at com.google.appengine.tools.development.gwt.AppEngineLauncher.start(AppEngin eLauncher.java: 97) at com.google.gwt.dev.DevMode.doStartUpServer(DevMode.java:509) at com.google.gwt.dev.DevModeBase.startUp(DevModeBase.java:1068) at com.google.gwt.dev.DevModeBase.run(DevModeBase.java:811) at com.google.gwt.dev.DevMode.main(DevMode.java:311) [ERROR] shell failed in doStartupServer method I don't see anything wrong with my backends.xml -- backends backend name=thumbnailservicebatchoptimize classB2/class instances1/instances options dynamictrue/dynamic fail-fasttrue/fail-fast publictrue/public /options /backend /backends Thanks for any help! Rick Mangi -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- We are the 20% -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] datastore.delete() not deleting objects and not generating exceptions
Yeah, if you're doing a delete by previously-known key I would expect constant time operation. I have no idea what might be causing your issue, sorry. Jeff On Sun, Dec 11, 2011 at 4:26 PM, Michael michael.ol...@gmail.com wrote: By what you wrote, I assume you're referring to the query speed, not the speed of the deletion. I presume that deleting an object or collection of object requires a time that's independent of other factors in the datastore; that is, once I know the object's key, deleting it takes an essentially constant amount of time. For my particular use case, the query speed is essentially negligible because there are not that many of this type of object, but the speed of the actual delete option is often quite a bit higher than I would prefer. If I'm reading what you wrote wrong and there's some way to speed up the delete operation, that would be great. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/-hADIjTHD3UJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- We are the 20% -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] datastore.delete() not deleting objects and not generating exceptions
I haven't looked too closely at your problem, but one thing that has come
up before on this list is that it's a bad idea to do things like this:
repeat {
delete N items;
}
Basically, deleting items just flags them as deleted in the underlying
store. They are vacuumed up later. So if you delete a lot of stuff like
this it ends up being a slow O(N^2) operation as you're effectively doing
an offset over all the previously deleted items.
To efficiently delete large numbers of entities, either delete them all in
a single request or use a cursor.
Of course, this information may be out of date.
Jeff
On Thu, Dec 8, 2011 at 8:22 PM, Michael michael.ol...@gmail.com wrote:
I have an hourly cron job that deletes data from the datastore older than
one month (this data is archived elsewhere for long term storage). The
first run of that cron job on Tuesday after the datastore came back up
behaved quite unusually, ended with java.lang.OutOfMemoryError: Java heap
space, and hasn't completed once since then. While it is possible that
this is pure coincidence, I'm wondering if something done during the
maintenance resulted in this behavior. I have been unable to get this cron
job to run correctly since then.
The job is quite simple, and has been running happily for about a year; I
will present the idea here for brevity and attach the source for those
interested. In essence the job does the following:
- retrieve at most 500 entities older than 1 month by their keys only
- send all resulting keys as a list to datastore.delete()
- repeat until no results are returned
The first run after maintenance produced the attached log-excerpt.txt.
The brief version is the following:
- deleted 500 objects
- deleted 465 objects
- deleted 213 objects (repeated 395 times)
- out of memory
It seems that, after actually deleting the first 752 objects of the query,
the datastore got stuck on the next 213. The same 213 objects were sent
repeatedly to datastore.delete(). No exceptions were generated, but the
data was obviously not deleted.
The next attempt (the job was retried since it crashed) produced almost
identical output. This time, it actually deleted 174 objects, then tried
to delete the same 213 objects over and over until it, too, crashed with an
OutOfMemoryError. The run after that actually deleted 8 objects before it
crashed in the same manner. This continued until the error ran my
application out of quota for the day, at which point I got a notification
email and went to go pause the queue that these jobs run under.
Note, I am not on the high replication datastore. I do not know why this
is happening, but it is currently an insurmountable obstacle. I tried
unpausing the queue temporarily and running the problematic job, and this
time I did not even get the previously frustrating but informative output;
instead, I merely got the A serious problem was encountered . . . message
on both runs.
Any help in getting this fixed or understanding the problem would be
greatly appreciated.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine-java/-/AU89aTTQOR8J.
To post to this group, send email to
google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
We are the 20%
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: An alternative to Objectify?
On Thu, Dec 8, 2011 at 5:11 AM, datanucleus andy_jeffer...@yahoo.comwrote: Well, it sounds so. But does (or can/will) Datanucleus support at least some of the features Jeff has mentioned? Yes, several actually. And nothing is dog slow, ... though why dogs get such a bad reputation I've no idea; take a greyhound for example ... :-) The question was is caching in DataNucleus simple and transparent; that's the question I answered. I'm sorry, but you clearly haven't actually used this cache in the real world. I've looked at DataNucleus' Level2Cache interface and noticed that it serializes requests entity-by-entity. This means that every single request in a batch get triggers 1 or 2 RPCs; on a cold cache, a batch get for 10 items will require 20 separate, linear RPCs. This doesn't work - it's an order of magnitude slower than a simple datastore fetch would have been in the first place. I have already stumbled across this problem and had to delay releasing Objectify 3.1 until GAE supported the batch CAS operations that make CAS caching efficient. I had to delay upgrading one of my real-world products because of this issue. So no, datanucleus caching is not simple and transparent if your application makes use of batch fetching (ie, what should be the most important operation in most applications). Not sure also why people always want to make things into a mines better than yours contest. The only statements you'll see from me on here are in answer to specific questions about software that uses DataNucleus (and Googles plugin does that, with the emphasis on being owned by Google, and not being part of the DataNucleus project), with the attempt to aid that person use the software better. You've dumped a bunch of marketing BS in this thread (nothing could be simpler!) which does not accurately reflect reality, which is complicated and messy. It's not personal, and it's not even because I work on a competing project - it's just that I happen to know a lot about this particular subject. If you were hawking any kind of software and I happened to know the truth behind the marketing, I would call you out on it. It does, however, irritate me. It takes up some of my time to rebut these issues (see the thread on Slim3 is fast!). It takes a few minutes to post a poorly thought out marketing claim or create a half-baked benchmark. It takes actual research to figure out exactly what's wrong with them. Your sloppy thinking generates real work for me. Nowhere have I ever made any comment on Objectify's capabilities or made any criticism of it (or of similar software, such as Twig, Morphia) - perhaps due to the fact that I understand only too well how much time and effort is taken to develop something of that form. All software has particular features, and their own advantages and disadvantages ... DataNucleus (and Google's plugin) has plenty of things going for it, as does Objectify. It's for the user to choose what best fits *their* requirements using their judgement, not for people to spend time criticising. Let's just make sure those users are well and accurately informed. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: An alternative to Objectify?
On Thu, Dec 8, 2011 at 10:32 AM, Jeff Schnitzer j...@infohazard.org wrote: I'm sorry, but you clearly haven't actually used this cache in the real world. I've looked at DataNucleus' Level2Cache interface and noticed that it serializes requests entity-by-entity. This means that every single request in a batch get triggers 1 or 2 RPCs; on a cold cache, a batch get for 10 items will require 20 separate, linear RPCs. This doesn't work - it's an order of magnitude slower than a simple datastore fetch would have been in the first place. Oops, that is not quite correct. On a cold cache, a batch get for 10 items will require *30* linear RPCs to get the data, plus another 10 to populate the cache. 40 RPCs in linear execution. And that's just 10 items. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: An alternative to Objectify?
On Wed, Dec 7, 2011 at 7:03 AM, datanucleus andy_jeffer...@yahoo.comwrote: *Does anybody know whether it's as simpletransparent as with Objectify to use GAE's caching in JPA2Datanucleus on GAE?* Assuming you're using v2 of the datanucleus-appengine plugin, just define the persistence property datanucleus.cache.level2.type as javax.cache and it will use Memcached, with persistable objects put in the L2 cache at commit, and accessed when needed ... totally transparent to the user. *Nothing* gets simpler than that. H. Another one of those threads. Nothing? How do you configure caching on an entity-by-entity basis? How do you configure cache expiry times? Does this cache use CAS operations to guarantee synchronization? Does it use batch CAS operations to prevent it from being dog slow? Does it perform query caching? Will it convert appropriate queries into keys-only queries + batch get to integrate with the cache? I'd say putting @Cached on each entity you want cached is pretty simple and elegant. FWIW, I just looked at the datanucleus Level2Cache interface and no, it doesn't support batch operations. So this is useless for any application that uses batch operations (which should be most). Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: prefetchSize vs chunkSize
Alfred made this comment, which sheds a lot of light on it: On Sat, Nov 19, 2011 at 8:56 PM, Alfred Fuller arfuller+appeng...@google.com wrote: if you set chunkSize and not prefetchSize every call will try to return chuckSize results (even the first call) if you set prefetchSize and chunkSize, the first call will try to return prefetchSize results, and every call after that will try to return chunkSize results. it is there mainly for legacy issues (originally the first call never returned any results, today most queries only ever need a single rpc). Here are some use cases: I know I need ~300 results but might need a few more. - prefetchSize=300, chunkSize=20 I want to start a bunch of queries, do something else, then process all the data. - prefetchSize=MAX_INT (to maximize the number of results fetched async in the first batch) I want all results - chunkSize=MAX_INT (prefetch size is not needed because the first call will use chunk size) I want limit results - limit=limit (chunkSize is automatically maxed when a limit is set) -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Which java framework to use?
My favored stack right now: Objectify (I'm the lead developer, so this one was easy) Resteasy + Htmleasy + Guice, tightly integrated Cambridge templates (with JEXL) Coffeescript + jQuery + Handlebars.js on the frontend Heavy use of Jackson to jsonify objects, including embedded in the page html to eliminate initial ajax hits A lot of the UI is dynamic (javascript templating), but the SEO-sensitive pages are rendered completely with Cambridge. I'm happy with it so far, although we had to build a lot of silly infrastructure to manage the javascript. Jeff On Sun, Nov 20, 2011 at 7:54 AM, sppl spol...@gmail.com wrote: I am starting off with a new project using app engine java. I am trying to decide what mvc framework to use. I went through previous messages posted by group members and found this linkhttp://?place=msg%2Fgoogle-appengine-java%2F7wkMDm6kptY%2F-hxbaAyfK4sJ. I then narrowed down my choices to following frameworks, 1) Play framework - I like this framework, but it seems to have long initialization time. Also the play GAE modulehttps://github.com/guillaumebort/play-gaesupport seems to have stopped/paused. 2) Stripes framework - This seems to be lightweight, but it does not seem to be widely used. 3) Slim3 - This is another lightweight framework I liked. But most of the users of this framework are in japan and the useful posts seem to be hidden in japanese. Are there any other java frameworks that work well on app engine (lightweight and good startup time)? Any advices, pointers are appreciated! -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/c_EbGr7n0KYJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- I am the 20% -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: package javax.servlet.annotation does not exist
Occasionally eclipse gets stupid like this. Some combination of Project Clean, closing/opening the project, and closing/opening eclipse eventually sets it right. Jeff On Sat, Nov 19, 2011 at 10:23 PM, Mauricio Porras P mauricio.porra...@gmail.com wrote: I'm having this same issue, did you find a solution already? Greetings. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/2vk_8GdySmYJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- I am the 20% -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Is BlobInfoFactory thread safe?
Look at the source code. BlobInfoFactory only has one piece of member data, a DatastoreService. So if DS is thread-safe, so is BIF. I don't know if DS is thread-safe though. Jeff On Sat, Nov 19, 2011 at 9:20 AM, Marcel Overdijk marceloverd...@gmail.comwrote: Is BlobInfoFactory thread safe? It's not mentioned in the Javadoc so should I assume it's not threadsafe? It would be good if Google would mention for each service if it's threadsafe or not. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- I am the 20% -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] prefetchSize vs chunkSize
Would someone explain the exact differences between prefetchSize and chunkSize and how these interact? The documentation doesn't really help. I presume chunkSize is the number of items that get fetched on each RPC to the datastore backend, equivalent to a fetch() call in python-land. Is prefetchSize effectively the same thing as chunkSize but only affects the first chunk? Why are these separate parameters? Under what circumstances would I want them to be different? Thanks, Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Google App Engine custom domain stuck in Activate this service infinite loop
This is going to sound really weird but I've had issues like this before and switching browsers seemed to fix it. Try Firefox. Jeff On Tue, Nov 15, 2011 at 2:49 PM, MH zvi.schrei...@gmail.com wrote: Hi there. I am trying to add a custom domain from the GAE Application Settings. It redirects me to You have requested that the ' (my app id) ' service be added to your domain https://www.google.com/a/cpanel/(my google apps domain)/ AddAppEngineService?appId=(my gae app id)pli=1 I accept the terms and press Activate this service. But the same page just loads again in an infinite loop. I tried with a different google apps domain and it worked! Something about my specific google apps domain is confusing Google and I can't Activate this service Any ideas please? -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Serializing crypto classes
On Tue, Nov 15, 2011 at 10:30 PM, Gerald Tan woefulwab...@gmail.com wrote: Only Serializable objects can be used as properties, and Cipher isn't serializable. The solution is to store the algorithm as a String, then reinstantiating the Cipher from the algorithm string after retrieving it from the datastore. This is almost right - it's not about being Serializable. GAE only lets you store certain basic types (Long, String, Text, Blob, GeoPt, and about a dozen others including collections of these types). They aren't stored with java serialization; I believe the native structure is protobuf but this is opaque outside of Google. Objectify extends this list somewhat by automatically converting some types into ones the datastore recognizes (for example, String[] into ListString, byte[] into Blob, fields with @Serialized annotation into Blob, etc). But there are practical limits. We didn't go through every single JDK class figuring out how to store it in appengine. Cipher, IvParameterSpec, SecretKeySpec are not storeable types. Cipher doesn't even make sense to store - you probably just want to store the algorithm with which the Cipher was created. IvParameterSpec and SecretKeySpec are probably best stored as byte[] (which Ofy will convert to Blob) or, if you prefer, can be Base64 or hex encoded into a String. There are situations when you will want to build Objectify Converters so that fields can have normal types; for example, if you use Joda Time or Joda Money classes (Objectify includes converters for these, but they aren't enabled by default). I won't tell you that it's wrong to use IvParameterSpec and SecretKeySpec fields, but for the most part you should be cautious of using exotic types in your data model - especially when they are interfaces without a concrete type definition. I wouldn't do it. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Templating Frameworks
I've used many different Java templating systems over the years and been terribly disappointed with all of them... until recently. I started using Cambridge Templates: http://code.google.com/p/cambridge/ Here's a good intro document: http://blog.erdincyilmazel.com/ The docs are a little rough and it doesn't have a huge community behind it, but the product is solid and the code is clean. The author is extremely responsive to bugfixes and enhancements. The best thing about Cambridge is the syntax works *very* well with HTML and HTML editors. The template inheritance mechanism is a very elegant way to apply common elements to multiple pages - this feature alone is worth gold. Performance is good. I recommend it. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Number of writes per second limitation
On Fri, Nov 4, 2011 at 3:56 PM, Ikai Lan (Google) ika...@google.com wrote: If all 4000 entites are in a single entity group, in theory you can do this because it counts as a single transactional write. There's a maximum RPC size of 11mb (implementation detail) so if you trip this, you're in some trouble - the RPC size include not only the size of the entity but also the size of all the indexes. One question: If I write to 5 entities in a single entity group within one transaction, do these get batched up by the client library into a single RPC on commit or will this produce 5 RPCs plus one more for the commit? I'm curious to know what this does to the critical section, which could get large just from RPC latency. Thanks, Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: many different Properties or Entity for name/value pairs
FWIW you can declare a field @Embedded MapString, anything in
Objectify and this will create a series of native properties in the
low level entity. Ie:
class MyEntity {
@Id Long id;
@Embedded MapString, Long strings;
}
If the map contained { prop1 : 12 } then you could filter by:
ofy.query(MyEntity.class).filter(strings.prop1, 12)
Of course the value could be a SetString instead of Long.
But yeah, this will make the datastore viewer difficult to read if you
have wildly different properties for each entity. Nothing you can do
about that if you don't want to make the collection an opaque
serialized blob.
Jeff
On Fri, Nov 4, 2011 at 1:44 AM, Matthew Jaggard matt...@jaggard.org.uk wrote:
From previous threads, it seems that the datastore prefers few large
entities to more smaller ones - this is also reflected in pricing, you
pay per operation as well as per byte. Storing a Map of Name-Value is
exactly what the datastore is made for.
I did talk to Jeff at Objectify about this (because I'm using
Objectify for all my other entities) and he said that although this
use case is probably valid and I'm (clearly now) not the only one
doing it, making Objectify handle this case would complicate it too
much. I think I agree with this, especially since the datastore
handles properties like this so nicely.
In addition, I can now make use of Objectify's CachingDatastoreService
which means I don't have to worry about Memcache :-)
One thought - can you search based on an item in a collection? If not
and you do need to search based on these values, you might need to be
more creative about how to split them.
MyName-MyValue1
and
MyName-MyValue2
which might become...
MyName-Collection([MyValue1, MyValue2])
or perhaps...
MyName.1-MyValue1
and
MyName.2-MyValue2
if MyName.1 will never be a property name itself, and you can deal
with parsing this.
Thanks,
Mat.
On 4 November 2011 07:53, Mister Schtief lisc...@gmail.com wrote:
hi gerald,
thx for your answer but this is the ugliest solution ;) why serializing all
pairs and storing them in one.property it will never be searchable...
using one entity property for every map.entry or using one entity of a key
value pair type, thats the question ;)
schtief
Am 04.11.2011 07:10 schrieb Gerald Tan woefulwab...@gmail.com:
You can serialize a MapString,String property into a byte array to be
stored in the entity
The easiest way to do this would be to use Objectify with the @Serialized
annotation
http://code.google.com/p/objectify-appengine/wiki/IntroductionToObjectify#@Embedded
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine-java/-/_tZZPKf5r6cJ.
To post to this group, send email to
google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] HRD - Eventual consistency
There's this: https://groups.google.com/d/msg/google-appengine/NdUAY0crVjg/3fJX3Gn3cOYJ Some thoughts: * I don't think there's really anything to save you from eventuality except careful consideration during the design of your data model. Pick your entity groups carefully and use XG transactions sparingly to move data across when you need atomicity. * A lot of the time eventuality is not a big deal. On the other hand, if you're building an accounting app, it probably is a big deal. * There's a difference between what your users experience and what your tests experience. Say you have a test for add this product to the catalog and then check to see if it's in the catalog. In the real world it doesn't matter if your users see the new product right away. But your tests demand it immediately. The problem is with the test. I've fixed the tests by doing this in my fixture setup: newLocalDatastoreServiceTestConfig().setDefaultHighRepJobPolicyUnappliedJobPercentage(0.1f) Unfortunately a good argument can be made that this diminishes the value of the tests. Really some tests should be run with eventuality and others without. Anyone know if setDefaultHighRepJobPolicyUnappliedJobPercentage() can be called repeatedly after setup to switch back and forth? Jeff On Mon, Oct 31, 2011 at 6:31 AM, Aswath Satrasala aswath.satras...@gmail.com wrote: Hi, I am trying HRD on local dev server. I have bunch of webdriver tests for different parts of my application that create entities and then doing a list of all the entities. My tests are failing in several places. It is due to the eventual consistency of the HRD. The tests run fine on M/S and on both production/local environment. I have 20-30 KINDS, and entities are created for 1-or more KINDs depending on the business logic, all in a transaction. Immediately, various reports are done. They fail. My question: What pattern needs to be followed? My application has grown very large during last 1.5 years or so. Doing re-factoring at every place in the application for eventual consistency will be a very large effort for a small team. I don't even know what extra work needs to be put to get this fixed. Am I better of sticking with M/S. So far, I have managed and designed entity relationships to work within one entity group. Are there any patterns coming out from the Objectify team to handle Eventual consistency issue of the HRD? -Aswath www.AccountingGuru.in -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Optimizations with Datastore + JDO
You shouldn't see any significant performance difference between the various java datastore APIs during normal operation. The JDO's classpath scanning may have a significant startup cost, but during execution the costs of RPCs and protocol buffer manipulation tend to dominate over any inefficiencies in object mapping. On the other hand, third-party APIs offer shorter learning curves, better transparency, and easier access to native datastore features. And of course that startup hit can be a significant issue. Jeff On Fri, Oct 28, 2011 at 10:35 AM, Sheado chad...@gmail.com wrote: Thanks Mat, I'll check out Objectify. If any Googlers are reading this, it would be nice to add a section to the docs clearly describing the performance differences between all of the apis (low level api vs jdo vs jpa vs etc.) and maybe even between all of the different languages. As it stands, with the current performance of my JDO solution the costs of using appengine (due to high api cpu usage) would be prohibitively high for me - it would have been nice to be able to guess that right from the start. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/UPtK58TwQEMJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Does JDO auto remove indexes that now have unindexed properties?
Wild guess: If you load an entity in JDO and set a property to be exactly what it was before, this won't actually flag the entity as dirty. It won't be written when you commit the session. I don't know how you would force a write using JDO. Jeff On Thu, Oct 13, 2011 at 6:23 AM, markabrucey markabru...@gmail.com wrote: Basically I have gone through my datastore persist-able classes and have annotated properties that I never need to query on like this: @Extension(vendorName=datanucleus, key=gae.unindexed, value=true) private Long random; I needed to test whether or not JDO removed the index when I changed the POJO by annotating a property as unindexed. So I created a simple test to read in an existing entity, updated it (with properties holding the exact same values as before), then finally queried on it using the now unindexed property random. I woud have expected for it to not appear again once updated (because the property is now unindexed), which would have suggested that JDO was handling the index deletion correctly, but it appears not. Are there any settings that I may need to change in order to allow JDO to delete the now un-needed single property indexes? Is there a way I can remove / vacuum these single property indexes (that I don't need anymore) myself? Thanks, Mark -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/whSOGsiNGrUJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] with the 1.5.5 sdk, the local_db.bin file format has changed
The local dev database frequently gets invalidated at SDK upgrades. It should not be considered a reliable store. No data is trashed in the production database. Jeff On Fri, Oct 21, 2011 at 10:28 AM, Tommy Fannon tfan...@gmail.com wrote: It seems to need to be deleted and re-created. I want to make sure that when I deploy my app to production that existing data is not going to be trashed. I'm sure I would have read about this already, but never hurts to be sure. Thank you, tommy -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/Jv_UYPMlMIUJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Using overloaded put, get, and delete methods for transactions?
Implicit transactions are a separate issue (and a terrible idea).
When the low-level api starts a transaction, it stores this txn as a thread
local. The get()/put()/etc methods that don't have a transaction are
effectively this logic:
Entity get(Key key) {
if (there is a thread local txn)
return this.get(theThreadLocalTxn, key);
else
return this.get(null, key);
}
Passing in null is the only way to guarantee that you're making a call
without a transaction.
This thread-local transaction stuff helps support the JDO/JPA plugin but
it's a bit of a mess if you use the low-level API. My advice is not to use
it at all - avoid the methods that don't have a txn parameter and pass null
in when you don't have/want a transaction. Hide this behind a facade.
Jeff
On Fri, Oct 21, 2011 at 5:23 AM, Mos mosa...@googlemail.com wrote:
I'm a bit confused:
Following some examples and looking at the API of the DatastoreService
it seems to be required to use the overloaded methods whenever
transactional behavior is required.
In other words: The current transaction needs to be specified on put() und
get() methods like this:
Transaction txn = datastore.beginTransaction();
//
Entity employee = datastore.get(txn, employeeKey);
employee.setProperty(vacationDays, 10);
datastore.put(txn, employee);
txn.commit();
But when looking at the official Google docs, there seems to be no need for
specifying the
transaction explicitly:
http://code.google.com/intl/de-DE/appengine/docs/java/datastore/transactions.html
?
Can someone explain this? Is this an issue with implicit transaction
behavior? How does it work?
Thanks a lot
Mos
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to
google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: JDO and XG transactions - Performance
Ugh. Is this a correct further paraphrase? * You wanted users to get exceptions which would force them to go to the manual to figure out why they are getting exceptions. There are less intrusive ways to force users into reading the manual! I really hate software like this; it's a bozo bit that defaults to true. Yeah, transactions are weird on GAE and you pretty much have to read the manual anyways. This additional hurdle doesn't bring any value to the code. If there truly is no behavioral change to single-group transactions when setting XG true, I will hardwire it true and completely hide the broken bit from Objectify users. If there's a significant reason to leave XG false for single-group transactions, please let me know. Jeff On Mon, Oct 17, 2011 at 11:05 PM, Max Ross (Google) max.r...@gmail.com wrote: We debated this quite a bit internally. To paraphrase the argument that carried the day: XG transactions are awesome but they are not true global transactions. You're limited to 5 entity groups, you're more likely to see partially applied transactions in global query results, you can get a concurrency exception from a read, and there is a performance impact (when you've got more than 1 entity group enlisted). In short, there's some surprising/subtle stuff that we really want developers to understand before they start using this feature. If we didn't require opt-in it would be too easy for our new developers (particularly those coming from the RDBMS world where global transactions are the norm) to miss these important considerations. We want it to be super easy to use the Datastore, but creating a false sense of simplicity can do more harm than good in the long run. Max -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/mq8YLNx0kaoJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: JDO and XG transactions - Performance
On Tue, Oct 18, 2011 at 9:07 AM, David Gay (Google) d...@google.com wrote: One other consideration: XG transactions do not work on master/slave. While the default could be different depending on whether HRD is used, that definitely has drawbacks. How can I detect if running on HRD vs M/S? I don't see anything on SystemProperty. I could try starting an XG transaction and catching the exception, but that would add an extra RPC to every Objectify app's startup time :-( The XG issue seems pretty easily solved by documentation: 1) If you're running on M/S, you can only use one entity group in a txn. 2) If you're running on HRD, you can use up to 5 entity groups in a txn. 2a) If you use multiple entity groups in a txn, this is implemented with 2pc and may produce surprising effects like ConcurrentModificationException on read. This doesn't really matter in practice if you use the RepeatInTransaction pattern. Put it this way: Non-ancestor queries work funky on the HRD too. Yet I don't need to set a special flag just to issue non-ancestor queries. Understanding eventual consistency and exotic transaction behavior is just part of working with GAE. Magic flags that enable otherwise expected behavior don't really help. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: JDO and XG transactions - Performance
Yeah, unfortunately it seems permanently stuck at MASTER_SLAVE in development mode. However, between this method for production and manually trying to commit a transaction in development mode it looks like I have a complete (but hacky) solution. Thanks Alfred! Jeff On Tue, Oct 18, 2011 at 2:19 PM, Alfred Fuller arfuller+appeng...@google.com wrote: This will tell you what type of datastore you are running against, but I don't think it will help you in the development env: http://code.google.com/p/googleappengine/source/browse/trunk/java/src/main/com/google/appengine/api/datastore/DatastoreService.java#388 On Tue, Oct 18, 2011 at 2:02 PM, Jeff Schnitzer j...@infohazard.org wrote: Dunno about production, but the developer mode takes issue with it. Even a simple single-entity transaction commit fails in M/S mode with the test harness - ie, without calling this method: LocalDatastoreServiceTestConfig.setDefaultHighRepJobPolicyUnappliedJobPercentage(100) commit() always produces this: java.lang.IllegalArgumentException: transactions on multiple entity groups only allowed in High Replication applications :-( Jeff On Tue, Oct 18, 2011 at 1:44 PM, Alfred Fuller arfuller+appeng...@google.com wrote: I believe (you should test this) that it won't do any harm to set XG=true in master slave (it will just ignore you). On Tue, Oct 18, 2011 at 1:28 PM, Jeff Schnitzer j...@infohazard.org wrote: On Tue, Oct 18, 2011 at 9:07 AM, David Gay (Google) d...@google.com wrote: One other consideration: XG transactions do not work on master/slave. While the default could be different depending on whether HRD is used, that definitely has drawbacks. How can I detect if running on HRD vs M/S? I don't see anything on SystemProperty. I could try starting an XG transaction and catching the exception, but that would add an extra RPC to every Objectify app's startup time :-( The XG issue seems pretty easily solved by documentation: 1) If you're running on M/S, you can only use one entity group in a txn. 2) If you're running on HRD, you can use up to 5 entity groups in a txn. 2a) If you use multiple entity groups in a txn, this is implemented with 2pc and may produce surprising effects like ConcurrentModificationException on read. This doesn't really matter in practice if you use the RepeatInTransaction pattern. Put it this way: Non-ancestor queries work funky on the HRD too. Yet I don't need to set a special flag just to issue non-ancestor queries. Understanding eventual consistency and exotic transaction behavior is just part of working with GAE. Magic flags that enable otherwise expected behavior don't really help. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: JDO and XG transactions - Performance
On Mon, Oct 17, 2011 at 9:38 AM, David Gay (Google) d...@google.com wrote: Just enable it for the whole app: an XG transaction that touches only a single entity group has exactly the same performance as a non-XG transaction. In that case, why is this a configurable setting? Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Facebook Login integration with java appengine
Facebook recently (as in, a couple weeks ago) made it possible to associate multiple domains with your application. Add my-appid.appspot.com as well as aggenda.com. Jeff On Thu, Oct 13, 2011 at 3:09 AM, marco marco.m.al...@gmail.com wrote: Hi guys. (i'm newbie) I'm trying to make an existing GWT+AppEngineJava (GWTGAE/J) app into a Facebook Canvas app (iframe). The GWTGAE/J app already supports login with Google. The problem: The Site Url on facebook is www.aggenda.com -- i bought the domain and associated it the App Engine via Google Accounts In order to make the GWTGAE/J app into a FB Canvas app, I have to provide an https url to FB App Engine does not provide https for www.aggenda.com, it only provides https to my-app-id.appspot.com -- this means i have to provide my-app-id.appspot.com/facebook/canvas/ (where a servlet handles FB https posts) If the user clicking on the canvas page inside facebook did not already authorize the app, I request an authorization following the normal facebook login flow that I use for when the user logs in the GWTGAE/J App via facebook Facebook says that the redirect_uri is not allowed by the application -- I thinks it's because it comes from my-app-id-appspot.com instead of www.aggenda.com If I try the request_uri to www.aggenda.com, it does not work because www.aggenda.com can not respond to https requests Have you had this problem? Any hints? Thanks! -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/sisf8zcwCyYJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Facebook Login integration with java appengine
It's in the Basic Info section in the upper part of the page (which you scrolled past). Look for App Domain - you can put multiple entries there. Jeff On Thu, Oct 13, 2011 at 4:01 AM, marco marco.m.al...@gmail.com wrote: Hi Jeff. Thanks for the quick reply. I'm looking in Apps Aggenda Basic (inside Facebook) and can not find where i can add additional URL. Check the attached screenshot. Thanks again for your help. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/hIcn2myHn-YJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Facebook Login integration with java appengine
Are you typing in a field called App Domain? It lets you type in multiple domains, one next to the other. Jeff On Thu, Oct 13, 2011 at 4:39 PM, marco marco.m.al...@gmail.com wrote: Hi Jeff. I've tried out what you suggest but it gives an error: my-app-id.appspot.com must be derived from your Site URL. The site URL is www.aggenda.com/facebook?param1=10. I must be doing something wrong... -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/DbNg07QR-mEJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] GeoPt vs. two floats
It really isn't a big deal. Store two floats if you want. The reasons I use GeoPt a lot: * GeoPt shows up nicely in the datastore viewer * You can store arrays of GeoPt * The storage cost should be slightly less than two floats because there is metadata only for one field. * Maybe some day in the future we will be able to do spatial indexing with GeoPt. I'll be honest though, the primary reason I use GeoPt is because it's more convenient to pass around one object than two floats. Jeff On Tue, Oct 4, 2011 at 7:30 AM, Eliot Stock 1...@eliotstock.com wrote: Hi there, What's the advantage, if any of using GeoPt (http://code.google.com/appengine/docs/java/datastore/entities.html#Properties_and_Value_Types) over using two float fields? I've found the disadvantage is that Spring form beans are difficult to work with when you have two hidden fields on the UI (lat and long) but one GeoPt field on the entity. Cheers, Eliot -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/c8oRwyDMIIgJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Automatically increment version number
Keep in mind you only get 10 versions, so you'll have to delete old ones as you add new ones. I don't see any way to automate that. Jeff On Tue, Oct 4, 2011 at 2:09 AM, Manilal libreg...@gmail.com wrote: I'm using the commandline version of appcfg command to deploy Java application. I would like to create a new version on each deployment. I know that the version number can be changed in appengine-web.xml, but I would like to automate the process and keep the old versions intact. thanks Manilal -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/rUDqqtJwkmIJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: how to bring a huge (http error 413) amount of data to the server?
That doesn't sound right. You should have no trouble POSTing 20k to GAE. I wouldn't be surprised if GETs choked though. Jeff On Tue, Oct 4, 2011 at 9:47 AM, Gerald Tan woefulwab...@gmail.com wrote: You'll have to use the Blobstore API and allow the user to upload the xml into the blobstore -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/T1c7gzsIj84J. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: GAE - Vote counting system
On Mon, Oct 3, 2011 at 9:24 AM, Mat Jaggard matjagg...@gmail.com wrote: Jeff - I'm a bit confused. I thought that the whole idea of the datastore was that you could read or write as much as you want, as fast as you want as long as they are not related? So one datastore write per vote (and being written to different entity groups) should be fine? I thought that the system just split tablets if they were being accessed too much - so as long as the traffic didn't suddenly increase, there'd be no scalability issues apart from cost. apart from cost he says :-) The OP posited millions of users and millions of things to vote for. Each million votes will cost you (at minimum) $1.70 for one write + one read, but it'll probably be more depending on how many page views you have and what caching strategy you have. Still, maybe this is no big deal. The bigger problem though is that vote traffic is likely to be focused on a handful of items. Popular things might get thousands of votes per second, unpopular things won't be voted for at all. It's hard to come up with a sharding strategy that works well for this - you probably don't want 1k shards for everything, storage costs go up and expense/latency of calculating totals goes up. I have to deal with a similar problem myself right now (with the added constraint that I need an instantaneously precise count). I'm considering a system that automatically tracks latency and increases the shard count when it crosses a threshold. It's not a pretty problem to solve. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: GAE - Vote counting system
Assuming the goal is massive write throughput, you don't want to do 1 write per vote. You need to aggregate writes into a batch - you can do that with pull queues, but then you're limited to the maximum throughput of a pull queue. And the biggest batch size is 1,000 which might actually be votes for 1,000 different things, which means you're back to 1-vote-1-write. Peter, you can certainly build a system whereby all vote totals are tracked in RAM in a backend but now you're putting higher memory requirements on the backend and life gets more complicated when you deal with sharding. Depending on how exact you need the counts to be, you can always use increment() on memcache in addition to incrementing the backend. The only catch is that bootstrapping the initial memcache value will be a little tricky - you'll need to use CAS and query the backends for any existing deltas. Or just not care if it's off by a few. Jeff On Thu, Sep 29, 2011 at 5:56 AM, Mat Jaggard matjagg...@gmail.com wrote: Will that end up being cheaper than the following? Put a new entity in the datastore for each vote - Kind: Vote, ID: Auto generated, Vote for: Item S Have a task queue query all the votes and delete them then write the count of votes to a global object. Cost = 1 datastore read + 1 datastore write + some fairly minor task processing per vote. On Sep 29, 1:47 pm, Peter Dev dev133...@gmail.com wrote: Price: - with backends lets say 3 B2 machines = 350USD/Month - UrlFetch Data Sent/Received 0,15USD/GB Limit: - URL Fetch Daily Limit 46,000,000 calls this can be a problem...but I see it is possible to request an increase Write data parallel in DB: Task Queue with rate every 30second could be a solution (check timestamps in cache and write in DB) RESET counters = empty cache in Backends reset counter of object in DB Backends cache = HashMap with shared counter values or counter values without sharding (just increment value in java hashmap is fast enough) With backends we don’t need sharding I thinkwhat do you think? Thx. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: GAE - Vote counting system
It's hard to say here if we're talking about the same thing, but here's how I would do it: * Updates go through to the backend, which stores write deltas in ram (not the total). * Reads read-through memcache into the datastore. * The backend writes deltas to the datastore in batch, updating memcache, then purging the delta from memory. * The backends can write once every 5 mins or 30s or however long you're comfortable having read data be stale. More freshness == more datastore expense. It's a simple dial. This system is not limited by backend RAM since each backend stores only deltas - you can probably run the smallest size. It won't be limited by read volume, which will come almost entirely from memcache. It will be limited by max request throughput on the backend. Given the update is practically a no-op, how many QPS can a single backend serve? That's your limit (times the number of backends running). I think you'd be hard-pressed to find a better solution to this problem on GAE. It does require that reads be stale to with a controlled bound, though. Jeff On Thu, Sep 29, 2011 at 5:47 AM, Peter Dev dev133...@gmail.com wrote: Price: - with backends lets say 3 B2 machines = 350USD/Month - UrlFetch Data Sent/Received 0,15USD/GB Limit: - URL Fetch Daily Limit 46,000,000 calls this can be a problem...but I see it is possible to request an increase Write data parallel in DB: Task Queue with rate every 30second could be a solution (check timestamps in cache and write in DB) RESET counters = empty cache in Backends reset counter of object in DB Backends cache = HashMap with shared counter values or counter values without sharding (just increment value in java hashmap is fast enough) With backends we don’t need sharding I thinkwhat do you think? Thx. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Transactions on entities in different entity groups ?
A month or so ago Google posted a request for trusted testers for this feature. It helps to read the mailing lists :-) Jeff On Thu, Sep 29, 2011 at 2:05 AM, Cyrille Vincey cvin...@qunb.com wrote: SERIOUSLY??? (why don't you give any visibility on this key feature, like you did with FTS?) On Sep 28, 10:52 pm, Ikai Lan (Google) ika...@google.com wrote: You currently can't, but we plan on rolling out a feature that will do this in the next few releases. It's currently in trusted tester at the moment. -- Ikai Lan Developer Programs Engineer, Google App Engine plus.ikailan.com | twitter.com/ikai On Mon, Sep 26, 2011 at 10:59 PM, Santosh kumar kopp@gmail.com wrote: Hi, I want to know how to create/update the entities which are in different entity groups ? for ex: I want to update the *apple* and *carrot* entities in a transaction but those parents are not same. (Here i am in such a situation) Thanks in advance :) *S*antosh *K*umar *K* http://www.accountingguru.in/** -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] GAE - Vote counting system
Have you looked into using pull queues combined with sharded counters? If you lease 1,000 tasks at a time you can probably condense writes considerably. There are several different sharding strategies you could take, even just one master set of shards each of which covers all items. 100 shards * 1,000 tasks should give you a minimum write rate limit of 100,000/sec, magnified by the amount of overlap (ie if those 1,000 tasks only updated 100 counts, your minimum write rate limit is 1 million/sec). I haven't tried this. It presumes you can write 1k entities to a single entity group in a single transaction - maybe a googler can chime in. It would probably be expensive to store. Jeff On Mon, Sep 26, 2011 at 12:41 AM, Peter Dev dev133...@gmail.com wrote: We are developing an application, where users can vote for many objects. (for example, voting the best music video of the week) - This means, we have millions of possible objects to vote for, and millions of users To our best knowledge, after taking in consideration different options, the best (or the only) voting system is: Memcache+Bulk DB write in DB - If number of objects in Memcache will achieve a specified limit (for example 3000) then write in DB. The writing speed into DB is about 100/sec. This also means, if we would set the above mentioned 3000 objects, the writing would last for 30 sec... The problem; during save in DB, voting must be blocked. In other words, if from many millions of objects, there are 3000 achieved voted objects, we need to write it into DB and it can happen too many times, and blocking the whole voting mechanism. If we do not block the voting whilst writing in DB, the result could be wrong number of votes from cache (see Workflow: step 3). Workflow: 1. vote received 2. find object in memcache 3. if not found in memcache get from DB and put it in 4. increment the number of votes of the object in memcache 5. check object number in memcache 6. if necessary, save in DB and empty memcache ... -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: GAE - Vote counting system
Yeah, messy. I'd use a backend for this. Possibly a set of backends if you need to shard the data for write volume. I'd use Memcache only to cache the count reads. The basic entity is just an id and a count. An increment request goes to a backend, which simply tracks the change. A batch process goes through and writes any changed counts to both datastore and memcache (as an update; read increment write in a txn) every 5 minutes and clears the memory count. If write volume is too high for a single backend to handle, shard it by thingId % number of shards. You can change the shard count on the fly this way. All reads should be read from the memcache, read-through to the datastore as necessary. This should be able to handle any volume you want. If a backend crashes you'll lose its accumulated counts but I presume that's not a big deal. Jeff On Tue, Sep 27, 2011 at 10:38 AM, Peter Dev dev133...@gmail.com wrote: Sorry, 100 000 000 counters On Sep 27, 4:53 pm, Peter Dev dev133...@gmail.com wrote: Shared counter is cool and I use it... but if you have millions of objects I cannot imagine how to manage them.1 000 000obj x 100 shards =10 000 000counters 1. How to reset them to 0 in specified periods? 2. How to set the shared sum for each object to show top 100 objects? 3. Too much DB API Calls (each vote makes write in DB) Any ideas...? Thx -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: The *real* cost of the billing changes
Amazon does something like this for its excess capacity: http://aws.amazon.com/ec2/spot-instances/ ...but it's just for excess capacity. I wouldn't want this kind of behavior for my core usage because it would make my bill highly unpredictable. And it does not incentivize Google to add more resources to the cluster. Jeff On Sat, Sep 10, 2011 at 10:39 AM, gk goran.kar...@googlemail.com wrote: Pricing should be as follows: Calculate a price of resources using the demand / availability of those resources in the cloud: - if Google has more of resource X than users demand, the price for that resource would go down and free quota up. - if users demand more of resource X than Google has, the price for that resource would go up and free quota down. Pricing model: GAE users would act as buyers and resources in the cloud as suppliers. - Suppliers (resources producers in the cloud) set the price of the resource above the current market price. - Supplier decreases the price until resource is sold - but a supplier nevers sells under the price of (production cost + Google profit margin). - Google continously buys certain amounts of the resource and declares it to be a shared good - that is the free quota for each App Engine. Users act as buyers and buy using a model of their own choice. - Some would never buy any resource - but want to use free quota. - Some would spend $ in a month / week / day / hour / minute / second on a specific resource. - Some would spend $ in a month / week / day / hour / minute / second on any resource. - Some would spend $ in a month / week / day / hour / minute / second on all resources. - Some would buy whenever the price is under $ - Some would buy certain amounts in a given time period. - Some would always buy the resource. In this way the GAE users can buy according to their need (production cost, if any + profit margin, if any). Is this not the logical solution for the pricing problem? Mix this with a Market for App Engine Apps and BOOM :-) -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Objectify-Appengine 2.1 released, supports Partial Indexes
Specifically, it's the difference between calling Entity.setProperty() or Entity.setUnindexedProperty() on the low level API. Objectify interacts with the datastore through the low level API, so there is always underlying support for features. Not necessarily in a convenient form, however. Jeff On Fri, Aug 12, 2011 at 8:40 AM, Scott Hernandez scotthernan...@gmail.com wrote: It is supported in appengine. Every time you set a property in an Entity (appengine Entity) you can specify if that property is indexed or not. The partial indexing support just lets you control that on a per instance basis instead of having the field always indexed or not. On Fri, Aug 12, 2011 at 11:14 AM, Tobias tgrundt...@gmail.com wrote: Hi There, Can you tell me a little about how partial indexes are implemented in objectify? Is it supported in the underlying API or is it solely implemented in objectify? Kind regards Tobias. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/AcFUXARv3sEJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Appengine Rest Services
I'm pretty happy with Resteasy: http://www.jboss.org/resteasy There was a minor trick to getting Guice integrated nicely (see my recent post on the resteasy-developers list). Other than that it works great, and you can use it to render HTML pages with a little extension: http://code.google.com/p/htmleasy/ It all works normally on appengine, no issues. Jeff On Wed, Aug 10, 2011 at 1:37 PM, Bruno Sandivilli bruno.sandivi...@gmail.com wrote: Hi, i'm doing some ressearch, because i want to do some rest services. I can do this by uusing Spring MVC + Jackson But i'm thinking in use some kind of stuff that integrates better with appengine. Im thinking in use : Guice + Jersey . But, i could not find some good examples on the web. I found this : http://blog.iparissa.com/jersey-guice-on-google-app-engine-java/ but i cant find the jersey-guice(wich contains GuiceContainer). Is this the best way to build a rest service in Java + Appengine? May i use Spring or Guice to acomplish this? Thanks! -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Joins!
On Fri, Aug 5, 2011 at 11:45 AM, William Levesque billleves...@gmail.com wrote: But is Google's position that all data should be denormalized? I don't think anyone would say that. I wrote up my thoughts around this subject here: http://blog.similarity.com/post/7541938593/how-to-build-an-online-dating-site-nosql-edition The upshot is that we've been conditioned by SQL theorists to believe that there is a proper way of modeling data; that the data itself defines the schema and the magic of the RDBMS behind the curtain makes it fast. Unfortunately, this is a lie. It worked to a point but the traffic demands of a mass consumer application have vastly outstripped the RDBMS. You're back to figuring out how to optimize your schema for your particular query profile. So the answer is not denormalize everything, it's denormalize the right things. And the right things will vary from application to application. You just have to build up a correct mental model of how the datastore performs and then design your application accordingly. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Contains query Low Level datastore
It's actually much simpler than you realize. Assuming that your ListLong is an indexed property named 'friends', you just need to filter entities for 'friends' equal to your id. This is how list indexes work - if you filter on the property, the entity matches if any value in the list matches the filter. Jeff On Tue, Jul 19, 2011 at 5:28 PM, keyvez key...@gmail.com wrote: Thanks for your quick response. After looking into the IN operator, it doesn't seem to be what I need. My table looks like this. User table: id friends 15 21, 4 31 41, 2 51 I would like to find out who are user 1's friends so the query would be something like this SELECT * FROM User WHERE friends.contains(1) the IN version looks like this SELECT * FROM User WHERE friends IN (1) which would return id=3 as a result but it is wrong, I am expecting to see id=2,3,4 be returned in the result set. Because they are all friends with one. Also I HAVE to do this through the low level datastore API to utilize the cursor feature. Thanks, Gaurav -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/XV-oATd4fjUJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: appspot.com without www domain names - but not a general word to remember and refer to -
Shrug. Given that any domain chosen cannot have google as part of it (G doesn't want to convey any kind of public endorsement of our applications), appspot seems about as good as any. Sure cloud.com would be nice but I'm pretty sure the people already using cloud.com have other ideas on the matter. Just shell out the $10/yr for your own domain. You can host any number of apps under your own domain as subdomains. Jeff On Sun, Jul 17, 2011 at 9:38 AM, dreamer venugopal.vasire...@gmail.comwrote: I agree. For existing businesses possibly they carry their domain. (app engine for business). But App Engine gives platform gives opportunity for creating new businesses. Having www domain is good, but having easy to remember free domain also does not hurt. On Jul 17, 9:17 am, jMotta jayrmo...@gmail.com wrote: I think you got the whole internet thing wrong. When you have a business, doesn't matter for your users if it's written in Java or ASM, they just care if it works or won't. So, if it will be appengine or anything else it's much more a choice from you than from anyone else. In other words, your user doesn't care if it's appspot or yourdomain.com ... but probably he will remeber yourdomain.com easily. In the future if you choose to move from appengine to something else, you'll be stuck on it. Instead request something from Google, a company that doesn't even know that you exist, buy a domain, do something useful, work on your business, that's it. *Jayr Motta* Software Developer * * I'm on BlackBeltFactory.com http://www.blackbeltfactory.com/ui#!User/jmotta/ref=jmotta ! On Sun, Jul 17, 2011 at 1:10 PM, dreamer venugopal.vasire...@gmail.com wrote: =Why don't you buy one? One extra requirement, takes long way in larger scope. What I mean is, for sure if app engine widely becomes popular, appspot.com may become normal word for non tech folks. but there are already popular words available like cloud , sky, which straight away can be leveraged for their popularity. On Jul 17, 8:20 am, jMotta jayrmo...@gmail.com wrote: You must be kidding, right? They're already giving you a free domain name with all possible services without charge anything and you request a domain easy to remember. Why don't you buy one? You've said that you could pay for this service, go on! *Jayr Motta* Software Developer * * I'm on BlackBeltFactory.com http://www.blackbeltfactory.com/ui#!User/jmotta/ref=jmotta ! On Sun, Jul 17, 2011 at 12:11 PM, dreamer venugopal.vasire...@gmail.com wrote: Hi, My apps url is http://schoolk12.appspot.com/;. I had hard time remembering this URL, for sure I can not expect general user to remember this. So I have to have www domain name. I din't have any intention of buying domain name any time soon.I am hoping App engine+GWT+Datastore - really takes off, And appspot.com domain becomes so popular, and we don't have to buy www - domain names for most of the hosted apps if not all. appspot is not a general term to remember, if this were some thing like cloud.com or googlecloud.com, it would have been easier to remember for general user to remember. We can pay these www domain costs to google, for paid service. The way domain names are going, I would be cleaner, if google brands some thing like appspot.com. So I request google come up with very generic domain name. Some of the suggestions. cloud.com globe.com Then my URL looks like this http://schoolk12.cloud.com/; http://schoolk12.globe.com/; For general user, I feel these URLS may be easy to remember and refer to. Please share your opinion. -Venu http://schoolk12.appspot.com/ -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send
Re: [appengine-java] Απ: ANN: BatchFB 2.1, a Java Facebook library with automatic batching
On Thu, Jul 14, 2011 at 10:59 PM, George Moschovitis george.moschovi...@gmail.com wrote: Thanks for the release, but why don't you use Future? The Future? interface is fundamentally defective because it uses checked exceptions. Checked exceptions are one of the handful of gross design flaws of the Java language. It's notable that not a single modern language in the post-Java era - not even the typed languages (groovy, scala, clojure, ceylon) - include the checked exception. It was one of those theoretically good ideas that just turned out to be wrong. But, like the JCP, it's one of those ugly warts we just can't seem to get rid of. If you use Future?, you're stuck with catching InterruptedException and ExecutionException *everywhere* in your code. Or you do what Google does and use FutureHelper everywhere. This is bullshit. I'm not going to do it in my code, and I'm not going to expect anyone else to do it. Objectify's async api is built around the same principle. There is a Request? interface that works almost exactly like Future? but throws runtime exceptions instead of the crappy checked exceptions. I wish I could use the same interface for BatchFB but I can't without making one project rely on the other. I blame the asshats at Sun (now Oracle) for dogmatically sticking to this anachronism and refusing to let the language evolve; this is one of the reasons the web development community has abandoned Java and moved on to Ruby and Python. It pisses me off. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Best way to work with a CellTable and many-to-one relationship with objectify?
The public/private/protected/package status of java fields is 100% irrelevant from a security perspective. It's just there to help keep your code clean. The data is still being passed across the wire in a simple, easily-decoded protocol that any sniffer can translate. If you're passing sensitive data across the internet, you *certainly* need to use SSL at the very least. Having getters/setters vs public fields is purely a stylistic concern in this context. Jeff On Tue, Jul 12, 2011 at 4:52 AM, Drew Spencer slugmand...@gmail.com wrote: Thanks for the great reply as always Jeff. In contrast to you, my app is going to be used only by employees of my company, so preferably I would like to have all data sent over the wire encrypted. We're talking about employees of my company accessing data about our clients - so it is very sensitive data and I would think therefore that whether I need to use a DTO or not, everything should be private. Is this correct, or is it ok to use public properties without getters and setters as long as I'm encrypting the data? Cheers, Drew -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/YQX1oC92g6AJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] ANN: BatchFB 2.1, a Java Facebook library with automatic batching
This is the first time I've announced this opensource project in this forum because until now there wasn't anything GAE-specific about it. You can still use BatchFB outside of GAE, but now BatchFB will use asynchronous fetching on App Engine to issue multiple batches to Facebook in parallel. This is *the* fastest way to pull down significant quantities of data from Facebook. http://batchfb.googlecode.com/ Here is a quick snippet of code: --- Batcher batcher = new FacebookBatcher(accessToken); LaterUser me = batcher.graph(me, User.class); LaterUser mark = batcher.graph(markzuckerberg, User.class); LaterListUser myFriends = batcher.query( SELECT uid, first_name, pic_square FROM user WHERE uid IN + (SELECT uid2 FROM friend WHERE uid1 = + myId + ), User.class); LaterUser bob = batcher.queryFirst(SELECT timezone FROM user WHERE uid = + bobsId, User.class); PagedLaterPost feed = batcher.paged(me/feed, Post.class); // No calls to Facebook have been made yet. The following get() will execute the // whole batch as a single Facebook call. String timezone = bob.get().timezone; --- You can stack up hundreds of Graph and FQL requests using the Future?-like interface; BatchFB will group them into batches and execute the batches in parallel on App Engine. Enjoy, Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Fast MVC Framework for GAE?
If you're just looking for an MVC framework, here are two that I wrote: Tagonist: http://www.tagonist.org/ This is about as simple as it can possibly get. Less than 500 lines of code, no classpath scanning, zero (as in nada, none, zip) effect on startup time. It just plays clever with JSP. For a lot of apps, this is enough. HTMLeasy: http://htmleasy.googlecode.com/ This is a layer on top of RESTeasy that lets you render HTML in a natural JAX-RS way. You can turn off resource scanning in RESTeasy to improve startup time, but it means you need to manually define all your resource classes in web.xml. The key to a fast startup is to avoid classpath scanning. Unfortunately this means you might have to do more work yourself. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Can we get lat/long data for given address ?
Google geocoding. Jeff On Wed, Jul 6, 2011 at 10:14 AM, oceandrive rams...@gmail.com wrote: Can anyone tell me if there is a way to get latitude/longitude values for a given address -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/lHQ8BAyea0oJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Changing datastore property type
On Wed, Jul 6, 2011 at 11:50 AM, jMotta jayrmo...@gmail.com wrote: Jamie, I don't know if get it right, but if it's part of the key it is not possible to change. It sounds like the id is being stored as a field/property of an entity - it's basically a foreign key. If this property is not part of the key, then you'll have to deal with it. Probably you're using some persistence framework such as JDO, JPA or Objectify. These are probably trying to bind the properties found in the datastore Entity object into somekind of DTO / POJO through reflection and then getting this class cast exception. Jamie, you didn't mention what API you are using to access the datastore. FWIW, Objectify will let you change the field type to a String and do the right thing out of the box. It's a little more tricky if you have an index on that field, but reprocessing the data (thus recreating the index) is pretty simple - just load and save all the entities. No need to go to the low-level api. Unfortunately JDO is more fragile - schema migration doesn't seem to have been planned into the API. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Why are app instances killed after 9000 requests
This is an interesting issue. Hopefully someone at Google has considered this when making changes to the scheduler. Instance death after 9k requests is a known behavior of GAE (well, known if you read these forums - I don't think there is any official documentation). As a wild guess on my part, it's probably related to loadbalancing hot instances across machines - as long as instances keep moving, any single machine is statistically unlikely to be stuck with multiple hot instances for long. Yeah, when a multithreaded appserver can hit the 9k limit in a few minutes, this approach probably needs to be rethought out. I would assume they're working on it. Jeff On Fri, Jul 1, 2011 at 6:03 AM, Juha K juha.kosk...@gmail.com wrote: Hi, I've been testing possible memory leak in my app using JDO. I created a small app with two servlets, the other servlet creating one small JDO entity and the other servlet creating an entity with Objectify. Running the test with JMeter I've realised the memory usage grows quite a bit when using JDO (same can be seen in profiler with dev server too). With Objectify the memory usage grows a little BUT it seems that GAE kills the instance as soon as it has got 9000 requests. I've run the (objectify) test using threaded and non-threaded mode. With non-threaded I'm sending about 3-4 requests per second which are being handled by single instance. It takes almost 40 minutes to reach 9000 requests and then instance memory usage is about 52-58mb and latency 22-27ms. And when 9000 requests are reached the instance is being killed. When I run the app threaded, I can send about 20 requests per second and it takes just over 7 minutes to reach that 9000 request limit (memory usage is usually at 53mb) and the instance is killed. With upcoming pricing model I pay for at least 15 minutes for each instance? Why does GAE kill my instance at exactly 9000 requests (no matter how soon the 9000 requests are reached and no matter what is the memory usage or latency)? I was closely watching my production app and the same happens also there. This is output of instances in the control panel for my test app just before reaching 9000 requests: QPS*Latency*RequestsErrorsAgeMemoryAvailability20.70026.8 ms897600:07:2151.8 MBytes[image: Dynamic Icon]Dynamic and this just seconds second later: *Instances *[image: help]http://code.google.com/appengine/docs/adminconsole/instances.html QPS*Latency*RequestsErrorsAgeMemoryAvailability0.0500.0 ms500:00:0140.3 MBytes[image: Dynamic Icon]Dynamic -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/EHmdh2ldi84J. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Big Entities vs Small Entities
I think John has the right idea... but since you probably don't need
dynamic form fields, I would probably do it like this:
class MyEntity {
@Id String uid;
@Embedded FormField field1;
@Embedded FormField field2;
...etc
}
(No doubt for Twig you just substitute @Embed for Objectify's @Embedded)
Jeff
On Fri, Jul 1, 2011 at 12:50 AM, John Patterson jdpatter...@gmail.com wrote:
Keep them all in one Entity so you only need to do a single datastore get()
but embed a List of FormField (or something) in the main entity. Both Twig
and Objectify support embedding collections of objects in a single Entity.
Entity size is less important than the number of datastore calls you must
make.
class MyEntity
{
@Embed private ListFormField fields;
}
class FormField
{
String value;
boolean modified;
}
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine-java/-/VgmHIl1m87QJ.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Best way to work with a CellTable and many-to-one relationship with objectify?
That's what I tend to do. Since I write consumer apps, passing my raw entities across the wire usually isn't an option in the first place - there's usually too much security-sensitive stuff, or at the very least data I would rather not disclose to a potential attacker. Also... using entities client-side tends to really thin them out, and you can easily fall into the anemic domain model antipattern. And when you work on the domain classes, it can break your RPC protocol (forcing users to reload their browser page, or worse if we're talking about web service type RPCs). Sometimes this is ok, sometimes it isn't. Just depends on your app. As a general rule, I wouldn't be so afraid of creating DTOs. Even if they're just a set of public fields - do they really need getters and setters? Jeff On Thu, Jun 30, 2011 at 2:12 AM, Drew Spencer slugmand...@gmail.com wrote: Thanks guys. So basically I should be grabbing the Supplies and Suppliers on the server side, combining the data I need from both into a SupplyDTO object, and passing that back to the client, rather than doing the processing on the client? Seems to make sense! Drew -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/B7fV3EzXtSEJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Best way to work with a CellTable and many-to-one relationship with objectify?
To be honest, whenever I start to have a less-than-perfect match between my entities and the content rendered in my client, I usually create DTOs. It's mildly annoying, but it results in a clean interface without a lot of extra payload (some of which might be security-sensitive) sent across the wire. Jeff On Tue, Jun 28, 2011 at 8:26 AM, Drew Spencer slugmand...@gmail.com wrote: Well I am just trying to get some kind of demo program running and learn about cells/celltable while I go because my app needs to display lots of tabular data and I understand this is the fastest way to do it. What I am doing is this: Get the supplies (List) Loop the supplies, adding each supplier key to supplierList Get the suppliers (Map) add supplierName to each Supply (as a String) Call setRowData for supplies Like you say, this is ok for now, but I am wondering if there is a better solution. At least this way I am updating every Supply record with the name from the Supplier when it is accessed - that way if a Supplier Name does change, it won't matter. Drew -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/bJevzjyvaLIJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Why the 10 second URL fetch limit?
The problem with using the Channel API this way is not quota usage, it's the rather significant amount of work required on the client server to make this work. Making an ajax call from client javascript is trivial. Making an ajax call PLUS setting up, tearing down, and processing messages from the channel API is a massive PITA. A modern development environment shouldn't make you go through this much trouble to do something so simple. Multithreading largely addresses this issue. Async services don't have the issue at all. Jeff On Fri, Jun 17, 2011 at 6:57 AM, Ikai Lan (Google) ika...@google.comwrote: It doesn't hold up the UI but it effectively blocks a connection. With multithreaded Java this isn't an issue, I suppose - but it will raise your average user facing latency and your application will not autoscale if it is frequent. If it is not frequent, the quote usage from the Channel API will be minimal. Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Fri, Jun 17, 2011 at 11:56 AM, Luis Montes monte...@gmail.com wrote: It's not in a task queue or cron job. It's definitely not holding up the UI since the fetch is happening in an AJAX call and the user knows that the call will take a few seconds. Putting it in a task queue means i have to store the results and poll the server to check if the results have finished, or use the channel API to push a notification back to the client. Those approaches both use more quota than would simply allowing for more time on a normal fetch. Luis On Thu, Jun 16, 2011 at 6:33 PM, Ikai Lan (Google) ika...@google.comwrote: Are you doing the urlfetch in an offline call (task queue or cron)? If so, the limit should be higher than 10 seconds as of SDK 1.4.1. I hope this isn't an online call, because it would suck for a user to wait 30 seconds before a page load. You're better off showing the user a loading screen, handing the job to a task queue, then updating the page when the fetch has finished. Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Fri, Jun 17, 2011 at 7:42 AM, Luis Montes monte...@gmail.com wrote: I understand why there's a 30 second limit for an http connection to GAE, but I don't understand why server to server URL fetches get timed out at 10 seconds. Seems like as long as you finish up all your processing in less than 30 you should be fine. How about upping the URL fetch max to 29 seconds? Thanks, Luis -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] How to design an in-memory cache?
Sounds like you could just keep it around for 30 minutes or an hour, refreshing it after it expires. What would be nice is to be able to perform the refresh asynchronously. Say it takes 5s to build a piece of data that every appserver uses for practically every request. Instead of relying on memcache, it would be nice to just keep this as static/application-scoped data in the appserver - and have a cron job that updates every instance every 15 minutes or so. This is easy to do if I can create threads, and a common pattern in the JavaEE world. Jeff On Tue, Jun 14, 2011 at 8:25 AM, tempy fay...@gmail.com wrote: Hello all, I'm wondering if its possible to design a cache using data in static variables. I have certain pieces of small and unchanging data, such as a banner for the top of the site whose contents might only change every couple of days. This is perfect for storing in a static variable and falling back onto the memcache as necessary. The one thing I don't know how to do is to clear that cache. If I want to update that data, I can clear the memcache, but there's no way that I can think of to clear static data cache, other than to wait until a particular instance goes down, and who knows when that will happen. Any ideas? Thanks! -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: When to use a String as ID? (using Objectify and GWT)
One case is when you need to save several different entities that all reference each other. Let's say A has a reference to B, and B has a reference to A. If you relied on key autogeneration you would need 3 puts: * Put A (which initializes the id) * Set B.a to the id of A, put B * Set A.b to the id of B, put A again Instead you can just allocate two ids up front and put both entities together (in a batch if you'd like). Jeff On Wed, Jun 15, 2011 at 3:26 AM, Drew Spencer slugmand...@gmail.com wrote: Thanks Jeff, I think I will use Longs as you suggest. I was originally thinking why add an extra piece of data when there's a unique value there? Well, now I know why! Just one question - the DatastoreService.allocateIdRange() - when would this be used? I have read the javadoc and understand that it reserves the IDs for bulk loading etc, but can you give an example of when it might be used. At the moment I have some code that creates 5 lists of my 5 entities (dummy data), puts all 5 lists to the datastore, links them up using keys, and then puts them all again. Is there a better way to do this? Thanks again for being so helpful. Drew -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/ErZFcwnAoRUJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Is the native API really so much faster than JDO and slim3?
I don't understand this code at all. You're querying the data and
then iterating over a separate call? And what does the extra call to
list.size() have anything to do with this?
The code linked from http://slim3demo.appspot.com/performance/ has
just gone back to calling list.size() without iterating the results,
which once again goes back to a bogus benchmark.
Jeff
On Wed, Jun 8, 2011 at 9:30 PM, Yasuo Higa higaya...@gmail.com wrote:
Slim3 uses LL API.
To resolve a strange issue that slim3 is faster than LL, I tried the
following samples:
One:
AsyncDatastoreService ds =
DatastoreServiceFactory.getAsyncDatastoreService();
Query q = new Query(Bar);
PreparedQuery pq = ds.prepare(q);
ListEntity list =
pq.asList(FetchOptions.Builder.withDefaults().limit(
Integer.MAX_VALUE));
for (Entity e : service.getBarListUsingLL()) {
e.getKey();
e.getProperty(sortValue);
}
Two:
AsyncDatastoreService ds =
DatastoreServiceFactory.getAsyncDatastoreService();
Query q = new Query(Bar);
PreparedQuery pq = ds.prepare(q);
ListEntity list =
pq.asList(FetchOptions.Builder.withDefaults().limit(
Integer.MAX_VALUE));
// VERY IMPORTANT
list.size();
for (Entity e : service.getBarListUsingLL()) {
e.getKey();
e.getProperty(sortValue);
}
The second one is much faster than the first one.
I fixed the samples to call list.size().
http://slim3demo.appspot.com/performance/
As a result, LL is as fast as slim3 (^^;
Yasuo Higa
On Thu, Jun 9, 2011 at 10:17 AM, Jeff Schnitzer j...@infohazard.org wrote:
Thank you for fixing the benchmark.
I am very curious. According to this new benchmark - it's hard to
tell without pushing the buttons a lot of times, but there seems to be
a trend - Slim3 is somewhat faster than the Low Level API.
Doesn't Slim3 use the Low Level API underneath? How can it possibly be
faster?
Jeff
On Wed, Jun 8, 2011 at 4:27 PM, Yasuo Higa higaya...@gmail.com wrote:
What I want to provide is a fair and casual benchmark.
As jeff advised, I modified samples as follows:
for (Entity e : service.getBarListUsingLL()) {
e.getKey();
e.getProperty(sortValue);
}
for (Bar bar : service.getBarListUsingSlim3()) {
bar.getKey();
bar.getSortValue();
}
for (BarObjectify bar : service.getBarListUsingObjectify()) {
bar.getKey();
bar.getSortValue();
}
for (BarJDO bar : service.getBarListUsingJDO()) {
bar.getKey();
bar.getSortValue();
}
LL API is much slower than before.
http://slim3demo.appspot.com/performance/
Yasuo Higa
On Thu, Jun 9, 2011 at 7:45 AM, Jeff Schnitzer j...@infohazard.org wrote:
Slim3 may be a nice piece of software, but it has not been
demonstrated to be faster than anything (including JDO). It might or
might not be faster - I don't know - but based on the sloppy
benchmarking, I'm pretty certain that the people making this claim
don't know either.
There's another ill-conceived performance claim on the Slim3 website:
You may worry about the overhead of global transactions. Don't worry.
It is not very expensive. There are three problems with their little
performance test:
1) It only measures wall-clock time, not cost.
2) It does not measure what happens under contention.
3) The numbers are obviously wrong - they don't even pass a smoke test.
Look at these numbers (from the Slim3 home page):
Entity Groups Local Transaction(millis) Global Transaction(millis)
1 67 70
2 450 415
3 213 539
4 1498 981
5 447 781
Just like the 1ms low-level API query performance in the benchmark
that spawned this thread, even a casual observer should be able to
recognize the obvious flaw - the numbers don't show any expected
relationship between # of entity groups or the use of global
transactions. Interpreted literally, you would assume that local
transactions are much faster for 5 entity groups, but global
transactions are much faster for 4 entity groups.
It's pretty obvious that the benchmark author just ran one pass and
took the numbers as-is. If you actually run multiple passes, you'll
find that there is enormous variance in the timing. The only way you
can realistically measure results like this on appengine is to execute
the test 100 times and take a median.
FWIW, I deliberately haven't made any performance claims for Objectify
because I haven't put the necessary amount of due diligence into
creating a proper set of benchmarks. It is not easy to measure
performance, especially in a dynamic environment like appengine. I
only hope that the Slim3 authors have put more care and thought into
crafting their library than they have their benchmarks.
Jeff
On Wed, Jun 8, 2011 at 2:34 PM, Gal Dolber gal.dol...@gmail.com wrote:
Slim3 is not only fast, the api is completely awesome. It has been my
choice
for a year now for all gae projects.
It includes name safety and and amazing querying
Re: [appengine-java] Re: Is the native API really so much faster than JDO and slim3?
Ok - so what you're saying is that the extra call to list.size()
before iterating through the list makes list iteration faster? Oddly
enough, this does seem to make a difference.
This looks like some sort of performance bug in the Low-Level API.
It's clearly not related to Slim3... except in as much as Slim3
inadvertently works around this bug. It's also very peculiar to
working with List results - you obviously can't call size() when
you're iterating through an unbounded result set.
The benchmark is bogus... which is to say, it does not show what you
think it shows.
Jeff
On Thu, Jun 9, 2011 at 12:09 AM, Yasuo Higa higaya...@gmail.com wrote:
It's *NO* bogus benchmark because the sample iterates the results.
http://code.google.com/p/slim3/source/browse/trunk/slim3demo/src/slim3/demo/controller/performance/GetLLController.java
Yasuo Higa
The code linked from http://slim3demo.appspot.com/performance/ has
just gone back to calling list.size() without iterating the results,
which once again goes back to a bogus benchmark.
Jeff
On Wed, Jun 8, 2011 at 9:30 PM, Yasuo Higa higaya...@gmail.com wrote:
Slim3 uses LL API.
To resolve a strange issue that slim3 is faster than LL, I tried the
following samples:
One:
AsyncDatastoreService ds =
DatastoreServiceFactory.getAsyncDatastoreService();
Query q = new Query(Bar);
PreparedQuery pq = ds.prepare(q);
ListEntity list =
pq.asList(FetchOptions.Builder.withDefaults().limit(
Integer.MAX_VALUE));
for (Entity e : service.getBarListUsingLL()) {
e.getKey();
e.getProperty(sortValue);
}
Two:
AsyncDatastoreService ds =
DatastoreServiceFactory.getAsyncDatastoreService();
Query q = new Query(Bar);
PreparedQuery pq = ds.prepare(q);
ListEntity list =
pq.asList(FetchOptions.Builder.withDefaults().limit(
Integer.MAX_VALUE));
// VERY IMPORTANT
list.size();
for (Entity e : service.getBarListUsingLL()) {
e.getKey();
e.getProperty(sortValue);
}
The second one is much faster than the first one.
I fixed the samples to call list.size().
http://slim3demo.appspot.com/performance/
As a result, LL is as fast as slim3 (^^;
Yasuo Higa
On Thu, Jun 9, 2011 at 10:17 AM, Jeff Schnitzer j...@infohazard.org wrote:
Thank you for fixing the benchmark.
I am very curious. According to this new benchmark - it's hard to
tell without pushing the buttons a lot of times, but there seems to be
a trend - Slim3 is somewhat faster than the Low Level API.
Doesn't Slim3 use the Low Level API underneath? How can it possibly be
faster?
Jeff
On Wed, Jun 8, 2011 at 4:27 PM, Yasuo Higa higaya...@gmail.com wrote:
What I want to provide is a fair and casual benchmark.
As jeff advised, I modified samples as follows:
for (Entity e : service.getBarListUsingLL()) {
e.getKey();
e.getProperty(sortValue);
}
for (Bar bar : service.getBarListUsingSlim3()) {
bar.getKey();
bar.getSortValue();
}
for (BarObjectify bar : service.getBarListUsingObjectify()) {
bar.getKey();
bar.getSortValue();
}
for (BarJDO bar : service.getBarListUsingJDO()) {
bar.getKey();
bar.getSortValue();
}
LL API is much slower than before.
http://slim3demo.appspot.com/performance/
Yasuo Higa
On Thu, Jun 9, 2011 at 7:45 AM, Jeff Schnitzer j...@infohazard.org
wrote:
Slim3 may be a nice piece of software, but it has not been
demonstrated to be faster than anything (including JDO). It might or
might not be faster - I don't know - but based on the sloppy
benchmarking, I'm pretty certain that the people making this claim
don't know either.
There's another ill-conceived performance claim on the Slim3 website:
You may worry about the overhead of global transactions. Don't worry.
It is not very expensive. There are three problems with their little
performance test:
1) It only measures wall-clock time, not cost.
2) It does not measure what happens under contention.
3) The numbers are obviously wrong - they don't even pass a smoke test.
Look at these numbers (from the Slim3 home page):
Entity Groups Local Transaction(millis) Global
Transaction(millis)
1 67 70
2 450 415
3 213 539
4 1498 981
5 447 781
Just like the 1ms low-level API query performance in the benchmark
that spawned this thread, even a casual observer should be able to
recognize the obvious flaw - the numbers don't show any expected
relationship between # of entity groups or the use of global
transactions. Interpreted literally, you would assume that local
transactions are much faster for 5 entity groups, but global
transactions are much faster for 4 entity groups.
It's pretty obvious that the benchmark author just ran one pass and
took the numbers as-is. If you actually run multiple passes, you'll
find that there is enormous variance in the timing. The only way you
can realistically measure results like this on appengine is to execute
Re: [appengine-java] Re: Is the native API really so much faster than JDO and slim3?
On Thu, Jun 9, 2011 at 1:32 AM, Gal Dolber gal.dol...@gmail.com wrote: I am not comparing reflexion vs byte-code generation or anything like that, apt generates code, is not a runtime technology. Like or not reflexion is known to be slower than actually writing the code. This is entirely irrelevant. We've now established that the issue at hand is a strange quirk in the Low-Level API and has nothing to do with reflection. 100% (or close to it) of the performance gain of Slim3 is because it calls .size() on a List before iterating it. Slim3 generates the minimal possible code you need to talk with the low level api, if the low level bench is faster is just because its not converting the Entity to the Pojo. You missed the point - in the benchmark, Slim3 was *faster* than the Low-Level API. This clearly indicated something was amiss, and now we've uncovered the actual cause. It turns out to be something quite interesting indeed. This is a classic case of what Feynman called Cargo Cult Science. You all believed that Slim3 should be faster than other APIs because code generation is faster than reflection, so when someone produced an ill-conceived benchmark that seemed to confirm your preconceived notion, you just accepted the entire narrative: Slim3 is fast because it doesn't use reflection! This is sloppy thinking (see: Confirmation Bias). If you haven't read this, it's a gem (I make a habit of re-reading it at least once a year): http://www.lhup.edu/~DSIMANEK/cargocul.htm Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Is the native API really so much faster than JDO and slim3?
Star this issue:
http://code.google.com/p/googleappengine/issues/detail?id=5167
I'm willing to bet I know exactly what's going on. When you call
size() first, the backing ArrayList is being created initially with
the proper size. If you don't call size() first, the backing
ArrayList is initialized small and reallocs organically up to the
10,000-element size. Thus the performance penalty.
The GAE internals obviously can obtain the size information ahead of
time; they just aren't initializing the ArrayList to the proper size
in this one case. I could probably find the exact line of code in the
SDK and submit a patch.
Jeff
On Thu, Jun 9, 2011 at 5:16 AM, Jeff Schnitzer j...@infohazard.org wrote:
Ok - so what you're saying is that the extra call to list.size()
before iterating through the list makes list iteration faster? Oddly
enough, this does seem to make a difference.
This looks like some sort of performance bug in the Low-Level API.
It's clearly not related to Slim3... except in as much as Slim3
inadvertently works around this bug. It's also very peculiar to
working with List results - you obviously can't call size() when
you're iterating through an unbounded result set.
The benchmark is bogus... which is to say, it does not show what you
think it shows.
Jeff
On Thu, Jun 9, 2011 at 12:09 AM, Yasuo Higa higaya...@gmail.com wrote:
It's *NO* bogus benchmark because the sample iterates the results.
http://code.google.com/p/slim3/source/browse/trunk/slim3demo/src/slim3/demo/controller/performance/GetLLController.java
Yasuo Higa
The code linked from http://slim3demo.appspot.com/performance/ has
just gone back to calling list.size() without iterating the results,
which once again goes back to a bogus benchmark.
Jeff
On Wed, Jun 8, 2011 at 9:30 PM, Yasuo Higa higaya...@gmail.com wrote:
Slim3 uses LL API.
To resolve a strange issue that slim3 is faster than LL, I tried the
following samples:
One:
AsyncDatastoreService ds =
DatastoreServiceFactory.getAsyncDatastoreService();
Query q = new Query(Bar);
PreparedQuery pq = ds.prepare(q);
ListEntity list =
pq.asList(FetchOptions.Builder.withDefaults().limit(
Integer.MAX_VALUE));
for (Entity e : service.getBarListUsingLL()) {
e.getKey();
e.getProperty(sortValue);
}
Two:
AsyncDatastoreService ds =
DatastoreServiceFactory.getAsyncDatastoreService();
Query q = new Query(Bar);
PreparedQuery pq = ds.prepare(q);
ListEntity list =
pq.asList(FetchOptions.Builder.withDefaults().limit(
Integer.MAX_VALUE));
// VERY IMPORTANT
list.size();
for (Entity e : service.getBarListUsingLL()) {
e.getKey();
e.getProperty(sortValue);
}
The second one is much faster than the first one.
I fixed the samples to call list.size().
http://slim3demo.appspot.com/performance/
As a result, LL is as fast as slim3 (^^;
Yasuo Higa
On Thu, Jun 9, 2011 at 10:17 AM, Jeff Schnitzer j...@infohazard.org
wrote:
Thank you for fixing the benchmark.
I am very curious. According to this new benchmark - it's hard to
tell without pushing the buttons a lot of times, but there seems to be
a trend - Slim3 is somewhat faster than the Low Level API.
Doesn't Slim3 use the Low Level API underneath? How can it possibly be
faster?
Jeff
On Wed, Jun 8, 2011 at 4:27 PM, Yasuo Higa higaya...@gmail.com wrote:
What I want to provide is a fair and casual benchmark.
As jeff advised, I modified samples as follows:
for (Entity e : service.getBarListUsingLL()) {
e.getKey();
e.getProperty(sortValue);
}
for (Bar bar : service.getBarListUsingSlim3()) {
bar.getKey();
bar.getSortValue();
}
for (BarObjectify bar : service.getBarListUsingObjectify()) {
bar.getKey();
bar.getSortValue();
}
for (BarJDO bar : service.getBarListUsingJDO()) {
bar.getKey();
bar.getSortValue();
}
LL API is much slower than before.
http://slim3demo.appspot.com/performance/
Yasuo Higa
On Thu, Jun 9, 2011 at 7:45 AM, Jeff Schnitzer j...@infohazard.org
wrote:
Slim3 may be a nice piece of software, but it has not been
demonstrated to be faster than anything (including JDO). It might or
might not be faster - I don't know - but based on the sloppy
benchmarking, I'm pretty certain that the people making this claim
don't know either.
There's another ill-conceived performance claim on the Slim3 website:
You may worry about the overhead of global transactions. Don't worry.
It is not very expensive. There are three problems with their little
performance test:
1) It only measures wall-clock time, not cost.
2) It does not measure what happens under contention.
3) The numbers are obviously wrong - they don't even pass a smoke test.
Look at these numbers (from the Slim3 home page):
Entity Groups Local Transaction(millis) Global
Transaction(millis)
1 67 70
2 450 415
3 213 539
4 1498 981
5 447 781
Re: [appengine-java] Re: Is the native API really so much faster than JDO and slim3?
Don't feel bad - I've now wasted over a day on this! Sorry if I've come across as grumpy in previous emails... but I've wasted over a day on this :-( Jeff On Thu, Jun 9, 2011 at 4:25 PM, Gal Dolber gal.dol...@gmail.com wrote: Ok, you actually got me to waste 1 hour on benchmarking and I hate to say that you are right. The use of reflexion seems not to be heavy enough to make the difference. Some extra features on slim3 make it slower when you don't make use them, I almost have a working path to avoid this problem, but even with the patch, both frameworks give almost the same performance On Thu, Jun 9, 2011 at 10:15 AM, Dennis Peterson dennisbpeter...@gmail.com wrote: Haha, excellent. I studied cargo cults a bit in anthropology classes, long ago, and never suspected how relevant they would be. You would probably enjoy this: http://www.fanfiction.net/s/5782108/1/Harry_Potter_and_the_Methods_of_Rationality Until Google makes a change, maybe the other frameworks should try the same trick? On Thu, Jun 9, 2011 at 8:31 AM, Jeff Schnitzer j...@infohazard.org wrote: On Thu, Jun 9, 2011 at 1:32 AM, Gal Dolber gal.dol...@gmail.com wrote: I am not comparing reflexion vs byte-code generation or anything like that, apt generates code, is not a runtime technology. Like or not reflexion is known to be slower than actually writing the code. This is entirely irrelevant. We've now established that the issue at hand is a strange quirk in the Low-Level API and has nothing to do with reflection. 100% (or close to it) of the performance gain of Slim3 is because it calls .size() on a List before iterating it. Slim3 generates the minimal possible code you need to talk with the low level api, if the low level bench is faster is just because its not converting the Entity to the Pojo. You missed the point - in the benchmark, Slim3 was *faster* than the Low-Level API. This clearly indicated something was amiss, and now we've uncovered the actual cause. It turns out to be something quite interesting indeed. This is a classic case of what Feynman called Cargo Cult Science. You all believed that Slim3 should be faster than other APIs because code generation is faster than reflection, so when someone produced an ill-conceived benchmark that seemed to confirm your preconceived notion, you just accepted the entire narrative: Slim3 is fast because it doesn't use reflection! This is sloppy thinking (see: Confirmation Bias). If you haven't read this, it's a gem (I make a habit of re-reading it at least once a year): http://www.lhup.edu/~DSIMANEK/cargocul.htm Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- Guit: Elegant, beautiful, modular and *production ready* gwt applications. http://code.google.com/p/guit/ -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: R: Re: New GWT/App Engine site
On Tue, Jun 7, 2011 at 11:07 PM, Giorgio Riccardi giorgio...@gmail.com wrote: Hi Jeff, 1) Polishing the navigation, as you said, it kind thought with GWT, but I absolutely agree with you that it needs to be improved... already some changes have been done, but going back at the same position level on the map is just a mystery ;-) Just leave the old window intact (but invisible) and render the new one on top. When you restore the old window, it should be more or less the same as it was. If the map widget gives you trouble, you don't even need to set it invisible... just layer it under the new panel with z-index. 2) I did not quite understand well what do you mean in just pick a random sample... Are you suggesting to load a pre-configured map/kml with some markers? When you arrive at the map, it's zoomed out all the way and blank. Never show an empty map to the user. Put some markers on there when zoomed out all the way, or use the google javascript client stuff to geoip the user and pick a reasonable level of zoom for them. Jeff -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Request was aborted after waiting too long to attempt to service your request.
Sorry, when I said I have a GWT app that takes one major hit at startup (possibly seconds) I meant the GWT client app itself, on startup, has a login() call to the server that can take seconds to process. Synchronizing facebook data and all that jazz. I'm not terribly concerned about this since the ratio of login()s to other queries is low, browser refreshes will find mostly cached data, and I am now hitting Facebook with multiple batches in parallel... but since I don't really know the limits, it would suck to find out my architecture doesn't work in-vivo. Jeff On Wed, Jun 8, 2011 at 1:50 AM, Ikai Lan (Google) ika...@google.com wrote: IIRC, the loading request latency doesn't affect the average time. It's an average, I believe, and it's a moving window. A long startup time, however, will result in the scheduler almost always favoring putting items in the pending queue instead of spinning up new instances, though. When scheduling a request, we do the lesser of: cold start time vs. sliding window of pending ms So if your app takes 10 seconds to startup and requests are spending 5 seconds in the pending queue, we will send the requests to the pending queue until the expected wait is 10+ seconds. If we decide to spin up new instances, we only spin up one instance at a time. During the spin up time, we continue to place new requests on the queue. I'm thinking we need a new section about scheduling, auto scaling and wallclock. The wallclock stuff is probably the most confusing ... Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Wed, Jun 8, 2011 at 1:50 PM, Jeff Schnitzer j...@infohazard.org wrote: Is this documented anywhere? It's something I figured out from occasional comments I've seen on this list over the years, but I've never seen it mentioned in the official documentation. And it's kinda important. In particular, I'd like to know what the bounds are for threaded java, and how the average time is computed - is it mean, median, or mode? I have a GWT app that takes one major hit at startup (possibly seconds) and then performs lots of smaller hits as the user interacts with the app. It would be nice to know if this will cause problems down the road... Jeff On Tue, Jun 7, 2011 at 6:41 PM, Ikai Lan (Google) ika...@google.com wrote: You get this error when your request waits in a pending queue. App Engine apps are autoscaled IF they can average under 1000ms. If they cannot, we do not give you additional instances, and requests line up in a pending queue. If your request waits in the pending queue for more than N seconds (I think N is 9 at the moment but I don't know for sure off the top of my head), this message is returned. You'll need to look at the average latency and figure out how to drop it to under 1000ms. Alternatively, if you enable concurrent Java requests, it'll be a higher ceiling before this error appears if you have Always On since you can serve a total of 3 (always on instances) * M (concurrency factor, I think this is 40 at the moment) requests at one time without requiring autoscaling. Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Wed, Jun 8, 2011 at 1:06 AM, holger holger.weissbo...@gmail.com wrote: Hello everyone, i am running a java REST service in the appengine and get this: Request was aborted after waiting too long to attempt to service your request. This may happen sporadically when the App Engine serving cluster is under unexpectedly high or uneven load. If you see this message frequently, please contact the App Engine team. I see this message frequently and customer requests fail because of it. Can anyone help me please! Thanks a lot. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/X3E4cnhhblA3Z3dK. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message
Re: [appengine-java] Re: Is the native API really so much faster than JDO and slim3?
You are wrong.
Try adding getProperty() calls to your LL performance test, and the
speed advantage of the LL API goes away. I don't know what to say
about Slim3, but here's my test case:
http://code.google.com/p/scratchmonkey/source/browse/#svn%2Fappengine%2Fperformance-test
I created 10,000 entities in the datastore that have the same format
as your test case - a single string property. Here's what happens
(try it - and remember to reload the urls several times to get a
realistic median value):
## Low Level API with just .size()
http://voodoodyne.appspot.com/fetchLLSize
The code:
ListEntity things =
DatastoreServiceFactory.getDatastoreService()
.prepare(new
Query(Thing.class.getAnnotation(javax.persistence.Entity.class).name()))
.asList(FetchOptions.Builder.withDefaults());
things.size();
Note that results are almost always under 2000ms. Wild guess I'd say
the median elapsed is ~1900, just like your example.
## Low Level API with actual fetch of the data
http://voodoodyne.appspot.com/fetchLL
The code:
ListEntity things =
DatastoreServiceFactory.getDatastoreService()
.prepare(new
Query(Thing.class.getAnnotation(javax.persistence.Entity.class).name()))
.asList(FetchOptions.Builder.withDefaults());
for (Entity ent: things)
{
ent.getKey();
ent.getProperty(value);
}
Note that the duration is now considerably longer. Eyeballing the
median elapsed time, I'd say somewhere around 3000ms.
## Objectify fetching from datastore
http://voodoodyne.appspot.com/fetch
Objectify ofy = ObjectifyService.begin();
ListThing things = ofy.query(Thing.class).list();
for (Thing thing: things)
{
thing.getId();
thing.getValue();
}
Note that the timing is pretty much the same as the LL API when it
includes actual fetches of the entity values. It is, no doubt, just a
little higher.
## A pure measurement of Objectify's overhead
http://voodoodyne.appspot.com/fakeFetch
This causes Objectify to translate 10,000 statically-created Entity
objects to POJOs. You can see the code here:
http://code.google.com/p/scratchmonkey/source/browse/appengine/performance-test/src/test/FakeFetchServlet.java
You'll notice (after you hit the URL a couple times to warm up the
JIT) that elapsed time converges to somewhere around 120ms.
---
Conclusion:
The numbers in the original benchmark are a result of improper
measurements. The actual wall-clock overhead for Objectify in this
test is ~4% (120ms out of 3000ms).
Further speculation on my part, but probably correct: The overhead of
reflection is unlikely to be a significant part of that 4%.
Sloppy work.
Jeff
On Wed, Jun 8, 2011 at 7:55 AM, Yasuo Higa higaya...@gmail.com wrote:
It is not bogus.
LazyList#size() fetches all data as follows:
public int size() {
resolveAllData();
return results.size();
}
Yasuo Higa
On Wed, Jun 8, 2011 at 11:32 PM, Dennis Peterson
dennisbpeter...@gmail.com wrote:
It's not my benchmark, it's Slim3's :) ...but you're right, it's bogus. I
asked on the main appengine group too, and it turns out the low-level
benchmark is doing lazy loading. With that fixed, their numbers come out
like yours.
I found this one too, which also gets results like yours:
http://gaejava.appspot.com/
On Wed, Jun 8, 2011 at 4:44 AM, Erwin Streur erwin.str...@gmail.com wrote:
Indeed Dennis's measurements are very suspicious. First you should do
a couple of warming ups on each of the implementations to prevent
pollution like the JDO classpath scan for enhanced classes (which is
one of the reasons for the high initial run). Then do a couple of run
to determine a range of measurements to spot outlyers. your low-level
API 2millis is definately one.
When I did the measurements I got the following results
low-level: 1150-1550
Slim3: 1150-1600
Objectify: 1950-2400
JDO: 2100-2700
These measurements confirm that GAE designed implementations are
faster then the GAE implementation of a generic data access layer
(JDO), but not so extrem as initially posted.
The initial response using JDO is a known issue and especially low
trafic website should not use it or use the always on feature (maybe
this will change in the new pricing model)
Regards,
Erwin
On Jun 7, 11:00 am, Ian Marshall ianmarshall...@gmail.com wrote:
The low-level API does indeed look very fast.
Just a comment on JDO: repeat runs roughly halve the JDO run time. I
presume that this is because for repeat runs the JDO persistence
manager factory has already been constructed.
On Jun 6, 8:44 pm, DennisP dennisbpeter...@gmail.com wrote:
I'm looking at this online
demo:http://slim3demo.appspot.com/performance/
Sample run:
The number of entities: 1
low-level API:get: 2 millis
Slim3: 2490 millis
JDO: 6030 millis
Is the low-level API really that much faster?
--
You received this message because
Re: [appengine-java] Does AppEngine support Servlet 3.0 ServletContainerInitializer
Unfortunately GAE does not support Servlet 3.0. Please star: http://code.google.com/p/googleappengine/issues/detail?id=3091 Jeff On Wed, Jun 8, 2011 at 11:45 AM, Marcel Overdijk marceloverd...@gmail.com wrote: Does AppEngine support Servlet 3.0 ServletContainerInitializer for code-based configuration of Servlet Container? -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Is the native API really so much faster than JDO and slim3?
Slim3 may be a nice piece of software, but it has not been
demonstrated to be faster than anything (including JDO). It might or
might not be faster - I don't know - but based on the sloppy
benchmarking, I'm pretty certain that the people making this claim
don't know either.
There's another ill-conceived performance claim on the Slim3 website:
You may worry about the overhead of global transactions. Don't worry.
It is not very expensive. There are three problems with their little
performance test:
1) It only measures wall-clock time, not cost.
2) It does not measure what happens under contention.
3) The numbers are obviously wrong - they don't even pass a smoke test.
Look at these numbers (from the Slim3 home page):
Entity Groups Local Transaction(millis) Global Transaction(millis)
1 67 70
2 450 415
3 213 539
4 1498981
5 447 781
Just like the 1ms low-level API query performance in the benchmark
that spawned this thread, even a casual observer should be able to
recognize the obvious flaw - the numbers don't show any expected
relationship between # of entity groups or the use of global
transactions. Interpreted literally, you would assume that local
transactions are much faster for 5 entity groups, but global
transactions are much faster for 4 entity groups.
It's pretty obvious that the benchmark author just ran one pass and
took the numbers as-is. If you actually run multiple passes, you'll
find that there is enormous variance in the timing. The only way you
can realistically measure results like this on appengine is to execute
the test 100 times and take a median.
FWIW, I deliberately haven't made any performance claims for Objectify
because I haven't put the necessary amount of due diligence into
creating a proper set of benchmarks. It is not easy to measure
performance, especially in a dynamic environment like appengine. I
only hope that the Slim3 authors have put more care and thought into
crafting their library than they have their benchmarks.
Jeff
On Wed, Jun 8, 2011 at 2:34 PM, Gal Dolber gal.dol...@gmail.com wrote:
Slim3 is not only fast, the api is completely awesome. It has been my choice
for a year now for all gae projects.
It includes name safety and and amazing querying utils.
Very recommendable!
On Wed, Jun 8, 2011 at 3:41 PM, Jeff Schnitzer j...@infohazard.org wrote:
You are wrong.
Try adding getProperty() calls to your LL performance test, and the
speed advantage of the LL API goes away. I don't know what to say
about Slim3, but here's my test case:
http://code.google.com/p/scratchmonkey/source/browse/#svn%2Fappengine%2Fperformance-test
I created 10,000 entities in the datastore that have the same format
as your test case - a single string property. Here's what happens
(try it - and remember to reload the urls several times to get a
realistic median value):
## Low Level API with just .size()
http://voodoodyne.appspot.com/fetchLLSize
The code:
ListEntity things =
DatastoreServiceFactory.getDatastoreService()
.prepare(new
Query(Thing.class.getAnnotation(javax.persistence.Entity.class).name()))
.asList(FetchOptions.Builder.withDefaults());
things.size();
Note that results are almost always under 2000ms. Wild guess I'd say
the median elapsed is ~1900, just like your example.
## Low Level API with actual fetch of the data
http://voodoodyne.appspot.com/fetchLL
The code:
ListEntity things =
DatastoreServiceFactory.getDatastoreService()
.prepare(new
Query(Thing.class.getAnnotation(javax.persistence.Entity.class).name()))
.asList(FetchOptions.Builder.withDefaults());
for (Entity ent: things)
{
ent.getKey();
ent.getProperty(value);
}
Note that the duration is now considerably longer. Eyeballing the
median elapsed time, I'd say somewhere around 3000ms.
## Objectify fetching from datastore
http://voodoodyne.appspot.com/fetch
Objectify ofy = ObjectifyService.begin();
ListThing things = ofy.query(Thing.class).list();
for (Thing thing: things)
{
thing.getId();
thing.getValue();
}
Note that the timing is pretty much the same as the LL API when it
includes actual fetches of the entity values. It is, no doubt, just a
little higher.
## A pure measurement of Objectify's overhead
http://voodoodyne.appspot.com/fakeFetch
This causes Objectify to translate 10,000 statically-created Entity
objects to POJOs. You can see the code here:
http://code.google.com/p/scratchmonkey/source/browse/appengine/performance-test/src/test/FakeFetchServlet.java
You'll notice (after you hit the URL a couple times to warm up the
JIT) that elapsed time converges to somewhere around 120ms.
---
Conclusion:
The numbers in the original benchmark are a result of improper
measurements. The actual wall-clock overhead for Objectify in this
test
