here's a really good talk about how objects are mapped into the BigTable datastore and how relationships are actually represented in the system: http://sites.google.com/site/io/under-the-covers-of-the-google-app-engine-datastore this sort of highlites how relationships actually work in BigTable (which is quite different than they work in an RDB).
here is another one from Max Ross where he advocates denormalization for query optimization: http://code.google.com/events/io/2009/sessions/SofterSideofSchemas.html he kind of goes into how the BigTable datastore is simply schema-less. Very interesting stuff. hope that helps. -bryce On Thu, Nov 5, 2009 at 12:34 PM, Rusty Wright <rwright.li...@gmail.com> wrote: > > Good points. In my case, so far, the copied objects are small and not > complicated, which is why my method appealed to me. > > I feel like there's some fundamental concept that I'm not getting and it has > to do with how objects are mapped onto the Big Table data store. Watching > the videos from Google I/O, those guys just wave their hands and make it > sound like it's all so easy, if you know what you're doing. > > > bryce cottam wrote: >> I don't think that duplicating the whole Department object as a child >> of a Person is all that good of an idea. First off, if the Department >> object gets complicated and has it's own child objects all that data >> will be living on the Person, which isn't really needed. The whole >> reason for the FK style classes is to pull over only those fields you >> would want to query on when selecting a Person object like: >> select from Person where department.name == "ABC" >> (which is something you cannot do otherwise) >> >> So, I've found that either having a list of Key (or encoded string >> ids) on a Person instance is usefull (when you are not wanting to >> query on fields of the Department when selecting a Person, >> Or having a list of lightweight copies of a Department (as opposed to >> the whole Department object) >> It seems very natural to me to determine equality of two persisted >> objects by comparing their Key instance, rather than simply their >> name. For one reason; it's somewhat difficult to ensure uniqueness >> for arbitrary fields on an object in the GAE unless that fields is >> being used to create the key (like an email address or something), if >> you are using auto-generated Keys, this isn't really an option. >> >> So, assuming you have the lightweight FK style object as a child of >> the Person class, then when the source of truth changes (i.e. the real >> Department instance), you'd need to find all lightweight FK style >> copies of that instance and update them accordingly. This is part of >> the whole root of this discussion: how do you do this in a way that >> minimized code duplication and boilerplate code blocks (like >> constructors that manually copy fields from getters to setters etc.) >> >> The solution I've been working with is the ligthweight FK style "copy" >> of the real instance of an object, and some simple reflection to map >> fields from the real object to the lightweight FK "copy". >> >> As far as detachable="true|false" my main concern with that right now >> is how the GWT RPC serializer handles detachable entities (i.e. it >> doesn't handle them well). There are certainly advantages to >> detaching an object, however, if the updates to the object are >> happening on the server, then there is no need to detach the object. >> >> >> >> On Thu, Nov 5, 2009 at 1:09 AM, Rusty Wright <rwright.li...@gmail.com> wrote: >>> I think this is an important point. The light bulb went off over my head >>> after watching this video. This is what he calls a property list. For me >>> it was also a good example of how to think about denormalizing, which comes >>> up repeatedly. >>> >>> So instead of the usual OO way of thinking of a department etc. aggregating >>> people, turn it on its head and look at it as a person has a property, >>> which is their department, and in this example, they can be in multiple >>> departments so it's a list of departments, a property list. As you point >>> out, the queries for this are slick. And it sounds like you don't even >>> need the list of people in the department; it's redundant. >>> >>> I'm still puzzled about what to do when a property value changes; for >>> example, suppose the department named the Ministry of Propaganda changes >>> its name to the Ministry of Disinformation. Does the property list on >>> Person contain a list of Department objects, or a list of Department Key >>> objects? I'm thinking the former, Department objects. If we have the >>> Department class configured with detachable="false", each time you fetch a >>> Department object (matching on its name let's say), you get a new >>> unparented Department object, which you add to the Person's department >>> list, whereupon it becomes parented by Person. The fetched Department >>> object is essentially a clone. (That's assuming I understand how it works >>> when you have detachable="false"; it's sort of like the singleton pattern.) >>> As part of fetching the cloned Department object, the Department class has >>> a masterKey field, type Key, which you set to the Key of the Department in >>> the table, the one it was cloned from. So > whe >>> n the Department's name changes, the query to get all its cloned >>> Departments could be simple and quick. But now I'm fuzzy on how you change >>> the master Department's name; since it's detachable="false" we can't fetch >>> it and update it and save it back, so I guess we have to replace it in the >>> master table and then replace all of its clones in all Person lists? >>> >>> >>> bryce cottam wrote: >>>> ah, gottcha, well there is actually a really good google i/o talk on >>>> using list/collections and how to optimize them. I generally am using >>>> "owned" lists for smaller collections,and un-owned ones are >>>> represented by the child object containing an FK type class as a >>>> "backwards pointer" to the parent. If I want to get all the children >>>> for a given parent, I can query "select from Child where parent.id = >>>> :myParentId". Of course, the Child.parent field needs to be embedded >>>> in order to query off it in this manner. So, in your case I'd imagine >>>> a Person would be in a handful of Institutions, but an Institution >>>> could have tons and tons of Person instances in it. So I'd think >>>> you'd want to keep the relationship defined in the Person.institutions >>>> collection, and anytime you need to query "who's in InstitutionX?" you >>>> can do the query against the Person data class. There is a rather >>>> slick way to do this in JDO: >>>> >>>> String filter = "institutions.contains(inst) && inst.id == >>>> :myInstitutionId"; >>>> Query q = pm.newQuery(Person.class, filter); >>>> q.declareVariables("Institution inst"); >>>> List<Person> peopleInMyInstitution = query.execute(myInstitutionId); >>>> >>>> the key is this line: >>>> String filter = "institutions.contains(inst) && inst.id == >>>> :myInstitutionId"; >>>> where you're (basically) inspecting fields of individual values found >>>> in the collection "institutions" in JDOQL. >>>> There are slicker ways of getting the results back (like a KEY_ONLY >>>> query) which can help performance. >>>> >>>> if course, if you don't care about querying against fields of the >>>> Institution, then changing the Person.institutions model to just be a >>>> List<Key> or List<String> would do you fine, in that case your query >>>> just becomes: >>>> String filter = "institutions.contains(:myInstitutionId)"; >>>> Query q = pm.newQuery(Person.class, filter); >>>> List<Person> peopleInMyInstitution = query.execute(myInstitutionId); >>>> >>>> (thus removing the declareVariables call and simplifying the filter >>>> >>>> here is the i/o session, it's got some good tips: >>>> http://code.google.com/events/io/2009/sessions/BuildingScalableComplexApps.html >>>> >>>> hope that helps! >>>> -bryce >>>> >>>> >>>> >>>> On Wed, Nov 4, 2009 at 11:50 PM, James H <james.hollier...@gmail.com> >>>> wrote: >>>>> Thanks Bryce...here's the earlier post: I no longer get the error >>>>> stack above...I just added >>>>> embeddedOnly="true" to the PersistanceCapable tag in BookFk class. >>>>> You're right, I could use the collection technique on the FKs but I >>>>> wonder the pros/cons of such collections on what could be very large >>>>> collections. For example, say an Institution like Baylor with 25,000 >>>>> Persons? Guess its >>>>> a matter of using "owned" for small Sets versus "unowned" for large >>>>> Sets of data right? >>>>> >>>>> On Nov 5, 12:36 am, bryce cottam <bcot...@gmail.com> wrote: >>>>>> sorry, I meant "1:45", not 1:15 >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Nov 4, 2009 at 11:36 PM, bryce cottam <bcot...@gmail.com> wrote: >>>>>>> yeah, the Key class hasn't caused me any issues. I've actually >>>>>>> created an appengine-utils gwt module that makes it easier to >>>>>>> serialize GAE data types over GWT RPC channels >>>>>>> (www.resmarksystems.com/code/app-engine). >>>>>>> I think I'm in a different time zone than you James, I see quite a few >>>>>>> posts from you on Nov 4, but I don't see one at (or really near) 1:15, >>>>>>> can you re-state what you'd like me to review? I'd be happy to help >>>>>>> where I can. >>>>>>> thanks! >>>>>>> -bryce >>>>>>> On Wed, Nov 4, 2009 at 11:30 PM, James H <james.hollier...@gmail.com> >>>>>>> wrote: >>>>>>>> Bryce, double-check me on my Nov 4, 1:45pm post. >>>>>>>> Bryce/Rusty, I'm getting the sense that if I don't need a natural key >>>>>>>> like email address then I should just use plain Key as the type and >>>>>>>> let GAE generate the full value including parent key for entity group >>>>>>>> objects. Otherwise, if I choose to use a natural key like an email >>>>>>>> address then I would use the encoded Key. Regardless of approach, the >>>>>>>> type Key can be converted to/from String using keyToString() and >>>>>>>> stringToKey() for stuffing into html form hidden fields, etc. >>>>>>>> Rusty, I skimmed the blog but it was too deep for me at this >>>>>>>> hour...I'll try again tomorrow! >>>>>>>> On Nov 4, 11:39 pm, bryce cottam <bcot...@gmail.com> wrote: >>>>>>>>> Yeah, I've been re-considering my detachable="true" configuration and >>>>>>>>> thinking that i may want to go for detachable="false" for gwt reasons. >>>>>>>>> Using detachable="true" creates a field that is of type Object[] >>>>>>>>> (which makes the gwt RPC raise fits). >>>>>>>>> Currently, I find that having the actual Key object is handy (for >>>>>>>>> getting the type, any parent key references etc.) but that is mainly >>>>>>>>> stuff I do on the server side, so the client could probably live just >>>>>>>>> fine and dandy with an encoded String. >>>>>>>>> On Wed, Nov 4, 2009 at 9:10 PM, Rusty Wright >>>>>>>>> <rwright.li...@gmail.com> wrote: >>>>>>>>>> Did you see this? This article has been referenced a couple of >>>>>>>>>> times today: >>>>>>>>>> http://timepedia.blogspot.com/2009/04/google-appengine-and-gwt-now-ma... >>>>>>>>>> The part about using detachable = "false" is interesting. >>>>>>>>>> James H wrote: >>>>>>>>>>> Datanucleus, both suggestions worked but I like the "embeddedOnly" >>>>>>>>>>> best in order to avoid a dummy key on every FK class! >>>>>>>>>>> Bryce, I no longer get the error above...I just added >>>>>>>>>>> embeddedOnly="true" to the PersistanceCapable tag in BookFk class. >>>>>>>>>>> You're right, I could use the collection technique on the FKs but I >>>>>>>>>>> wonder the pros/cons of such collections on what could be very large >>>>>>>>>>> collections. For example, say Baylor with 25,000 members? Guess >>>>>>>>>>> its >>>>>>>>>>> a matter of using "owned" for small Sets versus "unowned" for large >>>>>>>>>>> Sets of data. Regardless, your FK pattern will save a whole lot of >>>>>>>>>>> code management!!! >>>>>>>>>>> Any further feedback welcome...I'll continue testing! >>>>>>>>>>> On Nov 4, 12:12 pm, datanucleus <andy_jeffer...@yahoo.com> wrote: >>>>>>>>>>>>> Needs a PK either way. Pick a field, any field. >>>>>>>>>>>> Or just set "embeddedOnly" as true ... if you really never want to >>>>>>>>>>>> persist one of those in its own right- Hide quoted text - >>>>>>>>> - Show quoted text -- Hide quoted text - >>>>>> - Show quoted text - >>>>>> >> >> > > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~----------~----~----~----~------~----~------~--~---
