imagine you have a 100 000 data set. each data has got 4 properties that you would like to use for filtering.
then, you will have one Kind with your data, and 4 children Kinds for the properties to be filtered. you can then query each children kind in different queries. in this case, 4 queries. each query will give a maximum of 1000 entities. for each query result, you will get the parent. and then, associate those 4 lists of parents (remove duplicate, apply logic, ...). if you want to get a complete result, you will add extra query to solve the 1000 entities limit. is this correct ? On appengine, it seems to be the best way to use complex filter. but requires a lot of queries. On Aug 20, 7:24 pm, Juraj Vitko <[email protected]> wrote: > Hi, yes, exactly - using filteredTableEntry.getKey().getParent() -- > that way they can be updated in a single transaction too. > > The filteredTableEntry can be created like new Entity > ('FT_User_isClient', parentUserKey). > > When any of the primary record (User in this example) data changes > (eg. the isClient field, or a value in one of the sortable columns), I > assume the fastest way to update the filtered entry is to overwrite it > with a new one, instead of fetching it first (I may be wrong here). > The only caveat is that when you generate a new unique suffix for the > new entry (which overwrites the previous one), the record might 'jump' > in the UI, ie in a Grid that is currently paged through. > > On Aug 20, 6:59 pm, Philippe <[email protected]> wrote: > > > > > looks interesting, but I do not understand all. > > how do you get back to your data (if I understood, you query on > > specific kinds that are your pre-filtered value). > > Do you get your data using ancestor ? > > > If you have time to give a short example, it will be great ! > > thank you, > > Philippe > > > On Aug 20, 3:04 pm, Juraj Vitko <[email protected]> wrote: > > > > So guys thanks for the good IRC chat:) > > > > Basically even if this was implemented, it would be too costly on > > > resources. > > > > What I needed, was to change my thinking to better fit App Engine. The > > > good thing is, that if one optimizes for App Engine, then his app will > > > run faster _everywhere_. > > > > Basically, the best method for being able to show filtered data in a > > > Grid that can be sorted _and_ paged on any column, one needs to pre- > > > filter the data (upon created/update/delete) into separate kinds. This > > > filtered kind then can be sorted on using the default indexes, because > > > filter was already applied. Also, the data stored in this filtered > > > kind is not the exact copy of the original data, but rather its string > > > representation with unique suffixes to support lexicographical > > > ordering and no-equality-restart pagination. > > > The advantage of this is also that a much more sophisticated filtering > > > can be used, not just the built-in query operators. > > > One has to add logic to support recomputation of the filtered kind if > > > the filter code or sorting-enabled columns change as the app is > > > evolving, but that's a one time job and can be reused if done > > > properly. > > > > On Aug 19, 6:01 am, Juraj Vitko <[email protected]> wrote: > > > > > sorry! correction of: "filter column is named 'f', value is named 'v'" > > > > > the 'f' and 'v' are actually a the same property now, so: > > > > > select from RelationIndex where f == 'roleA' && f == 'registered' && f > > > > == '{User}' && f == '(lastName') && f > 'a.Smith:xJ8mFc9r' order by f > > > > asc; > > > > > like that. > > > > > On Aug 19, 3:49 am, Juraj Vitko <[email protected]> wrote: > > > > > > On a query with equality filter on a list property, would it be > > > > > possible to: > > > > > > 1) _not_ discard the sort order on that property, > > > > > 2) allow for inequality test on smallest/highest value of that list > > > > > property (similar to ordering) > > > > > > That way it would be possible to implement "relation indexes" using a > > > > > table with a single list property (no custom indexes required!) > > > > > > The property would look something like: [ a.val:unique, filter1, > > > > > filter2, filterN, z.val:unique ] > > > > > > The two 'val' is the same value to be lexicographically ordered (so > > > > > numeric types must be converted to appropriate strings). > > > > > The a. and z. prefixes would be actually the lowest and highest > > > > > printable ascii chars, just so that they are always the lowest and > > > > > highest value of the list property, and can be used in asc and desc > > > > > ordering. > > > > > The filters would be used for equality filtering. > > > > > > Then this query would be possible: > > > > > select from kind where prop == filter1 && prop == filter2 && prop > > > > > > 'a.someval:unique' order by prop asc > > > > > (the optional greater-than operator is there to support paging) > > > > > > The sort order would _not_ be dropped only if specifically requested > > > > > by the user, as to not incur needless overhead on current queries. > > > > > > A real-use example of a relation index table shared by multiple kinds: > > > > > > select from RelationIndex where f == 'roleA' && f == 'registered' && f > > > > > == '{User}' && f == '(lastName') && v > 'Smith:xJ8mFc9r' order by v > > > > > asc; > > > > > > - filter column is named 'f', value is named 'v' > > > > > - the { and } are used to distinguish a kind name filter > > > > > - the ( and ) are used to distinguish a column name filter > > > > > - the 'v' always contains value belonging to that table/column > > > > > - 'xJ8mFc9r' is the unique suffix > > > > > > The benefits of the "relation indexes" technique using a single list > > > > > property: > > > > > 1) single-query forward/backward bookmark paging (no equality restarts > > > > > needed), > > > > > 2) asc/desc ordering on any number of columns (one column at a time) > > > > > 3) no custom indexes required (hard-limited to 100 per app) > > > > > > One could argue that it is simpler to implement relation indexes using > > > > > a table with 2 columns: "filters" and "value" - however: > > > > > 1. this table needs a custom index > > > > > 2. due to the custom index, max. 4 filters can be queried on at the > > > > > same time (tested) > > > > > 3. this custom index must enumerate all the filters (filter asc, > > > > > filter asc, filter asc, filter asc, value asc + the same for value > > > > > desc) - otherwise custom index space is wasted with permutations > > > > > 4. due to point 3., each row in the index table must contain a 'dummy' > > > > > filter value that is used to "park" the unused filters, only so that > > > > > the query always matches the custom index definition, ie. "where > > > > > filter == 'abc' && filter == 'def' && filter == 'dummy' && filter == > > > > > 'dummy'" > > > > > 5. both runtime and storage overhead of this approach are much higher > > > > > than the single-property approach (due to the list property being > > > > > multiplied by 2 for indexing) > > > > > > Your opinions? > > > > > J. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
