What is a Zigzag Merge? Can somebody point me towards some documentation. I've not heard of this before.
On Sep 27, 4:29 am, Alfred <[email protected]> wrote: > Hi Jason, > > Thanks for filing this issue. These are all good suggestions. You can > do a large portion of this analysis today by collecting performance > metrics. I would suggest statistically sampling a % of your queries in > the following way: > > eq_properties = a set of the property names that have equality filters > on them > sort_order = a ordered list of the sort orders > has_ancestor = if the query has an ancestor filter > key = (has_ancestor, eq_properties, sort_order) > > then sample: > > key -> occurrences > key -> time / results > > This way you can find and optimize slow frequently used queries (and > monitor the performance over time). > > - Alfred > > On Sep 26, 8:08 am, Jason Collins <[email protected]> wrote: > > > > > > > > > We've been working on a "billing optimization project" for our large > > app, steprep. Right now, we are tackling index optimizations - > > datastore writes form the largest part of our bill. > > > Zig zag merge has let us do some really crazy and unexpected things - > > in a good way. Thanks! > > > However, we're somewhat concerned with the long term health of these > > queries. That is, today, they work and will offer us substantial > > savings over previous index approaches. But algorithm changes and data > > distribution changes may make our queries perform quite differently > > over time. (We have the additional difficulty of our test deployment > > having a different data distribution than our prod deployment, so we > > kind of have to cross our fingers and hope for the best when dropping > > indexes on prod...) > > > It would be great to get some meta information back from a query to > > help track the efficiency of our zig zag merge queries over time > > (i.e., as data distributions change). E.g.: > > > - which indexes were used to generate the results > > - how much zig-zag "thrashing" occurred (perhaps how many entries you > > needed to consider to find the results) > > - tell the query planner to _not_ use a particular index, so that we > > can simulate the drop of an optimal index to test if zig zag will be > > efficient > > > I'm really excited about what we can do with zig zag, but we have > > concerns about how we will keep it healthy over time. > > > I've openedhttp://code.google.com/p/googleappengine/issues/detail?id=5978 > > for this request; please star it if you feel it would be helpful. > > > Thanks, > > j -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
