On 02/07/13 19:50, Boris Pavlovic wrote: > > *) DB Archiving > a) create shadow tables > b) add tests that checks that shadow and main table are synced. > c) add code that work with shadow tables. >
Hi Boris & all, I have a few points regarding db archiving work that I am growing more concerned about, so I though I might mention them on this thread. I pointed them out ad-hoc on a recent review https://review.openstack.org/#/c/34643/ and there is some discussion there already, although was not very fruitful. I feel that there were a few design oversights and as a result it has a couple of rough edges I noticed. First issue is about the fact that shadow tables do not present a "view of the world" themselves but are just unconstrained rows copied from live tables. This is understandably done for performance reasons while archiving (with current design ideas in place), but also causes issues when migrations affect more than one table. Especially if data migrations need to look at more tables at once, the actual number of table joins needed in order to consider everything grows exponentially. It could be argued that these are not that common, but is something that will make development more difficult and migrations painful once it comes up. To put it shortly - this property generally makes it harder to reason about data. Second point (and it ties in with the first one since it makes it difficult to fix) - Maybe shadow table migrations should be kept separate, and made optional? Currently there is a check that will fail the tests unless the migration is done on both tables, which I think should be removed in favour of separate migrations. Developers should still migrate both of course - but deployers should be able to choose not to do it according to their needs/scale. I am sure there are people on this list that can chip in more on this subject (I've had a brief discussion with lifeless on this topic on IRC). I'm afraid that if you agree that these are in fact problems - you might also agree that we might want to go back on some of the design decisions made around db archiving (like having unconstrained tables in the same db for example). I'd be happy to hear some of the angles that I may have missed, Cheers, Nikola _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev