Excerpts from Steven Hardy's message of 2013-09-11 03:37:40 -0700: > Hi all, > > I'm investigating some issues, where data stored to a text column in mysql > is silently truncated if it's too big. > > It appears that the default configuration of mysql, and the sessions > established via sqlalchemy is to simply warn on truncation rather than > raise an error. > > This seems to me to be almost never what you want, since on retrieval the > data is corrupt and bad/unexpected stuff is likely. > > This AFAICT is a mysql specific issue[1], which can be resolved by setting > sql_mode to "traditional"[2,3], after which an error is raised on truncation, > allowing us to catch the error before the data is stored. > > My question is, how do other projects, or oslo.db, handle this atm? > > It seems we either have to make sure the DB enforces the schema/model, or > validate every single value before attempting to store, which seems like an > unreasonable burden given that the schema changes pretty regularly. > > Can any mysql, sqlalchemy and oslo.db experts pitch in with opinions on > this?
I do think that setting stricter sql modes is the right way to go. Note that I worked around this within Heat for JSON fields thusly: https://git.openstack.org/cgit/openstack/heat/commit/?id=1e16ed2d However, I do think we should make it a priority to protect the database and the entire service from abnormally large values. The moment at which we are serializing a data structure to the database is a bit late to mitigate the cost of handling it. Here is an example of the kind of border protection we need: https://review.openstack.org/#/c/44585/ I want to detect that we overflowed a big column, and I think that if it ever actually happens, it is a critical bug. _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
