On 2018/07/10 8:27 PM, Randall Smith wrote:
> One follow-up: Do you know if the dump output is "deterministic" over
> time? That is, if I diff two dumps taken at different times, will the
> unchanged material be in the same order and so on? Or is the ordering
> effectively random?
> My underlying question is "can text-comparing two DB dumps be used to
> determine what has changed?"
On 2018/07/11 Simon wrote:
I am not sure if it is 100% deterministic - it probably is, however, I would
like to point out that while parsing a dump (supposing it IS deterministic) is
possible, writing some code to check congruence between two DBs at the business
end of the SQLite API is significantly better, much easier and always 100%
deterministic.
Not only that, but the sqldiff command-line utility (download page) does it
already (though you may require a more specific result, but at a minimum its a
good start).
Is there perhaps a specific difficulty which makes you think that parsing the
dump would provide a better/easier insight into which data changed?
Fundamental problems with SQLite or other binary representations are (a) the
information represented is opaque unless one spends time and money creating
bespoke tools to allow viewing and technical reviews of the content, and (b)
there is no simple way to allow concurrent development of info by several
people and to reconcile independent changes into a coherent whole ("merging").
These are both mission critical for a team effort of any size (even size=2!).
The software industry has historically avoided these problems by storing
everything in the form of text files, and has developed elaborate tools and
procedures for viewing, reviewing, storing, and merging information in this
form and as a result large teams can collaborate on a rapidly evolving body of
digital information easily and well.
Binary file formats like SQLite, while having many compelling advantages, have
a hard time penetrating into areas where multiple people need to collaborate on
an evolving body of information because of the limitations described above.
IMO this is an urgent problem and one that has not been solved very well for
SQLite. I don't have the wherewithal to solve it generally, but I am trying to
see if there are ways to bridge the gap between SQLite DBs and existing
team-capable workflows built around text files.
My wishlist is:
(o) Allow humans to view the contents of a DB without custom tools.
(o) Have a way to see what has changed between V1 and V2 of a database, e.g.,
for a "change review."
(o) Have a way to merge two independent sets of database changes into a single
result in an understandable way.
(o) Have a way to make changes (update, insert, delete) to the DB data in a
pinch without specialized tools.
I'm thinking the dump approach you described previously has promise provided
certain criteria are met. Interestingly, the text representation produced by
dump is about the same size as the "normal" binary form, and it will compress
to about 1/8 the size of the binary form. So it's not a bad archival format.
Randall.
_______________________________________________
sqlite-users mailing list
[email protected]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
