Hi, Many thanks for the very detailed answer, it is of great help
2017-12-26 15:28 GMT+01:00 Dan Kennedy <danielk1...@gmail.com>: > On 12/25/2017 11:17 PM, Josu Diaz de Arcaya wrote: > >> Hi all, >> >> I am very happy with the inclusion of LSM extension in sqlite3's code >> base. >> I have a question regarding >> the merging of segments within the database. >> >> I noticed that autowork enabled was not enough for keeping disk >> consumption >> at bay, the generated database >> was N times larger than the generated by sqlite3. >> > > If you're doing lots of operations on the same key - i.e. deleting and > recreating the entries, or updating them often, then an LSM database can > end up much larger than the equivalent b-tree - basically because it is so > lazy about deleting old keys. Is your workload like that? Yes, it is a general purpose workload, I guess it makes perfect sense that it grows larger than a b-tree database One other thing that worries me is memory consumption. When running a workload with sqlite3 memory grows to a certain point but stays more or less stable for the entire duration of the workload, however, with lsm memory starts being lower than sqlite3 but keeps growing with no apparent ceiling (some GB), it is not until I close the connection with the database that everything goes back to normal. For more details, the workload is pretty intense, it opens the database and executes thousands of operations (lsm_insert, lsm_delete, lsm_delete_range, cursors...). It inserts data into the database to a certain point (~2 million entries) and then it deletes, and add data but without going beyond that threshold. I've tried running valgrind on it but it does not detect any memory leak, it seems as closing the database puts memory back to normal. Is there any know issue that fits this description? Thanks! _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
