Re: [sqlite] Status of LSM1 extension
On Thursday, 5 March, 2020 20:39, Charles Leifer wrote: >Keith, if you could share a bit more details on how you do that, I'd be >interested. I presume you mean how to create a "built-in" extension, which is available for all connections, just the built-in functions and modules. There is a built-in define SQLITE_EXTRA_INIT used in main.c which allows for a "user function" to be called at the end of the SQLite3 initialization process. You compile the code/amalgamation with SQLITE_EXTRA_INIT defined to the name of a function that takes a NULL parameter and returns an SQLITE error code (SQLITE_OK, SQLITE_ERROR, etc). This function is called *after* the SQLite3 core is initialized. It does additional initialization of the library. For example, to autoload some extensions on every connection you can append the code for the extension to the amalgamation sqlite3.c file and then append you EXTRA_INIT function to add the extension init procedure to the auto extension list: -DSQLITE_EXTRA_INIT=coreinit static int coreinit(const char* dummy) ( return sqlite3_auto_extension((void*)sqlite3_series_init); } If you append series.c to the amalgamation, and then the coreinit function, and compile with -DSQLITE_EXTRA_INIT=coreinit, then when SQLite3 is initialized the coreinit function will run after the initialization is complete and add the sqlite3_series_init function to the auto_extension list. Then when any new connection is opened the generate_series extension will be registered on that connection. Of course, the extensions and the coreinit function do not have to be part of the amalgamation compilation unit -- they can be in a separate file and statically linked. -- The fact that there's a Highway to Hell but only a Stairway to Heaven says a lot about anticipated traffic volume. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Status of LSM1 extension
Keith, if you could share a bit more details on how you do that, I'd be interested. On Thu, Mar 5, 2020 at 8:43 AM Keith Medcalf wrote: > > On Thursday, 5 March, 2020 05:51, Dominique Devienne > wrote: > > >PS: I'd still very much appreciate an LSM1 amalgamation > > cd ext/lsm1 > tclsh tool/mklsm1c.tcl > > which will write an lsm1.c amalgamation in the current directory (ext/lsm1) > > You can append this to the amalgamation and use an EXTRA_INIT hook to > initialize it, just like building in any other extension (though you need > to define SQLITE_ENABLE_LSM1 in order for the extension code to be compiled) > > -- > The fact that there's a Highway to Hell but only a Stairway to Heaven says > a lot about anticipated traffic volume. > > > > > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Status of LSM1 extension
On Thursday, 5 March, 2020 05:51, Dominique Devienne wrote: >PS: I'd still very much appreciate an LSM1 amalgamation cd ext/lsm1 tclsh tool/mklsm1c.tcl which will write an lsm1.c amalgamation in the current directory (ext/lsm1) You can append this to the amalgamation and use an EXTRA_INIT hook to initialize it, just like building in any other extension (though you need to define SQLITE_ENABLE_LSM1 in order for the extension code to be compiled) -- The fact that there's a Highway to Hell but only a Stairway to Heaven says a lot about anticipated traffic volume. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Status of LSM1 extension
On Thu, Mar 5, 2020 at 12:35 PM Dan Kennedy wrote: > On 5/3/63 16:11, Dominique Devienne wrote: > > I'm interested in LSM1 [1] as an alternative to SQLite [...] > > [...], I don't think it's too bad of an implementation. The > automated tests are reasonably good - although of course not as good as > SQLite's though. And the docs are stored in kind of a ridiculous place > at the moment, but I think they're quite complete. > > Not planning to develop this any further unless a big user emerges, > which is not impossible. I do intend to fix any reported bugs though. Thanks Dan. I appreciate your candor, and support commitment. Given the above, I'll start with SQLite, and when/if I'm done with what I need to do, will try to give LSM1 a try, to compare performance, and report back. Thanks again, --DD PS: I'd still very much appreciate an LSM1 amalgamation ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Status of LSM1 extension
On 5/3/63 16:11, Dominique Devienne wrote: Hi, I'm interested in LSM1 [1] as an alternative to SQLite, since in a particular use-case, I'm using SQLite mostly as a key-value store, and write performance is particularly important, in addition to MVCC. Sounds like it could be an excellent fit here, and the fact it comes from the SQLite team is something I value. That said, the only online doc for LSM1 ([2] and [3]) are from the defunct SQLite4 web-site, and the main blog post is starting to look dated [4]. I thus wonder about the level of quality and support on LSM1, and lack of doc for it in the main SQLite web-site. In terms of practicality, there's also no amalgamation for LSM1. And the virtual table over LSM1 data-files [5], something I was look for, does not appear to be documented anywhere. Notably whether using that vtable using the familiar SQLite API is advisable instead of using the different and unfamiliar LSM1-specific API. I'm just looking for clarity and advice around LSM1, as well as commitments regarding its level of quality, testing, and support. And whether we can hope to have more doc and amalgamation deliverables in the future. It sounds like it's a really nice piece of code, but the fact there's very little noise and advertisement about it is somewhat worrying. It's very much a solution looking for a problem at this point. It's not, as far as I know, used in anything that is too widely deployed. That said, I don't think it's too bad of an implementation. The automated tests are reasonably good - although of course not as good as SQLite's though. And the docs are stored in kind of a ridiculous place at the moment, but I think they're quite complete. Not planning to develop this any further unless a big user emerges, which is not impossible. I do intend to fix any reported bugs though. Dan. Thanks, --DD [1] https://www2.sqlite.org/src/dir?name=ext/lsm1 [2] https://sqlite.org/src4/doc/trunk/www/lsmusr.wiki [3] https://sqlite.org/src4/doc/trunk/www/lsmapi.wiki [4] https://charlesleifer.com/blog/lsm-key-value-storage-in-sqlite3/ [5] https://www2.sqlite.org/src/finfo?name=ext/lsm1/lsm_vtab.c ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Status of LSM1 extension
Hi, I'm interested in LSM1 [1] as an alternative to SQLite, since in a particular use-case, I'm using SQLite mostly as a key-value store, and write performance is particularly important, in addition to MVCC. Sounds like it could be an excellent fit here, and the fact it comes from the SQLite team is something I value. That said, the only online doc for LSM1 ([2] and [3]) are from the defunct SQLite4 web-site, and the main blog post is starting to look dated [4]. I thus wonder about the level of quality and support on LSM1, and lack of doc for it in the main SQLite web-site. In terms of practicality, there's also no amalgamation for LSM1. And the virtual table over LSM1 data-files [5], something I was look for, does not appear to be documented anywhere. Notably whether using that vtable using the familiar SQLite API is advisable instead of using the different and unfamiliar LSM1-specific API. I'm just looking for clarity and advice around LSM1, as well as commitments regarding its level of quality, testing, and support. And whether we can hope to have more doc and amalgamation deliverables in the future. It sounds like it's a really nice piece of code, but the fact there's very little noise and advertisement about it is somewhat worrying. Thanks, --DD [1] https://www2.sqlite.org/src/dir?name=ext/lsm1 [2] https://sqlite.org/src4/doc/trunk/www/lsmusr.wiki [3] https://sqlite.org/src4/doc/trunk/www/lsmapi.wiki [4] https://charlesleifer.com/blog/lsm-key-value-storage-in-sqlite3/ [5] https://www2.sqlite.org/src/finfo?name=ext/lsm1/lsm_vtab.c ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
