> I've also attached results of "EXPLAIN UPDATE ITEM SET FIELD4 = 10 WHERE DS = 15;".
Not zipvfs specific, but that kind of update can be quite inefficient if the record is large (in terms of bytes, or records per database page) as your table declaration hints to. This will be especially prominent with compression, as a single byte changed early in the uncompressed stream can lead to changing pretty much every byte of the compressed data, and so is bound to defeat any optimizations the storage could have to handle unmodified pages and data. Depending on your update patterns and frequencies, it could thus be beneficial to split your table into two tables (or more) with a common primary key: - one table for the rarely changed fields (even better if they are immutable) - one table (or more) for the fields that change often This will increase the database size and reduce insert performance, and it can complicate querying when you need everything (extra inner joins), but it can help reduce the amount of raw data that is affected by updates quite drastically. I have been able to use the above strategy sucessfully, however it was under Windows compressed folders, so YMMV. On Thu, Sep 7, 2017 at 9:13 PM, Yue Wu <yue...@datascan.com> wrote: > Hello, > As mentioned in the subject, our goal is to improve performance regarding > to batch sql updates. > The update sql as follow, > > > UPDATE ITEM SET FIELD4 =? WHERE DS=? > > We run 100,000 updates in a single transaction. The zipvfs version takes > about 20 min while uncompressed version takes about 7 min. > I've also attached results of "EXPLAIN UPDATE ITEM SET FIELD4 = 10 WHERE DS > = 15;". > > Any suggestions is very helpful. > > Some background: > We have an app running on Android 4.2 using zipvfs. > The page size is 4096 > The cache size is - 2000 > > Table schema for ITEM > > > CREATE TABLE ITEM ( > > FIELD0 NUMERIC, > > FIELD1 NUMERIC, > > DS TEXT, > > FIELD2 TEXT, > > FIELD3 TEXT, > > FIELD4 NUMERIC, > > FIELD5 NUMERIC, > > FIELD6 NUMERIC, > > FIELD7 NUMERIC, > > FIELD8 NUMERIC, > > FIELD9 NUMERIC, > > FIELD10 NUMERIC, > > FIELD11 TEXT); > > > The third column: "DS" is what we query by almost all the time. We also > created index: > > > CREATE INDEX DS_INDEX ON ITEM(DS); > > > There are 30 million records. Zipvfs size of 471mb vs uncompressed db (the > one shipped with Android 4.2) size of 1.39gb. > Zipvfs db using zlib and aes128, which are default. > > Thanks > -- > > Yue Wu | Android Developer > > _______________________________________________ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > > _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
