Re: [sqlite] Custom VFSes and PENDING_BYTE position

2020-03-09 Thread Alexey Alyaev 
Hi D. Richard Hipp,

Thank you for quick response.

I did consider all options as it stands:

1) Skipping the lock bytes in our vfs. This becomes really messy very fast, as
   it is based on test_onefile vfs and both db and journal are part of the same
   file, and lock bytes may end up in any of 3 segments: db, free space, or
   journal.
   
   In addition we have logic to grow (during writes)/shrink (via PRAGMA) the
   file as necessary. Basically, really a lot of skipping logic in many places.
   
   And even if I'd go ahead and coded all this up, we'd end up breaking all our
   currently deployed applications for all our users, since the skipped offsets
   would contain valid pages.
   
2) Set new PENDING_BYTE via sqlite3_test_control(). This looked like an option
   at first, until I realized that the same value [sqlite3PendingByte]
   is used in both: in pager, and the os_* vfses.
   
   Since our VFS is based on test_onefile with db, free space, and journal all
   occupying one file, it becomes impossible to handle all 3 cases that could
   end up writing into the lock byte region.
   
   And again, modifying the PENDING_BYTE would break the application for all
   users, due to lock-byte page position change.

3) Which is what I considered as the most suitable solution in cases similar
   to ours (one file vfses), and the reason I decide to write to this list:
   
   To be able to set (via config), an offsets for pending byte region
   in os_* VFSes, but have no effect on the lock-byte page offset as seen by
   higher layers.
   
   Since os_* VFSes are pretty much generic and are used as auxilaries in many
   other "wrapper" VFSes, it would be logical to at least have an option to
   configure the pending byte region only for os_* VFSes.
   
In our case, having an option such as (3), we could move the pending byte
region into the head space (occurs before the database) of the file, which
contains sufficient space to house all the lock bytes and that always exists.

This would handle all possible scenarios with one config tweak, as well as not
break any current deployments.

For others, any custom VFS that allocates space before the database in file,
and use os_* VFSes as auxilary, the same option can be used to add a shift
to the pending byte region, aligning it correctly with the lock byte page
offset used by pager.

Ofcourse, I can patch a few lines in sqlite3 source that we use, but I just
wanted to let you guys know of a situation we ran into and had no real simple
way of going forward with, even though everything else seemed to be configurable
and flexible.

Thanks.

-- Alexey
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Custom VFSes and PENDING_BYTE position

2020-03-09 Thread Richard Hipp 
On 3/9/20, Alexey Alyaev  wrote:
>
> I have been working on a custom VFS project since 2017
>
> would it not be reasonable to allow setting a different
> lock byte offset just in windows/unix VFS, while keeping the lock-byte page
> position untouched?
>
> This would allow VFSes such as ours to move the lock-byte region used by
> default OS VFS, outside of the database segment, say into the head space
> which we have allocated and reserved for meta.

In the default implementation, the location of the PENDING_BYTE is set
by a global variable sqlite3PendingByte.  If you want to change that
global variable, you can.  Be warned, however, that changing the
location of the pendingByte results in an incompatible file format.
You will also run into severe problems (assertion-faults and/or
segfaults) if you set the pendingByte to some value that is not a
multiple of the page size or that overlaps with page 1 of the database
file.

Aside:  The PENDING_BYTE location can be adjusted in this way for
testing purposes.  Even release builds are able to move the
PENDING_BYTE (using the sqlite3_test_control() interface) since our
dogma is "fly what you test and test what you fly."

Maybe a better solution for you would be to modify your custom xRead()
and xWrite() routines so that the skip over the bytes in the range of
0x4000 through 0x4200.  If the offset is greater than or equal
to 0x4000 just add 512 to the offset before continuing.  Depending
on your circumstances, I suppose you would also have to do something
similar with xTruncate() and xFileSize().

-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users