Re: [sqlite] upstreaming Chromium patches for file handle passing support

[Paweł Hajdan , Jr .]

On Tue, Aug 31, 2010 at 19:51, Richard Hipp  wrote:

> My suggestion is that you make a copy of the os_unix.c source file (call it
> chromium_vfs.c or anything else that you like) and apply your edits to that
> copy.


FYI, that's what finally happened.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] upstreaming Chromium patches for file handle passing support

[Paweł Hajdan , Jr .]

On Tue, Aug 31, 2010 at 10:51, Richard Hipp  wrote:

> My suggestion is that you make a copy of the os_unix.c source file (call it
> chromium_vfs.c or anything else that you like) and apply your edits to that
> copy.  Since you started with an exact copy of the original os_unix.c
> source
> file, your copy will all the internal utilities and mechanisms as
> os_unix.c.  You can use all those internal mechanism as much as you want.
> Then compile your customized VFS separately and add it to SQLite at
> start-time using sqlite3_vfs_register().
>

I have done a proof-of-concept patch, attached to
https://bugs.webkit.org/show_bug.cgi?id=45416

Direct link:
https://bugs.webkit.org/attachment.cgi?id=66952=prettypatch

However, Dumitru Daniliuc (dumi) has noticed that we could still reuse large
portions of os_unix.c and only provide our own implementation for opening a
file (instead of calling "open" directly, it would call
"ChromiumBridge::databaseOpenFile". Similarly for database deletion, etc.

Now that we have the code to talk about (see above), could you take another
look and see if it would be possible to avoid copy-pasting parts of
os_unix.c? I'm just trying to explore all possibilities we have - both
copying the code (done now), and re-using more of os_unix.c code.

On Wed, Sep 1, 2010 at 21:33, Dan Kennedy  wrote:

> How do you know which journal file to open or where to create
> a master journal file if the client writes to the database?


It is hard for me to answer this question (I'm not very familiar with sqlite
yet). See the patch above - I think it preserves the semantics of current
code used in Chrome, just moves it to WebKit. Any feedback on whether that
code is right or wrong and how it can be improved would be appreciated.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] upstreaming Chromium patches for file handle passing support

[Shawn Wilsher]

On Wed, Sep 1, 2010 at 5:57 PM, Paweł Hajdan, Jr.
 wrote:
> Additionally, browsers seem to be moving to the multi-process architecture
> (Chrome, Firefox, WebKit2), so I wouldn't be surprised if you get more
> questions about this in the future, or just more forked copies.
FWIW, this won't be a concern for Mozilla.

Cheers,

Shawn
Mozilla Developer
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] upstreaming Chromium patches for file handle passing support

[Scott Hess]

On Wed, Sep 1, 2010 at 10:20 PM, Max Vlasov  wrote:
>> I wonder whether it would be possible to extend the VFS in a way that would
>> make our use case possible (transferring a file handle over process
>> boundary). Please note that we do it on all platforms. On POSIX we pass an
>> integer file descriptor, and on Windows we pass a HANDLE.
>>
>>
> I used vfs for encryption, but there were not so many changed so I may be
> wrong, but ...
>
> It seems that a filename is known entity for vfs implementation, but for
> sqlite level is an abstraction (except for adding sub-extensions for special
> files). So you could fully replace vfs with yours and invent your own
> semantic and recognize it when file open request goes to you, for example
> for opening main db file ("filehandle:1234"). In this case you probably will
> get a request for journal like "filehandle:1234.db-journal" and this would
> mean that you should create some place you prefer a temporal file. I know it
> looks a little stange, but can someone confirm this approach at least works?

I was going to suggest that, too, but it doesn't really resolve the
problem, it just shifts it around.  When the module tries to open(2)
or unlink(2) the handle, it won't work because it's not a real file.
If those functions could be overriden without modifying anything else,
then they could be passed across to a broker ... but in that case you
could just use the original path directly rather than encoding things.
 Basically this code wants to have separate abstractions for handling
paths versus handling file handles.

Actually ... on Linux the file descriptor could be setup and then you
could rewrite the pathname to open the fd in /proc.  Obvious issues
with that for unlink(), but maybe that could be worked around by using
a persistent journal file.  I don't think OSX or Windows have a
similar abstraction which could be used directly, though.

I kind of wonder if something like sqlite/ext/async couldn't be used,
completely delegating everything to the broker.

-scott
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] upstreaming Chromium patches for file handle passing support

[Max Vlasov]

> I wonder whether it would be possible to extend the VFS in a way that would
> make our use case possible (transferring a file handle over process
> boundary). Please note that we do it on all platforms. On POSIX we pass an
> integer file descriptor, and on Windows we pass a HANDLE.
>
>
I used vfs for encryption, but there were not so many changed so I may be
wrong, but ...

It seems that a filename is known entity for vfs implementation, but for
sqlite level is an abstraction (except for adding sub-extensions for special
files). So you could fully replace vfs with yours and invent your own
semantic and recognize it when file open request goes to you, for example
for opening main db file ("filehandle:1234"). In this case you probably will
get a request for journal like "filehandle:1234.db-journal" and this would
mean that you should create some place you prefer a temporal file. I know it
looks a little stange, but can someone confirm this approach at least works?

Max
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] upstreaming Chromium patches for file handle passing support

[Dan Kennedy]

>
> What do you think about extending the SQLite VFS to make it possible  
> to open
> a database having a file handle (fd on Unix, HANDLE on Windows)?  
> Opening
> based on file path would still be there (to preserve compatibility).

How do you know which journal file to open or where to create
a master journal file if the client writes to the database?

Dan.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] upstreaming Chromium patches for file handle passing support

[Paweł Hajdan , Jr .]

2010/8/31 Richard Hipp 

> os_unix.c is very unlikely to change in ways that you care about.


I think the biggest concern is just copying that code, please see
http://groups.google.com/a/chromium.org/group/chromium-dev/browse_thread/thread/7f8e9bf0f034322f/b26e422d4aa711c7.
I think that keeping it in sync is a relatively minor concern, given
that
we update SQLite so rarely.


> The VFS is a well-defined and supported interface for SQLite.  We maintain
> backwards compatibility.  So even if you had been using an older custom VFS
> built using an older 3.6.23-or-earlier os_unix.c, it will still continue
> working on newer versions of SQLite - you just wouldn't be able to take
> advantage of the enhanced write-ahead log feature.
>

I wonder whether it would be possible to extend the VFS in a way that would
make our use case possible (transferring a file handle over process
boundary). Please note that we do it on all platforms. On POSIX we pass an
integer file descriptor, and on Windows we pass a HANDLE.

Additionally, browsers seem to be moving to the multi-process architecture
(Chrome, Firefox, WebKit2), so I wouldn't be surprised if you get more
questions about this in the future, or just more forked copies.

What do you think about extending the SQLite VFS to make it possible to open
a database having a file handle (fd on Unix, HANDLE on Windows)? Opening
based on file path would still be there (to preserve compatibility).


> The only thing that might trip you up is a bug fix in os_unix.c.  Those are
> very, very uncommon.  In fact, I can't call to mind the last time we had
> any
> serious bug in os_unix.c.


My biggest concern in this thread is a nice design. We have to bundle SQLite
in Chromium repository anyway because non-Linux platforms usually don't
provide SQLite, or have an older version, etc. Updating a custom-patched
os_unix.c (our current situation) is equally difficult to updating a copied
and patched os_unix.c, in case we have to apply a bugfix.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] upstreaming Chromium patches for file handle passing support

[Richard Hipp]

On Tue, Aug 31, 2010 at 1:56 PM, Paweł Hajdan, Jr.
wrote:

> On Tue, Aug 31, 2010 at 10:51, Richard Hipp  wrote:
>
> > My suggestion is that you make a copy of the os_unix.c source file (call
> it
> > chromium_vfs.c or anything else that you like) and apply your edits to
> that
> > copy.  Since you started with an exact copy of the original os_unix.c
> > source
> > file, your copy will all the internal utilities and mechanisms as
> > os_unix.c.  You can use all those internal mechanism as much as you want.
> > Then compile your customized VFS separately and add it to SQLite at
> > start-time using sqlite3_vfs_register().
> >
> > Why won't that work for you?
>
>
> It effectively forks os_unix.c, which makes upgrades of SQLite equally
> painful to the current state. The above solution offers no advantage over
> the current state of our bundled SQLite copy. I'd like to stay as close to
> upstream as possible, because then we can update very easily.
>

os_unix.c is very unlikely to change in ways that you care about.  It has
not changed in years in ways that would effect you, except for the addition
of new methods to support WAL for the 3.6.23->3.7.0 transition.  Such
changes are very uncommon.  You can continue to pull in newer versions of
SQLite without having to modify your customized chromium VFS.  If you are
ever in doubt about whether or not it is safe to continue to use an old VFS
based on an older os_unix.c, then simply ask.

The VFS is a well-defined and supported interface for SQLite.  We maintain
backwards compatibility.  So even if you had been using an older custom VFS
built using an older 3.6.23-or-earlier os_unix.c, it will still continue
working on newer versions of SQLite - you just wouldn't be able to take
advantage of the enhanced write-ahead log feature.

The same is true moving forward.  If we ever make changes to the VFS
interface, your older implementation will still work.  You just might not be
able to use it with some new feature enhancement or something.

The only thing that might trip you up is a bug fix in os_unix.c.  Those are
very, very uncommon.  In fact, I can't call to mind the last time we had any
serious bug in os_unix.c.




> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] upstreaming Chromium patches for file handle passing support

[Paweł Hajdan , Jr .]

On Tue, Aug 31, 2010 at 10:51, Richard Hipp  wrote:

> My suggestion is that you make a copy of the os_unix.c source file (call it
> chromium_vfs.c or anything else that you like) and apply your edits to that
> copy.  Since you started with an exact copy of the original os_unix.c
> source
> file, your copy will all the internal utilities and mechanisms as
> os_unix.c.  You can use all those internal mechanism as much as you want.
> Then compile your customized VFS separately and add it to SQLite at
> start-time using sqlite3_vfs_register().
>
> Why won't that work for you?


It effectively forks os_unix.c, which makes upgrades of SQLite equally
painful to the current state. The above solution offers no advantage over
the current state of our bundled SQLite copy. I'd like to stay as close to
upstream as possible, because then we can update very easily.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] upstreaming Chromium patches for file handle passing support

[Richard Hipp]

On Tue, Aug 31, 2010 at 1:19 PM, Paweł Hajdan, Jr.
wrote:

> On Fri, Aug 27, 2010 at 18:27, Paweł Hajdan, Jr.  >wrote:
>
> >
> >
> >> At start-time, simply call sqlite3_vfs_register() to register your
> >> customized VFS with SQLite.
> >>
> >
> > That's what our implementation effectively does, see
> >
> http://trac.webkit.org/browser/trunk/WebCore/platform/sql/chromium/SQLiteFileSystemChromiumPosix.cpp
> >  .
> >
> > Would it be possible to add a variant of the unix vfs inside of
> os_unix.c?
> > I'm still not sure if the current VFS interface would be sufficient for
> > that. It seems that the most needed exposed call is ability to transform
> a
> > file path into a file descriptor
> > (chromium_sqlite3_get_reusable_file_handle).
> chromium_sqlite3_get_reusable_file_handle
> > currently needs access to things like unixFile, findReusableFd,
> > UnixUnusedFd, and sqlite3_malloc. Would you have any suggestions how to
> fit
> > that into the SQLite's VFS implementation?
> >
>
> Let me clarify what the current implementation in Chrome and WebKit does.
> I'd appreciate suggestions how to fit it into the SQLite API, and I'd be
> glad to do the necessary work.
>
> So let's start with
>
> http://trac.webkit.org/browser/trunk/WebCore/platform/sql/chromium/SQLiteFileSystemChromiumPosix.cpp
> .
> In SQLiteFileSystem::registerSQLiteVFS (line 171), we register our own
> SQLite VFS ("chromium_vfs") mostly based on the Unix VFS with a few
> overrides.
>

My suggestion is that you make a copy of the os_unix.c source file (call it
chromium_vfs.c or anything else that you like) and apply your edits to that
copy.  Since you started with an exact copy of the original os_unix.c source
file, your copy will all the internal utilities and mechanisms as
os_unix.c.  You can use all those internal mechanism as much as you want.
Then compile your customized VFS separately and add it to SQLite at
start-time using sqlite3_vfs_register().

Why won't that work for you?



>
> Now please take a look at chromiumOpen (line 62). This code runs in the
> sandboxed renderer process, and does not have direct access to the
> filesystem.   It uses ChromiumBridge to communicate with the outside world
> (it communicates with the main browser process via IPC, and the browser
> process acts as a broker). So as you can see the renderer sends most of the
> details to the browser process (which runs with full user's rights), and
> gets back a file descriptor.
>
> And the hard part now is that we need to pass that file descriptor to
> SQLite. However, the public SQLite APIs all seem to be path-based. However,
> the internals of os_unix.c have definitions of unixFile, findReusableFd,
> UnixUnusedFd that allow us to make SQLite use the file descriptor we got.
>
> Please note that we get the file descriptor via ChromiumBridge, a class
> that
> lives in WebKit. This makes it non-trivial to put the entire implementation
> of "chromium_vfs" in SQLite, because we need to ask the browser process for
> the file descriptor. And we can't have the entire implementation on the
> WebKit/Chrome side either, because then we can't compile against vanilla
> SQLite (and Linux distributions don't like it).
>
> Could you suggest a better way to do the above, that would fit better
> within
> SQLite's design? Please let me know if you have more questions about how we
> currently use SQLite, the multi-process architecture of Chrome, etc.
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] upstreaming Chromium patches for file handle passing support

[Pavel Ivanov]

> And we can't have the entire implementation on the
> WebKit/Chrome side either, because then we can't compile against vanilla
> SQLite (and Linux distributions don't like it).

Could you explain in detail where this "can't" comes from and what's
the problem with Richard's suggestion to copy necessary pieces of code
from os_unix.c to your SQLiteFileSystemChromiumPosix.cpp? If you are
just scared of words "copy-paste the code" then think of it as if you
write your own implementation of VFS, it just accidentally happened
that in many places it looks very similar as os_unix.c. There's no
other way to perfectly fit into SQLite's paradigm and to be compiled
and work with vanilla SQLite.


Pavel

On Tue, Aug 31, 2010 at 1:19 PM, Paweł Hajdan, Jr.
 wrote:
> On Fri, Aug 27, 2010 at 18:27, Paweł Hajdan, Jr. 
> wrote:
>
>>
>>
>>> At start-time, simply call sqlite3_vfs_register() to register your
>>> customized VFS with SQLite.
>>>
>>
>> That's what our implementation effectively does, see
>> http://trac.webkit.org/browser/trunk/WebCore/platform/sql/chromium/SQLiteFileSystemChromiumPosix.cpp
>>  .
>>
>> Would it be possible to add a variant of the unix vfs inside of os_unix.c?
>> I'm still not sure if the current VFS interface would be sufficient for
>> that. It seems that the most needed exposed call is ability to transform a
>> file path into a file descriptor
>> (chromium_sqlite3_get_reusable_file_handle). 
>> chromium_sqlite3_get_reusable_file_handle
>> currently needs access to things like unixFile, findReusableFd,
>> UnixUnusedFd, and sqlite3_malloc. Would you have any suggestions how to fit
>> that into the SQLite's VFS implementation?
>>
>
> Let me clarify what the current implementation in Chrome and WebKit does.
> I'd appreciate suggestions how to fit it into the SQLite API, and I'd be
> glad to do the necessary work.
>
> So let's start with
> http://trac.webkit.org/browser/trunk/WebCore/platform/sql/chromium/SQLiteFileSystemChromiumPosix.cpp.
> In SQLiteFileSystem::registerSQLiteVFS (line 171), we register our own
> SQLite VFS ("chromium_vfs") mostly based on the Unix VFS with a few
> overrides.
>
> Now please take a look at chromiumOpen (line 62). This code runs in the
> sandboxed renderer process, and does not have direct access to the
> filesystem.   It uses ChromiumBridge to communicate with the outside world
> (it communicates with the main browser process via IPC, and the browser
> process acts as a broker). So as you can see the renderer sends most of the
> details to the browser process (which runs with full user's rights), and
> gets back a file descriptor.
>
> And the hard part now is that we need to pass that file descriptor to
> SQLite. However, the public SQLite APIs all seem to be path-based. However,
> the internals of os_unix.c have definitions of unixFile, findReusableFd,
> UnixUnusedFd that allow us to make SQLite use the file descriptor we got.
>
> Please note that we get the file descriptor via ChromiumBridge, a class that
> lives in WebKit. This makes it non-trivial to put the entire implementation
> of "chromium_vfs" in SQLite, because we need to ask the browser process for
> the file descriptor. And we can't have the entire implementation on the
> WebKit/Chrome side either, because then we can't compile against vanilla
> SQLite (and Linux distributions don't like it).
>
> Could you suggest a better way to do the above, that would fit better within
> SQLite's design? Please let me know if you have more questions about how we
> currently use SQLite, the multi-process architecture of Chrome, etc.
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] upstreaming Chromium patches for file handle passing support

[Paweł Hajdan , Jr .]

On Fri, Aug 27, 2010 at 18:27, Paweł Hajdan, Jr. wrote:

>
>
>> At start-time, simply call sqlite3_vfs_register() to register your
>> customized VFS with SQLite.
>>
>
> That's what our implementation effectively does, see
> http://trac.webkit.org/browser/trunk/WebCore/platform/sql/chromium/SQLiteFileSystemChromiumPosix.cpp
>  .
>
> Would it be possible to add a variant of the unix vfs inside of os_unix.c?
> I'm still not sure if the current VFS interface would be sufficient for
> that. It seems that the most needed exposed call is ability to transform a
> file path into a file descriptor
> (chromium_sqlite3_get_reusable_file_handle). 
> chromium_sqlite3_get_reusable_file_handle
> currently needs access to things like unixFile, findReusableFd,
> UnixUnusedFd, and sqlite3_malloc. Would you have any suggestions how to fit
> that into the SQLite's VFS implementation?
>

Let me clarify what the current implementation in Chrome and WebKit does.
I'd appreciate suggestions how to fit it into the SQLite API, and I'd be
glad to do the necessary work.

So let's start with
http://trac.webkit.org/browser/trunk/WebCore/platform/sql/chromium/SQLiteFileSystemChromiumPosix.cpp.
In SQLiteFileSystem::registerSQLiteVFS (line 171), we register our own
SQLite VFS ("chromium_vfs") mostly based on the Unix VFS with a few
overrides.

Now please take a look at chromiumOpen (line 62). This code runs in the
sandboxed renderer process, and does not have direct access to the
filesystem.   It uses ChromiumBridge to communicate with the outside world
(it communicates with the main browser process via IPC, and the browser
process acts as a broker). So as you can see the renderer sends most of the
details to the browser process (which runs with full user's rights), and
gets back a file descriptor.

And the hard part now is that we need to pass that file descriptor to
SQLite. However, the public SQLite APIs all seem to be path-based. However,
the internals of os_unix.c have definitions of unixFile, findReusableFd,
UnixUnusedFd that allow us to make SQLite use the file descriptor we got.

Please note that we get the file descriptor via ChromiumBridge, a class that
lives in WebKit. This makes it non-trivial to put the entire implementation
of "chromium_vfs" in SQLite, because we need to ask the browser process for
the file descriptor. And we can't have the entire implementation on the
WebKit/Chrome side either, because then we can't compile against vanilla
SQLite (and Linux distributions don't like it).

Could you suggest a better way to do the above, that would fit better within
SQLite's design? Please let me know if you have more questions about how we
currently use SQLite, the multi-process architecture of Chrome, etc.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] upstreaming Chromium patches for file handle passing support

[Paweł Hajdan , Jr .]

On Fri, Aug 27, 2010 at 04:58, Richard Hipp  wrote:

> The VFS interface is published, stable, and documented.  I think the VFS is
> the interface you should be using.
>
> You do not have to patch os_unix.c.  Leave it unchanged.


Sounds good to me. This is one of my goals - to avoid those custom patches,
or send them upstream.


> Instead, create your own os_chromium.c based off of your patched os_unix.c
> (changing just a
> few identifiers to avoid conflicts) and link it with your process as a
> separate file.


I think this is the non-trivial step. Those chromium_sqlite3 functions use
some symbols that are declared static in os_unix.c, so they're not available
outside of it. That means we would have to copy a lot of os_unix.c file's
contents, and I'd prefer to avoid that.


> (You really should be using the SQLite amalgamation file
> sqlite3.c - not separate source files - since the amalgamation runs about
> 5%
> or 10% faster due to better compiler optimizations.)


At least currently Chromium isn't going to use the amalgamation because we
have a lot of custom patches. I'm trying to clean up this situation though.


> At start-time, simply call sqlite3_vfs_register() to register your
> customized VFS with SQLite.
>

That's what our implementation effectively does, see
http://trac.webkit.org/browser/trunk/WebCore/platform/sql/chromium/SQLiteFileSystemChromiumPosix.cpp
 .

Would it be possible to add a variant of the unix vfs inside of os_unix.c?
I'm still not sure if the current VFS interface would be sufficient for
that. It seems that the most needed exposed call is ability to transform a
file path into a file descriptor
(chromium_sqlite3_get_reusable_file_handle).
chromium_sqlite3_get_reusable_file_handle
currently needs access to things like unixFile, findReusableFd,
UnixUnusedFd, and sqlite3_malloc. Would you have any suggestions how to fit
that into the SQLite's VFS implementation?
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] upstreaming Chromium patches for file handle passing support

[Richard Hipp]

On Thu, Aug 26, 2010 at 6:02 PM, Paweł Hajdan, Jr.
wrote:

> Please take a look at chromium_sqlite3 functions in
>
> http://src.chromium.org/viewvc/chrome/trunk/src/third_party/sqlite/src/src/os_unix.c?view=markup
>
> They are needed because in Chrome the browser process will pass a file
> descriptor to the child renderer process instead of a file path. Here's the
> code that handles it on the renderer side:
>
> http://trac.webkit.org/browser/trunk/WebCore/platform/sql/chromium/SQLiteFileSystemChromiumPosix.cpp
>
> Currently those chromium_sqlite3 functions live in patched os_unix.c file
> because they access slite internals that are only visible in os_unix.c.
>
> Could you suggest a way to do the same thing in a way that wouldn't require
> custom patches on the Chromium project side? If we can get a similar
> interface exposed in sqlite that'd be great. Another possible solution
> would
> be to allow those chromium_sqlite3 functions to be implemented outside of
> os_unix.c file (that will probably require exposing more internals from
> that
> file).
>

The VFS interface is published, stable, and documented.  I think the VFS is
the interface you should be using.

You do not have to patch os_unix.c.  Leave it unchanged.  Instead, create
your own os_chromium.c based off of your patched os_unix.c (changing just a
few identifiers to avoid conflicts) and link it with your process as a
separate file.  (You really should be using the SQLite amalgamation file
sqlite3.c - not separate source files - since the amalgamation runs about 5%
or 10% faster due to better compiler optimizations.)  At start-time, simply
call sqlite3_vfs_register() to register your customized VFS with SQLite.
Then use sqlite3_open_v2() to specify your custom VFS implementation.  Or
make your custom VFS implementation the default when you register it.

Another option is for you to append your custom routines to the end of
sqlite3.c.

cat sqlite3.c chromium_vfs.c >chromium_sqlite3.c

That why, you custom routine will have access to all of the SQLite
internals.  Of course, those internals will change from time to time.  We do
not gratuitously change things, but every not and then we do refactor
interfaces to add new features or to make them more efficient.  So the
custom code you append would need to track those changes.

If we were to expose the interfaces that you are using, that means we would
have to lock them down and support them forever.  That would preclude many
future enhancements and is something we are very reluctant to do.



> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] upstreaming Chromium patches for file handle passing support

[Paweł Hajdan , Jr .]

Please take a look at chromium_sqlite3 functions in
http://src.chromium.org/viewvc/chrome/trunk/src/third_party/sqlite/src/src/os_unix.c?view=markup

They are needed because in Chrome the browser process will pass a file
descriptor to the child renderer process instead of a file path. Here's the
code that handles it on the renderer side:
http://trac.webkit.org/browser/trunk/WebCore/platform/sql/chromium/SQLiteFileSystemChromiumPosix.cpp

Currently those chromium_sqlite3 functions live in patched os_unix.c file
because they access slite internals that are only visible in os_unix.c.

Could you suggest a way to do the same thing in a way that wouldn't require
custom patches on the Chromium project side? If we can get a similar
interface exposed in sqlite that'd be great. Another possible solution would
be to allow those chromium_sqlite3 functions to be implemented outside of
os_unix.c file (that will probably require exposing more internals from that
file).
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users