Re: [sqlite] adding fdopen to VFS?

[Roger Binns]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 03/01/2011 01:07 AM, Philip Graham Willoughby wrote:
> I argue that this implies a problem in the Chromium sandbox rather than a 
> problem in the SQLite code.

SQLite's implementation makes things worse because the xOpen implementation
is not 5 lines of code wrapping a call to open() but considerably more
meaning that overriding it is a lot of work.

As for the sandbox, the traditional Unix mechanism for doing that has been
chroot which works well for daemons, but is too hard to use for something
like the Flash player or webkit renderer.

Some docs:

  http://lwn.net/Articles/347547/
  http://code.google.com/p/seccompsandbox/wiki/overview

Roger
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1tLiMACgkQmOOfHg372QQlxgCgjEyFPKg8KT2riZitw6hcX/78
vAwAoNo3h1y6exbGpmIy3eIHIvNGVStO
=GvCD
-END PGP SIGNATURE-
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] adding fdopen to VFS?

[Philip Graham Willoughby]

On 28 Feb 2011, at 20:38, Roger Binns wrote:

>> Why is it not possible to create a custom VFS for the intended purpose
>> and register it for the connections that need it?
> 
> Because that custom VFS would be an almost duplicate of the existing VFS but
> with a few key places changed.
> 
> If the changes were at the granularity of the VFS methods then it would be
> no problem to "inherit" as needed.  Unfortunately they are within - for
> example wanting to use the xOpen method but changing only the open() call
> within.

I argue that this implies a problem in the Chromium sandbox rather than a 
problem in the SQLite code. Either the Chromium sandbox does not allow files to 
be opened at all (in which case you can't have SQLite) or it does. If it does, 
why is the routine that allows you to do so named something other than open()?

If you cannot open files but can ask a supervisor to open files on your behalf, 
the natural thing to do is to write a routine named open() that invokes the 
supervisor and give it the semantics of POSIX open(). Similarly for close() 
stat() and the various locking calls.

Not only would this make SQLite work unmodified but it would make the next n 
things that have a similar issue work for free as well.

Best Regards,

Phil Willoughby
-- 
Managing Director, StrawberryCat Limited

StrawberryCat Limited is registered in England and Wales with Company No. 
7234809.

The registered office address of StrawberryCat Limited is:

107 Morgan Le Fay Drive
Eastleigh
SO53 4JH

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] adding fdopen to VFS?

[Roger Binns]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 02/28/2011 01:44 PM, Drake Wilson wrote:
> I'm inclined to believe this nullifies the main argument against
> forking os_unix.c in the absence of strong evidence to the contrary.

Here are some broken things in VFS of varying severity where fixing them may
involve changing the VFS code.  (Coincidentally I reported most of them.)

  http://www.sqlite.org/src/tktview?name=5e0423b058
  http://www.sqlite.org/src/tktview?name=c060923a54
  http://www.sqlite.org/src/tktview?name=d8fe6cef65
  http://www.sqlite.org/src/tktview?name=0b803bff85

> essentially "subclassing" the VFS.

That was also my initial reaction.  However it doesn't solve the problem.

In essence copying a few thousand lines of code and changing 3 of them feels
wrong.  Note that no one is disputing that it works.

> I haven't looked as closely as Pavel, Richard, et al. presumably
> have.

Exactly :-)

> filename-related and easily refactorable into a short stack of local
> functions plus wrappers for the methods that take a filename argument,

Go through the exercise of making a VFS where the only difference is that
the open() call buried inside unixOpen calls a different function.  Now make
sure it is tested, kept up to date, works with the platform SQLite etc.

It will work, but it is ugly.  The question is if there is a better way.

Roger
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1sHQMACgkQmOOfHg372QQQ7QCgjyLkJHIr3wpZOfa7ANehAn0u
hRkAoNS83bpoZRUoC7RZgdVIYMu0dKq9
=0SbV
-END PGP SIGNATURE-
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] adding fdopen to VFS?

[Drake Wilson]

Quoth Drake Wilson , on 2011-02-28 14:44:38 -0700:
> Furthermore, another approach if the name<->FD thing is the only
> requirement would be to retrieve all the original VFS methods at init
> time (using sqlite3_vfs_find) and only alter a few of them when
> registering the new one, essentially "subclassing" the VFS.  I haven't
> seen anything that would require changes except ones that are
> filename-related and easily refactorable into a short stack of local
> functions plus wrappers for the methods that take a filename argument,
> but I haven't looked as closely as Pavel, Richard, et al. presumably
> have.

Actually, wait a minute.  Here I was assuming there'd be a mechanism
similar to /dev/fd or /proc/self/fd on most of the Unixes supported,
per what Roger mentioned upthread.  But that may wind up broken if the
sandbox denies open() unilaterally, too.  Hmm.

I shouldn't try to do this before coffee...

   ---> Drake Wilson
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] adding fdopen to VFS?

[Drake Wilson]

Quoth Roger Binns , on 2011-02-28 13:03:43 -0800:
> On 02/28/2011 12:41 PM, Drake Wilson wrote:
> > Back on the original topic, I would rather think a custom VFS sounds
> > like the way to go;
> 
> It is technically correct that will work.  However it is a *lot* of
> maintenance work.
> 
> The custom VFS would be a duplicate of the platform one (lots of code) but
> with only a very small number of lines changed.

Perhaps, but DRH in the earlier thread "upstreaming Chromium patches
for file handle passing support" wrote:

| os_unix.c is very unlikely to change in ways that you care about.  It has
| not changed in years in ways that would effect you, except for the addition
| of new methods to support WAL for the 3.6.23->3.7.0 transition.  Such
| changes are very uncommon.  You can continue to pull in newer versions of
| SQLite without having to modify your customized chromium VFS.  If you are
| ever in doubt about whether or not it is safe to continue to use an old VFS
| based on an older os_unix.c, then simply ask.
[...]
| The only thing that might trip you up is a bug fix in os_unix.c.  Those are
| very, very uncommon.  In fact, I can't call to mind the last time we had any
| serious bug in os_unix.c.

I'm inclined to believe this nullifies the main argument against
forking os_unix.c in the absence of strong evidence to the contrary.

Furthermore, another approach if the name<->FD thing is the only
requirement would be to retrieve all the original VFS methods at init
time (using sqlite3_vfs_find) and only alter a few of them when
registering the new one, essentially "subclassing" the VFS.  I haven't
seen anything that would require changes except ones that are
filename-related and easily refactorable into a short stack of local
functions plus wrappers for the methods that take a filename argument,
but I haven't looked as closely as Pavel, Richard, et al. presumably
have.

The main tricky case is where POSIX advisory locking requires bogusly
keeping earlier file descriptors open, and that's the case if you open
the same underlying database file multiple times.  Are there comments
from the Chromium folks about whether multiple descriptors for the
same file might get passed down to the sandboxed process?  Could that
be reasonably prevented on the parent-link side somehow?

   ---> Drake Wilson
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] adding fdopen to VFS?

[Roger Binns]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 02/28/2011 12:41 PM, Drake Wilson wrote:
> Back on the original topic, I would rather think a custom VFS sounds
> like the way to go;

It is technically correct that will work.  However it is a *lot* of
maintenance work.

The custom VFS would be a duplicate of the platform one (lots of code) but
with only a very small number of lines changed.

Imagine you want your own VFS where the only difference is that open() in
the unixOpen implementation (235 lines plus others) is changed to myopen().
 If you provide your own libsqlite then this is doable at compile time.  If
you want to use the version supplied by the distro vendor (normal Linux
practise) then you are out of luck.

About the only solution that will work for everyone is the ability to pass
the VFS a table of function pointers that by default point to the operating
system implementations (eg open(), access(), getcwd()).  You can then
override those as needed.  This would require the VFS api be updated.

Alternatively Chromium can be changed so that it traces the child and
intercepts these calls returning data as appropriate.

If there wasn't the need to deal with additional filenames then (on Linux)
the filename could be passed as /proc/self/fd/NN.  The default unix vfs
could be changed such that filename pattern is recognized and NN+1 is
treated as the journal etc.

Roger
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1sDaoACgkQmOOfHg372QRDFACdEHjdrW8cEVB1lfKybxi+eL/g
+1YAn1W5TpEl+ASRVkqji7DFknWLcnGM
=DzRU
-END PGP SIGNATURE-
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] adding fdopen to VFS?

[Drake Wilson]

Quoth Roger Binns , on 2011-02-28 12:20:04 -0800:
> You could do that in Unix in many years.  I shipped product in 1994 that
> used it extensively (a master daemon would use accept() and then pass the
> new file handles to child processes as appropriate).
> 
> "I_SENDFD" is the magic needed.  The topic is even covered in Stevens.

SCM_RIGHTS is a modern (and, I think, the most usual) way, operating
over a Unix-domain socket.  The header constant, at least, is
specified in POSIX:2001, though I can't find a specification of its
functionality there.  On every Unix I've personally observed have it,
it transmits an array of FDs, which are duplicated into the receiving
process (and the numbers in the message translated accordingly).

http://www.opengroup.org/onlinepubs/95399/basedefs/sys/socket.h.html

The corresponding call on Windows is DuplicateHandle, which can copy a
handle from one process to another, and is AIUI generally paired with
other IPC to transmit the new handle value.

http://msdn.microsoft.com/en-us/library/bb202788.aspx

Back on the original topic, I would rather think a custom VFS sounds
like the way to go; one should be able to foo_snprintf the underlying
platform-specific handle into an open-string and then get it back out
inside the custom VFS, no?  There can be no incompatibility with the
existing expected uses of the open-string, because those will all be
broken in the use case described anyway.  If it's inconvenient, you
could provide your own chromium_sqlite3_openhandle(handle, ...)
function which would do the conversion and call sqlite3_open behind
the scenes.

   ---> Drake Wilson
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] adding fdopen to VFS?

[Roger Binns]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 02/28/2011 11:59 AM, Robert Hairgrove wrote:
> I'm only beginning to delve into the possibilities offered by the VFS
> API in SQLite. However, from what I have seen so far, it seems like a
> very complete abstraction to me.

For one thing a SQLite database at runtime is actually multiple related
files which makes things complicated.  Things like their names matter.  The
existing VFS implementation assumes you will be working with names while the
Chromium folk want a child process to not have permissions to open files and
instead have a supervisor process do so as appropriate.  The VFS API never
anticipated this split process model passing handles rather than file/pathnames.

> Why is it not possible to create a custom VFS for the intended purpose
> and register it for the connections that need it?

Because that custom VFS would be an almost duplicate of the existing VFS but
with a few key places changed.

If the changes were at the granularity of the VFS methods then it would be
no problem to "inherit" as needed.  Unfortunately they are within - for
example wanting to use the xOpen method but changing only the open() call
within.

unixOpen() is 235 lines of code plus several helper functions.  Having to
copy all that code just to change one call on one line is not sustainable in
the long term.

It would be possible to remote the VFS instead - ie have a an implementation
in the child process that just forwards all calls to the supervisor parent.
 If parent and child are using the same version of SQLite then this should
work well at the expense of adding latency.  It would however require the
parent to implement security and resource limiting.  The operating system
does that for "free" for child processes appropriately initialized.

Roger
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1sB70ACgkQmOOfHg372QTFkwCgioqwaibI+6WrMedG4L0KOPKJ
E5UAoLXUimQ3sLncdOiJL2Gbb+Nouvzb
=NonM
-END PGP SIGNATURE-
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] adding fdopen to VFS?

[Jay A. Kreibich]

On Mon, Feb 28, 2011 at 12:20:04PM -0800, Roger Binns scratched on the wall:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> On 02/28/2011 11:33 AM, Richard Hipp wrote:
> > and then sends over a file descriptor using IPC. 

> You could do that in Unix in many years.

  Apache 1.x depended on this for.. well, everything.

> The topic is even covered in Stevens.

  "Even" ?!?  Between TCP/IP Illustrated, Unix Network Programming, and
  Advanced Programming in the UNIX Environment, what the heck *isn't*
  covered in Stevens *somewhere*?

   -j

-- 
Jay A. Kreibich < J A Y  @  K R E I B I.C H >

"Intelligence is like underwear: it is important that you have it,
 but showing it to the wrong people has the tendency to make them
 feel uncomfortable." -- Angela Johnson
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] adding fdopen to VFS?

[Roger Binns]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 02/28/2011 11:33 AM, Richard Hipp wrote:
> and then sends
> over a file descriptor using IPC.  (I didn't know you could do that in
> unix.  In fact, I bet it requires a linux-only extension of some kind.
> Correct me if I'm wrong.)

You could do that in Unix in many years.  I shipped product in 1994 that
used it extensively (a master daemon would use accept() and then pass the
new file handles to child processes as appropriate).

"I_SENDFD" is the magic needed.  The topic is even covered in Stevens.

Roger
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1sA2sACgkQmOOfHg372QR4YQCfbf8a+qw2kK+ZXHf1A1vLb3Lh
lA0AoNQ0xZpnOtvg8xLzxtyCBthx5WCb
=DdoU
-END PGP SIGNATURE-
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] adding fdopen to VFS?

[Robert Hairgrove]

On Mon, 2011-02-28 at 14:33 -0500, Richard Hipp wrote:
> On Mon, Feb 28, 2011 at 1:00 PM, Paweł Hajdan, Jr.

[many interesting things snipped ...]

> wrote:
> > What do you think about adding a call like below to the VFS interface?
> >
> > int (*xOpenWrapper)(const char *path, int flags, int mode);
> >
> 
> There are 2 billion instances of SQLite in active use today.  We don't want
> to go changing interfaces unless we absolutely have to.
> 
> And there are additional complications.  Based on what we can figure out
> about how your sandbox works, you'll also need overloads for access(),
> stat(), and unlink().  Presumably you have no need for dlopen() which is
> used for run-time loadable extensions to SQLite, but if you use that
> features you'll need an overload for it too.  We might yet think of others,
> but already we see there are at least three interfaces (we could probably
> combine access and stat) that need to be overloaded.
> 
> We are also concerned about locking.  The documentation on Chromium
> sandboxing implies that a separate process does the open() and then sends
> over a file descriptor using IPC.  (I didn't know you could do that in
> unix.  In fact, I bet it requires a linux-only extension of some kind.
> Correct me if I'm wrong.)  But that makes us very concerned about how posix
> advisory locking will behave.  Posix advisory locks are mis-designed.  See
> 
>  http://www.sqlite.org/src/artifact/64a2e0ebbb?ln=621,685
> 
> Are you sure that posix advisory locking will work on a file descriptor that
> was allocated in a different process?  Have you tested it?
> 
> So this all gets very complicated very fast.  And very hard to test.

I'm only beginning to delve into the possibilities offered by the VFS
API in SQLite. However, from what I have seen so far, it seems like a
very complete abstraction to me.

Why is it not possible to create a custom VFS for the intended purpose
and register it for the connections that need it?

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] adding fdopen to VFS?

[Richard Hipp]

On Mon, Feb 28, 2011 at 2:33 PM, Richard Hipp  wrote:

>
>
> On Mon, Feb 28, 2011 at 1:00 PM, Paweł Hajdan, Jr. <
> phajdan...@chromium.org> wrote:
>
>>
>> What do you think about adding a call like below to the VFS interface?
>>
>> int (*xOpenWrapper)(const char *path, int flags, int mode);
>>
>
> There are 2 billion instances of SQLite in active use today.  We don't want
> to go changing interfaces unless we absolutely have to.
>
> And there are additional complications.  Based on what we can figure out
> about how your sandbox works, you'll also need overloads for access(),
> stat(), and unlink().  Presumably you have no need for dlopen() which is
> used for run-time loadable extensions to SQLite, but if you use that
> features you'll need an overload for it too.  We might yet think of others,
> but already we see there are at least three interfaces (we could probably
> combine access and stat) that need to be overloaded.
>

SQLite converts relative pathnames (in sqlite3_open()) into absolute
pathnames before sending them into xOpen().  A call to getcwd() is necessary
for that.  Looks like that call, too, will need to be intercepted and
handled by the supervisor process.  Or you can work around it by always
handing full pathnames to sqlite3_open().



>
> We are also concerned about locking.  The documentation on Chromium
> sandboxing implies that a separate process does the open() and then sends
> over a file descriptor using IPC.  (I didn't know you could do that in
> unix.  In fact, I bet it requires a linux-only extension of some kind.
> Correct me if I'm wrong.)  But that makes us very concerned about how posix
> advisory locking will behave.  Posix advisory locks are mis-designed.  See
>
>  http://www.sqlite.org/src/artifact/64a2e0ebbb?ln=621,685
>
> Are you sure that posix advisory locking will work on a file descriptor
> that was allocated in a different process?  Have you tested it?
>
> So this all gets very complicated very fast.  And very hard to test.
>
> Suggestions:
>
> (1) Let's move this discussion to sqlite-...@sqlite.org
>
> (2) Can you set up a conf-call or video-conf between the Chromium
> developers who have an interest in this and the SQLite development team so
> that we can talk through the issues?
>
>
> --
> D. Richard Hipp
> d...@sqlite.org
>



-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] adding fdopen to VFS?

[Richard Hipp]

On Mon, Feb 28, 2011 at 1:00 PM, Paweł Hajdan, Jr.
wrote:

> On Mon, Feb 28, 2011 at 15:12, Richard Hipp  wrote:
>
>> On Mon, Feb 28, 2011 at 8:36 AM, Paweł Hajdan, Jr. <
>> phajdan...@chromium.org> wrote:
>>
>>>
>>> I remember a part of earlier discussion that there is a problem with
>>> journal and possibly other auxiliary files, i.e. we can't create/open them
>>> just based on one file descriptor.
>>>
>>
>> Right.  If you call sqlite3_open("xyzzy",...) then you end up opening
>> various files, including but no limited to "xyzzy", "xyzzy-journal",
>> "xyzzy-wal", and "xyzzy-shm".
>>
>
> I see. Does sqlite call the VFS's xOpen function for each of those files?
>
> I'd need your advice on the above, i.e. assuming that the code can't call
>>> open directly (the call will fail because of the sandbox) but it can ask the
>>> browser process to open a file and send back a file descriptor.
>>>
>>> It is possible that instead of this entire xFdOpen thing that I
>>> suggested, it would be better to make it possible to pass a custom function
>>> used by sqlite instead of POSIX's open. In this case, that would roughly
>>> be PlatformBridge::databaseOpenFile.
>>>
>>
>> Can't you simply compile the standard "sqlite3.c" amalgamated source file
>> using
>>
>> -Dopen=openUsingPlatformBridge
>>
>> and then implement a single function openUsingPlatformBridge() that does
>> all the necessary magic for you?
>>
>
> It has two drawbacks I'd like to avoid:
>
> 1) It requires two copies of sqlite code, one compiled with
> openUsingPlatformBridge (for the renderer), and one using the real open (for
> the browser)
>
> 2) It requires a non-standard compilation of sqlite, and ideally the
> solution should work with a vanilla sqlite, for example as shipped by Linux
> distributions.
>
> What do you think about adding a call like below to the VFS interface?
>
> int (*xOpenWrapper)(const char *path, int flags, int mode);
>

There are 2 billion instances of SQLite in active use today.  We don't want
to go changing interfaces unless we absolutely have to.

And there are additional complications.  Based on what we can figure out
about how your sandbox works, you'll also need overloads for access(),
stat(), and unlink().  Presumably you have no need for dlopen() which is
used for run-time loadable extensions to SQLite, but if you use that
features you'll need an overload for it too.  We might yet think of others,
but already we see there are at least three interfaces (we could probably
combine access and stat) that need to be overloaded.

We are also concerned about locking.  The documentation on Chromium
sandboxing implies that a separate process does the open() and then sends
over a file descriptor using IPC.  (I didn't know you could do that in
unix.  In fact, I bet it requires a linux-only extension of some kind.
Correct me if I'm wrong.)  But that makes us very concerned about how posix
advisory locking will behave.  Posix advisory locks are mis-designed.  See

 http://www.sqlite.org/src/artifact/64a2e0ebbb?ln=621,685

Are you sure that posix advisory locking will work on a file descriptor that
was allocated in a different process?  Have you tested it?

So this all gets very complicated very fast.  And very hard to test.

Suggestions:

(1) Let's move this discussion to sqlite-...@sqlite.org

(2) Can you set up a conf-call or video-conf between the Chromium developers
who have an interest in this and the SQLite development team so that we can
talk through the issues?


-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] adding fdopen to VFS?

[Paweł Hajdan , Jr .]

On Mon, Feb 28, 2011 at 15:12, Richard Hipp  wrote:

> On Mon, Feb 28, 2011 at 8:36 AM, Paweł Hajdan, Jr. <
> phajdan...@chromium.org> wrote:
>
>>
>> I remember a part of earlier discussion that there is a problem with
>> journal and possibly other auxiliary files, i.e. we can't create/open them
>> just based on one file descriptor.
>>
>
> Right.  If you call sqlite3_open("xyzzy",...) then you end up opening
> various files, including but no limited to "xyzzy", "xyzzy-journal",
> "xyzzy-wal", and "xyzzy-shm".
>

I see. Does sqlite call the VFS's xOpen function for each of those files?

I'd need your advice on the above, i.e. assuming that the code can't call
>> open directly (the call will fail because of the sandbox) but it can ask the
>> browser process to open a file and send back a file descriptor.
>>
>> It is possible that instead of this entire xFdOpen thing that I suggested,
>> it would be better to make it possible to pass a custom function used by
>> sqlite instead of POSIX's open. In this case, that would roughly
>> be PlatformBridge::databaseOpenFile.
>>
>
> Can't you simply compile the standard "sqlite3.c" amalgamated source file
> using
>
> -Dopen=openUsingPlatformBridge
>
> and then implement a single function openUsingPlatformBridge() that does
> all the necessary magic for you?
>

It has two drawbacks I'd like to avoid:

1) It requires two copies of sqlite code, one compiled with
openUsingPlatformBridge (for the renderer), and one using the real open (for
the browser)

2) It requires a non-standard compilation of sqlite, and ideally the
solution should work with a vanilla sqlite, for example as shipped by Linux
distributions.

What do you think about adding a call like below to the VFS interface?

int (*xOpenWrapper)(const char *path, int flags, int mode);

This roughly corresponds to open's signature from fcntl.h (man 2 open), and
"by default" it would be just the real open. Chromium would
substitute PlatformBridge::databaseOpenFile in that place in its own VFS
(and re-use most of other calls from the "unix" VFS).

Of course such a call is not cross-platform, so I know it's not going to get
accepted. However, I'm wondering what you think about it (assuming for a
while we don't care about other platforms), and maybe you have some ideas
for interfaces to make sqlite's os_unix VFS obtain the file descriptor in
other way than just calling open.

Also, this is not necessarily POSIX-only. Chromium also has a similar hack
for Windows, just a HANDLE is passed instead of an int file descriptor.

On Mon, Feb 28, 2011 at 17:16, Nico Williams  wrote:

> It's not much of a sandbox if you let the sandbox have direct access
> to SQLite3 DB files...  I worry that a compromised sandbox could muck
> with the database in ways that are more harmful than merely
> inserting/updating/deleting random rows.
>

If the database file we give it is only ever used by the sandboxed
processes, that should be fine, right? They can only break themselves.
Anyway, that's slightly beyond the topic, as I didn't design the original
Chromium code that handles it.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] adding fdopen to VFS?

[Richard Hipp]

On Mon, Feb 28, 2011 at 8:36 AM, Paweł Hajdan, Jr.
wrote:

>
> I remember a part of earlier discussion that there is a problem with
> journal and possibly other auxiliary files, i.e. we can't create/open them
> just based on one file descriptor.
>

Right.  If you call sqlite3_open("xyzzy",...) then you end up opening
various files, including but no limited to "xyzzy", "xyzzy-journal",
"xyzzy-wal", and "xyzzy-shm".


> I'd need your advice on the above, i.e. assuming that the code can't call
> open directly (the call will fail because of the sandbox) but it can ask the
> browser process to open a file and send back a file descriptor.
>
> It is possible that instead of this entire xFdOpen thing that I suggested,
> it would be better to make it possible to pass a custom function used by
> sqlite instead of POSIX's open. In this case, that would roughly
> be PlatformBridge::databaseOpenFile.
>

Can't you simply compile the standard "sqlite3.c" amalgamated source file
using

-Dopen=openUsingPlatformBridge

and then implement a single function openUsingPlatformBridge() that does all
the necessary magic for you?



>
> Paweł Hajdan, Jr.
>



-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] adding fdopen to VFS?

[Paweł Hajdan , Jr .]

2011/2/28 Richard Hipp 

> On Mon, Feb 28, 2011 at 7:39 AM, Paweł Hajdan, Jr. <
> phajdan...@chromium.org> wrote:
>
>> I'd like to add an equivalent of fdopen to the sqlite's VFS (
>> http://www.sqlite.org/c3ref/vfs.html). The signature would similar to
>> this:
>>
>> int (*xFdOpen)(sqlite3_vfs*, int fd, sqlite3_file*, int flags, int
>> *pOutFlags);
>>
>
> What does xFdOpen() do in your proposal?  Who calls xFdOpen()?  What does
> xFdOpen() return?  What processing does xFdOpen() perform?
>

I think I'd need your help to make this more precise/complete, but here's
what I have:

I'd need a corresponding sqlite3_fdopen call, that would be similar to
sqlite3_open but take a file descriptor instead of a path, and then use
xFdOpen to actually perform the operation. xFdOpen would return an error
code I guess (I intend to make its behavior almost identical to xOpen, with
the only difference being using fd instead of a path).

Internally xFdOpen would do everything that xOpen would do after getting a
file descriptor.

The context where I'd like to use sqlite3_fdopen is
http://codesearch.google.com/codesearch/p#OAMlx_jo-ck/src/third_party/WebKit/Source/WebCore/platform/sql/chromium/SQLiteFileSystemChromiumPosix.cpp=72_package=chromium
 .

PlatformBridge::databaseOpenFile returns a file descriptor (received from
the browser process, and transferred over a local socket to the sandboxed
renderer process; the linked code runs inside the renderer; it can't call
open directly).

I remember a part of earlier discussion that there is a problem with journal
and possibly other auxiliary files, i.e. we can't create/open them just
based on one file descriptor. There is a possibility that I'm not
seeing/posting all relevant parts of the code, or that it's just broken.

I'd need your advice on the above, i.e. assuming that the code can't call
open directly (the call will fail because of the sandbox) but it can ask the
browser process to open a file and send back a file descriptor.

It is possible that instead of this entire xFdOpen thing that I suggested,
it would be better to make it possible to pass a custom function used by
sqlite instead of POSIX's open. In this case, that would roughly
be PlatformBridge::databaseOpenFile.

Paweł Hajdan, Jr.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] adding fdopen to VFS?

[Richard Hipp]

On Mon, Feb 28, 2011 at 7:39 AM, Paweł Hajdan, Jr.
wrote:

> I'd like to add an equivalent of fdopen to the sqlite's VFS (
> http://www.sqlite.org/c3ref/vfs.html). The signature would similar to
> this:
>
> int (*xFdOpen)(sqlite3_vfs*, int fd, sqlite3_file*, int flags, int
> *pOutFlags);
>

What does xFdOpen() do in your proposal?  Who calls xFdOpen()?  What does
xFdOpen() return?  What processing does xFdOpen() perform?



>
> This would be needed to cleanly implement fd-passing in Chromium, from
> the browser to the sandboxed renderer process.
>
> I was writing about this earlier, see below (all link to the same thread):
>
> http://old.nabble.com/upstreaming-Chromium-patches-for-file-handle-passing-support-td29547456.html
> http://permalink.gmane.org/gmane.comp.db.sqlite.general/59066
>
> http://sqlite.org:8080/cgi-bin/mailman/private/sqlite-users/2010-August/023922.html
>
> One of the problems with just copy-pasting the os_unix.c code to our WebKit
> port was that the changes to os_unix.c made between 3.6.x and 3.7.x were
> not
> so trivial to apply (parts of the code were removed to reduce the amount of
> code from ~5000 LOC to ~1000 LOC, and it was re-formatted to match WebKit
> style). The change has been reverted anyway for unrelated reasons, but I
> started thinking about trying to solve it in sqlite.
>
> If we had an fdopen-like call in sqlite's VFS (and I'm willing to implement
> it), the functionality needed for Chromium could be implemented very
> cleanly. What do you think about that?
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users