Re: [sqlite] adding fdopen to VFS?

2011-02-28 Thread Paweł Hajdan , Jr . 
On Mon, Feb 28, 2011 at 15:12, Richard Hipp <d...@sqlite.org> wrote:

> On Mon, Feb 28, 2011 at 8:36 AM, Paweł Hajdan, Jr. <
> phajdan...@chromium.org> wrote:
>
>>
>> I remember a part of earlier discussion that there is a problem with
>> journal and possibly other auxiliary files, i.e. we can't create/open them
>> just based on one file descriptor.
>>
>
> Right.  If you call sqlite3_open("xyzzy",...) then you end up opening
> various files, including but no limited to "xyzzy", "xyzzy-journal",
> "xyzzy-wal", and "xyzzy-shm".
>

I see. Does sqlite call the VFS's xOpen function for each of those files?

I'd need your advice on the above, i.e. assuming that the code can't call
>> open directly (the call will fail because of the sandbox) but it can ask the
>> browser process to open a file and send back a file descriptor.
>>
>> It is possible that instead of this entire xFdOpen thing that I suggested,
>> it would be better to make it possible to pass a custom function used by
>> sqlite instead of POSIX's open. In this case, that would roughly
>> be PlatformBridge::databaseOpenFile.
>>
>
> Can't you simply compile the standard "sqlite3.c" amalgamated source file
> using
>
> -Dopen=openUsingPlatformBridge
>
> and then implement a single function openUsingPlatformBridge() that does
> all the necessary magic for you?
>

It has two drawbacks I'd like to avoid:

1) It requires two copies of sqlite code, one compiled with
openUsingPlatformBridge (for the renderer), and one using the real open (for
the browser)

2) It requires a non-standard compilation of sqlite, and ideally the
solution should work with a vanilla sqlite, for example as shipped by Linux
distributions.

What do you think about adding a call like below to the VFS interface?

int (*xOpenWrapper)(const char *path, int flags, int mode);

This roughly corresponds to open's signature from fcntl.h (man 2 open), and
"by default" it would be just the real open. Chromium would
substitute PlatformBridge::databaseOpenFile in that place in its own VFS
(and re-use most of other calls from the "unix" VFS).

Of course such a call is not cross-platform, so I know it's not going to get
accepted. However, I'm wondering what you think about it (assuming for a
while we don't care about other platforms), and maybe you have some ideas
for interfaces to make sqlite's os_unix VFS obtain the file descriptor in
other way than just calling open.

Also, this is not necessarily POSIX-only. Chromium also has a similar hack
for Windows, just a HANDLE is passed instead of an int file descriptor.

On Mon, Feb 28, 2011 at 17:16, Nico Williams <n...@cryptonector.com> wrote:

> It's not much of a sandbox if you let the sandbox have direct access
> to SQLite3 DB files...  I worry that a compromised sandbox could muck
> with the database in ways that are more harmful than merely
> inserting/updating/deleting random rows.
>

If the database file we give it is only ever used by the sandboxed
processes, that should be fine, right? They can only break themselves.
Anyway, that's slightly beyond the topic, as I didn't design the original
Chromium code that handles it.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] adding fdopen to VFS?

2011-02-28 Thread Paweł Hajdan , Jr . 
2011/2/28 Richard Hipp <d...@sqlite.org>

> On Mon, Feb 28, 2011 at 7:39 AM, Paweł Hajdan, Jr. <
> phajdan...@chromium.org> wrote:
>
>> I'd like to add an equivalent of fdopen to the sqlite's VFS (
>> http://www.sqlite.org/c3ref/vfs.html). The signature would similar to
>> this:
>>
>> int (*xFdOpen)(sqlite3_vfs*, int fd, sqlite3_file*, int flags, int
>> *pOutFlags);
>>
>
> What does xFdOpen() do in your proposal?  Who calls xFdOpen()?  What does
> xFdOpen() return?  What processing does xFdOpen() perform?
>

I think I'd need your help to make this more precise/complete, but here's
what I have:

I'd need a corresponding sqlite3_fdopen call, that would be similar to
sqlite3_open but take a file descriptor instead of a path, and then use
xFdOpen to actually perform the operation. xFdOpen would return an error
code I guess (I intend to make its behavior almost identical to xOpen, with
the only difference being using fd instead of a path).

Internally xFdOpen would do everything that xOpen would do after getting a
file descriptor.

The context where I'd like to use sqlite3_fdopen is
http://codesearch.google.com/codesearch/p#OAMlx_jo-ck/src/third_party/WebKit/Source/WebCore/platform/sql/chromium/SQLiteFileSystemChromiumPosix.cpp=72_package=chromium
 .

PlatformBridge::databaseOpenFile returns a file descriptor (received from
the browser process, and transferred over a local socket to the sandboxed
renderer process; the linked code runs inside the renderer; it can't call
open directly).

I remember a part of earlier discussion that there is a problem with journal
and possibly other auxiliary files, i.e. we can't create/open them just
based on one file descriptor. There is a possibility that I'm not
seeing/posting all relevant parts of the code, or that it's just broken.

I'd need your advice on the above, i.e. assuming that the code can't call
open directly (the call will fail because of the sandbox) but it can ask the
browser process to open a file and send back a file descriptor.

It is possible that instead of this entire xFdOpen thing that I suggested,
it would be better to make it possible to pass a custom function used by
sqlite instead of POSIX's open. In this case, that would roughly
be PlatformBridge::databaseOpenFile.

Paweł Hajdan, Jr.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] adding fdopen to VFS?

2011-02-28 Thread Paweł Hajdan , Jr . 
I'd like to add an equivalent of fdopen to the sqlite's VFS (
http://www.sqlite.org/c3ref/vfs.html). The signature would similar to this:

int (*xFdOpen)(sqlite3_vfs*, int fd, sqlite3_file*, int flags, int
*pOutFlags);

This would be needed to cleanly implement fd-passing in Chromium, from
the browser to the sandboxed renderer process.

I was writing about this earlier, see below (all link to the same thread):
http://old.nabble.com/upstreaming-Chromium-patches-for-file-handle-passing-support-td29547456.html
http://permalink.gmane.org/gmane.comp.db.sqlite.general/59066
http://sqlite.org:8080/cgi-bin/mailman/private/sqlite-users/2010-August/023922.html

One of the problems with just copy-pasting the os_unix.c code to our WebKit
port was that the changes to os_unix.c made between 3.6.x and 3.7.x were not
so trivial to apply (parts of the code were removed to reduce the amount of
code from ~5000 LOC to ~1000 LOC, and it was re-formatted to match WebKit
style). The change has been reverted anyway for unrelated reasons, but I
started thinking about trying to solve it in sqlite.

If we had an fdopen-like call in sqlite's VFS (and I'm willing to implement
it), the functionality needed for Chromium could be implemented very
cleanly. What do you think about that?
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Involving in sqlite development

2010-10-08 Thread Paweł Hajdan , Jr . 
On Fri, Oct 8, 2010 at 15:28, sjtirtha  wrote:

> 1. setup an development environment for SQLite (I'm using Ubuntu)
>Do you have any preference which editor or IDE should I use?


It's very easy. Download the sources, compile the "standard" way
(./configure; make). Any editor is fine.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] upstreaming Chromium patches for file handle passing support

2010-10-04 Thread Paweł Hajdan , Jr . 
On Tue, Aug 31, 2010 at 19:51, Richard Hipp  wrote:

> My suggestion is that you make a copy of the os_unix.c source file (call it
> chromium_vfs.c or anything else that you like) and apply your edits to that
> copy.


FYI, that's what finally happened.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] [PATCH] Fix locale-unsafe usage of tolower (upstreaming Chromium change)

2010-09-16 Thread Paweł Hajdan , Jr . 
If you have some time for that, I'd appreciate patching at least fts2. This
will make it possible for Chromium to drop another custom patch for SQLite.

On Wed, Aug 18, 2010 at 13:49, Richard Hipp <d...@sqlite.org> wrote:

> I'm thinking that you shouldn't be using FTS1 and FTS2 in the first place.
> They are untested and unsupported.  We'll get around to patching them, if
> you insist, but right now we are busy trying to 3.7.1 out the door.
>
> On Wed, Aug 18, 2010 at 4:41 PM, Paweł Hajdan, Jr.
> <phajdan...@chromium.org>wrote:
>
> > On Tue, Aug 10, 2010 at 13:16, Paweł Hajdan, Jr. <
> phajdan...@chromium.org
> > >wrote:
> >
> > > Now, how about fts1 and fts2? The original chromium patch is at
> > > http://codereview.chromium.org/174387 . Could you take a look and
> > suggest
> > > a way to upstream those fixes to SQLite?
> > >
> >
> > Ping about the above. Or have the fixes already been made and we just
> need
> > to upgrade to new sqlite?
> > ___
> > sqlite-users mailing list
> > sqlite-users@sqlite.org
> > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
> >
>
>
>
> --
> D. Richard Hipp
> d...@sqlite.org
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] upstreaming Chromium patches for file handle passing support

2010-09-09 Thread Paweł Hajdan , Jr . 
On Tue, Aug 31, 2010 at 10:51, Richard Hipp  wrote:

> My suggestion is that you make a copy of the os_unix.c source file (call it
> chromium_vfs.c or anything else that you like) and apply your edits to that
> copy.  Since you started with an exact copy of the original os_unix.c
> source
> file, your copy will all the internal utilities and mechanisms as
> os_unix.c.  You can use all those internal mechanism as much as you want.
> Then compile your customized VFS separately and add it to SQLite at
> start-time using sqlite3_vfs_register().
>

I have done a proof-of-concept patch, attached to
https://bugs.webkit.org/show_bug.cgi?id=45416

Direct link:
https://bugs.webkit.org/attachment.cgi?id=66952=prettypatch

However, Dumitru Daniliuc (dumi) has noticed that we could still reuse large
portions of os_unix.c and only provide our own implementation for opening a
file (instead of calling "open" directly, it would call
"ChromiumBridge::databaseOpenFile". Similarly for database deletion, etc.

Now that we have the code to talk about (see above), could you take another
look and see if it would be possible to avoid copy-pasting parts of
os_unix.c? I'm just trying to explore all possibilities we have - both
copying the code (done now), and re-using more of os_unix.c code.

On Wed, Sep 1, 2010 at 21:33, Dan Kennedy  wrote:

> How do you know which journal file to open or where to create
> a master journal file if the client writes to the database?


It is hard for me to answer this question (I'm not very familiar with sqlite
yet). See the patch above - I think it preserves the semantics of current
code used in Chrome, just moves it to WebKit. Any feedback on whether that
code is right or wrong and how it can be improved would be appreciated.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] upstreaming Chromium patches for file handle passing support

2010-09-01 Thread Paweł Hajdan , Jr . 
2010/8/31 Richard Hipp 

> os_unix.c is very unlikely to change in ways that you care about.


I think the biggest concern is just copying that code, please see
http://groups.google.com/a/chromium.org/group/chromium-dev/browse_thread/thread/7f8e9bf0f034322f/b26e422d4aa711c7.
I think that keeping it in sync is a relatively minor concern, given
that
we update SQLite so rarely.


> The VFS is a well-defined and supported interface for SQLite.  We maintain
> backwards compatibility.  So even if you had been using an older custom VFS
> built using an older 3.6.23-or-earlier os_unix.c, it will still continue
> working on newer versions of SQLite - you just wouldn't be able to take
> advantage of the enhanced write-ahead log feature.
>

I wonder whether it would be possible to extend the VFS in a way that would
make our use case possible (transferring a file handle over process
boundary). Please note that we do it on all platforms. On POSIX we pass an
integer file descriptor, and on Windows we pass a HANDLE.

Additionally, browsers seem to be moving to the multi-process architecture
(Chrome, Firefox, WebKit2), so I wouldn't be surprised if you get more
questions about this in the future, or just more forked copies.

What do you think about extending the SQLite VFS to make it possible to open
a database having a file handle (fd on Unix, HANDLE on Windows)? Opening
based on file path would still be there (to preserve compatibility).


> The only thing that might trip you up is a bug fix in os_unix.c.  Those are
> very, very uncommon.  In fact, I can't call to mind the last time we had
> any
> serious bug in os_unix.c.


My biggest concern in this thread is a nice design. We have to bundle SQLite
in Chromium repository anyway because non-Linux platforms usually don't
provide SQLite, or have an older version, etc. Updating a custom-patched
os_unix.c (our current situation) is equally difficult to updating a copied
and patched os_unix.c, in case we have to apply a bugfix.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] upstreaming Chromium patches for file handle passing support

2010-08-31 Thread Paweł Hajdan , Jr . 
On Tue, Aug 31, 2010 at 10:51, Richard Hipp  wrote:

> My suggestion is that you make a copy of the os_unix.c source file (call it
> chromium_vfs.c or anything else that you like) and apply your edits to that
> copy.  Since you started with an exact copy of the original os_unix.c
> source
> file, your copy will all the internal utilities and mechanisms as
> os_unix.c.  You can use all those internal mechanism as much as you want.
> Then compile your customized VFS separately and add it to SQLite at
> start-time using sqlite3_vfs_register().
>
> Why won't that work for you?


It effectively forks os_unix.c, which makes upgrades of SQLite equally
painful to the current state. The above solution offers no advantage over
the current state of our bundled SQLite copy. I'd like to stay as close to
upstream as possible, because then we can update very easily.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] upstreaming Chromium patches for file handle passing support

2010-08-31 Thread Paweł Hajdan , Jr . 
On Fri, Aug 27, 2010 at 18:27, Paweł Hajdan, Jr. <phajdan...@chromium.org>wrote:

>
>
>> At start-time, simply call sqlite3_vfs_register() to register your
>> customized VFS with SQLite.
>>
>
> That's what our implementation effectively does, see
> http://trac.webkit.org/browser/trunk/WebCore/platform/sql/chromium/SQLiteFileSystemChromiumPosix.cpp
>  .
>
> Would it be possible to add a variant of the unix vfs inside of os_unix.c?
> I'm still not sure if the current VFS interface would be sufficient for
> that. It seems that the most needed exposed call is ability to transform a
> file path into a file descriptor
> (chromium_sqlite3_get_reusable_file_handle). 
> chromium_sqlite3_get_reusable_file_handle
> currently needs access to things like unixFile, findReusableFd,
> UnixUnusedFd, and sqlite3_malloc. Would you have any suggestions how to fit
> that into the SQLite's VFS implementation?
>

Let me clarify what the current implementation in Chrome and WebKit does.
I'd appreciate suggestions how to fit it into the SQLite API, and I'd be
glad to do the necessary work.

So let's start with
http://trac.webkit.org/browser/trunk/WebCore/platform/sql/chromium/SQLiteFileSystemChromiumPosix.cpp.
In SQLiteFileSystem::registerSQLiteVFS (line 171), we register our own
SQLite VFS ("chromium_vfs") mostly based on the Unix VFS with a few
overrides.

Now please take a look at chromiumOpen (line 62). This code runs in the
sandboxed renderer process, and does not have direct access to the
filesystem.   It uses ChromiumBridge to communicate with the outside world
(it communicates with the main browser process via IPC, and the browser
process acts as a broker). So as you can see the renderer sends most of the
details to the browser process (which runs with full user's rights), and
gets back a file descriptor.

And the hard part now is that we need to pass that file descriptor to
SQLite. However, the public SQLite APIs all seem to be path-based. However,
the internals of os_unix.c have definitions of unixFile, findReusableFd,
UnixUnusedFd that allow us to make SQLite use the file descriptor we got.

Please note that we get the file descriptor via ChromiumBridge, a class that
lives in WebKit. This makes it non-trivial to put the entire implementation
of "chromium_vfs" in SQLite, because we need to ask the browser process for
the file descriptor. And we can't have the entire implementation on the
WebKit/Chrome side either, because then we can't compile against vanilla
SQLite (and Linux distributions don't like it).

Could you suggest a better way to do the above, that would fit better within
SQLite's design? Please let me know if you have more questions about how we
currently use SQLite, the multi-process architecture of Chrome, etc.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] upstreaming Chromium patches for file handle passing support

2010-08-27 Thread Paweł Hajdan , Jr . 
On Fri, Aug 27, 2010 at 04:58, Richard Hipp  wrote:

> The VFS interface is published, stable, and documented.  I think the VFS is
> the interface you should be using.
>
> You do not have to patch os_unix.c.  Leave it unchanged.


Sounds good to me. This is one of my goals - to avoid those custom patches,
or send them upstream.


> Instead, create your own os_chromium.c based off of your patched os_unix.c
> (changing just a
> few identifiers to avoid conflicts) and link it with your process as a
> separate file.


I think this is the non-trivial step. Those chromium_sqlite3 functions use
some symbols that are declared static in os_unix.c, so they're not available
outside of it. That means we would have to copy a lot of os_unix.c file's
contents, and I'd prefer to avoid that.


> (You really should be using the SQLite amalgamation file
> sqlite3.c - not separate source files - since the amalgamation runs about
> 5%
> or 10% faster due to better compiler optimizations.)


At least currently Chromium isn't going to use the amalgamation because we
have a lot of custom patches. I'm trying to clean up this situation though.


> At start-time, simply call sqlite3_vfs_register() to register your
> customized VFS with SQLite.
>

That's what our implementation effectively does, see
http://trac.webkit.org/browser/trunk/WebCore/platform/sql/chromium/SQLiteFileSystemChromiumPosix.cpp
 .

Would it be possible to add a variant of the unix vfs inside of os_unix.c?
I'm still not sure if the current VFS interface would be sufficient for
that. It seems that the most needed exposed call is ability to transform a
file path into a file descriptor
(chromium_sqlite3_get_reusable_file_handle).
chromium_sqlite3_get_reusable_file_handle
currently needs access to things like unixFile, findReusableFd,
UnixUnusedFd, and sqlite3_malloc. Would you have any suggestions how to fit
that into the SQLite's VFS implementation?
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] upstreaming Chromium patches for file handle passing support

2010-08-26 Thread Paweł Hajdan , Jr . 
Please take a look at chromium_sqlite3 functions in
http://src.chromium.org/viewvc/chrome/trunk/src/third_party/sqlite/src/src/os_unix.c?view=markup

They are needed because in Chrome the browser process will pass a file
descriptor to the child renderer process instead of a file path. Here's the
code that handles it on the renderer side:
http://trac.webkit.org/browser/trunk/WebCore/platform/sql/chromium/SQLiteFileSystemChromiumPosix.cpp

Currently those chromium_sqlite3 functions live in patched os_unix.c file
because they access slite internals that are only visible in os_unix.c.

Could you suggest a way to do the same thing in a way that wouldn't require
custom patches on the Chromium project side? If we can get a similar
interface exposed in sqlite that'd be great. Another possible solution would
be to allow those chromium_sqlite3 functions to be implemented outside of
os_unix.c file (that will probably require exposing more internals from that
file).
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] errors running test suite (couldn't execute "testfixture")

2010-08-20 Thread Paweł Hajdan , Jr . 
After updating the tests run fine now. Thanks for the quick update!

By the way, I usually avoid having "." in the PATH for security reasons.

On Fri, Aug 20, 2010 at 05:30, Dan Kennedy <danielk1...@gmail.com> wrote:

>
> On Aug 20, 2010, at 6:07 AM, Paweł Hajdan, Jr. wrote:
>
> > I updated to latest fossil version, ran make distclean, ./configure,
> > make,
> > make test and got this:
>
> Thanks for this report. The test code was assuming that "."
> was in your PATH variable. Fixed now.
>
> > What should I do to make it pass?
>
> Either add "." to the PATH variable or do another update.
>
> Dan.
>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] errors running test suite (couldn't execute "testfixture")

2010-08-19 Thread Paweł Hajdan , Jr . 
I updated to latest fossil version, ran make distclean, ./configure, make,
make test and got this:

avtrans-9.19.4-5116... Ok
avtrans-9.19.5-5116... Ok
avtrans-9.20.1-5640... Ok
avtrans-9.20.2-5640... Ok
avtrans-10.1... Ok
avtrans.test-closeallfiles... Ok
avtrans.test-sharedcachesetting... Ok
Time: avtrans.test 6915 ms
Memory used:  now 16  max2372872  max-size 530991
Allocation count: now  1  max   3346
Page-cache used:  now  0  max  0  max-size   1272
Page-cache overflow:  now  0  max 759264
Scratch memory used:  now  0  max  0
Scratch overflow: now  0  max   8384  max-size   8384
./testfixture: couldn't execute "testfixture": no such file or directory
while executing
"open "|$prg tf_main.tcl" r+"
(procedure "launch_testfixture" line 5)
invoked from within
"launch_testfixture $binary"
(procedure "get_version" line 2)
invoked from within
"get_version $bin"
("foreach" body line 2)
invoked from within
"foreach bin $binaries {
  puts "Testing against $bin - version [get_version $bin]"
}"
(file "./test/backcompat.test" line 55)
invoked from within
"source ./test/backcompat.test"
invoked from within
"interp eval tinterp $script"
(procedure "slave_test_script" line 24)
invoked from within
"slave_test_script [list source $zFile] "
invoked from within
"time { slave_test_script [list source $zFile] }"
(procedure "slave_test_file" line 14)
invoked from within
"slave_test_file $::testdir/$file"
(procedure "run_tests" line 12)
invoked from within
"run_tests veryquick -presql {} -files {shared3.test bigfile.test
where9.test tkt3419.test sync.test fts1o.test fts2f.test misc2.test
tkt3541.test type..."
("uplevel" body line 1)
invoked from within
"uplevel run_tests $name $::testspec($name)"
(procedure "run_test_suite" line 5)
invoked from within
"run_test_suite veryquick"
(file "./test/veryquick.test" line 16)
make: *** [test] Error 1

What should I do to make it pass?
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] [PATCH] Fix locale-unsafe usage of tolower (upstreaming Chromium change)

2010-08-18 Thread Paweł Hajdan , Jr . 
On Tue, Aug 10, 2010 at 13:16, Paweł Hajdan, Jr. <phajdan...@chromium.org>wrote:

> Now, how about fts1 and fts2? The original chromium patch is at
> http://codereview.chromium.org/174387 . Could you take a look and suggest
> a way to upstream those fixes to SQLite?
>

Ping about the above. Or have the fixes already been made and we just need
to upgrade to new sqlite?
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] [PATCH] cache preloading

2010-08-16 Thread Paweł Hajdan , Jr . 
ping

Could you take a look at the original Chromium patch:
http://src.chromium.org/viewvc/chrome/trunk/src/third_party/sqlite/preload-cache.patch?revision=26596=markup?

Is it something you'd like to include in SQLite? If so, does the patch need
any adjustments before that's possible?

On Wed, Aug 11, 2010 at 23:23, Damian Pietras <da...@daper.net> wrote:

> On Tue, Aug 10, 2010 at 05:46:26PM -0700, Paweł Hajdan, Jr. wrote:
> > So this is another chromium patch I'd like to submit:
> >
> http://src.chromium.org/viewvc/chrome/trunk/src/third_party/sqlite/preload-cache.patch?revision=26596=markup
> >
> > I'm not the author of that one, but the main idea seems to be that with
> > preloading we get better performance with many scenarios chromium uses
> > sqlite (like for omnibox local suggestions). What do you think about the
> > patch and the idea? Is there better way to do the same thing? Would you
> like
> > to see some benchmarks?
>
> I thought about something like that. I took a little different approach:
> I've put posix_fadvise(fd, 0, 0, POSIX_FADV_WILLNEED) just after opening
> the database file and in my case with a fragmented database (may
> insert/update/delete without vacuum) and cold page it was a huge
> difference.
>
> Using posix_fadvise() is asynchronous without any separate thread and
> opening the database isn't slowed down - you don't need to wait
> until the file is loaded.
>
> I'm interested in speeding up SQLite when opening a database that is not
> in the OS's page cache. Next thing I want to try is to preload only
> selected tables/indexes with help of posix_fadvise(). I want to call
> posix_fadvice for every child page when traveling through the btree just
> before moving down to the first child of the current page.  I hope
> this will be faster that a regular table/index read because the OS/disk
> will have the IO queue full and can reorder read requestes if necessary.
> It will be slower than preloading the whole DB, but I don't want to
> waste disk bandwidth for that.
>
> --
> Damian Pietras
>
> http://www.linuxprogrammingblog.com
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] [PATCH] cache preloading

2010-08-10 Thread Paweł Hajdan , Jr . 
So this is another chromium patch I'd like to submit:
http://src.chromium.org/viewvc/chrome/trunk/src/third_party/sqlite/preload-cache.patch?revision=26596=markup

I'm not the author of that one, but the main idea seems to be that with
preloading we get better performance with many scenarios chromium uses
sqlite (like for omnibox local suggestions). What do you think about the
patch and the idea? Is there better way to do the same thing? Would you like
to see some benchmarks?
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] [PATCH] Fix locale-unsafe usage of tolower (upstreaming Chromium change)

2010-08-10 Thread Paweł Hajdan , Jr . 
Thank you, I have backported it to chromium as
http://src.chromium.org/viewvc/chrome?view=rev=55504

Now, how about fts1 and fts2? The original chromium patch is at
http://codereview.chromium.org/174387 . Could you take a look and suggest a
way to upstream those fixes to SQLite?

On Fri, Aug 6, 2010 at 12:10, Richard Hipp <d...@sqlite.org> wrote:

> FTS3 updated here:  http://www.sqlite.org/src/ci/b8b465ed2c
>
> On Fri, Aug 6, 2010 at 2:24 PM, Scott Hess <sh...@google.com> wrote:
>
> > This bug comment describes the problem:
> >   http://code.google.com/p/chromium/issues/detail?id=15261#c20
> >
> > excerpt:
> > > Apparently the problem is caused by tolower(), whose behavior is
> affected
> > by current
> > > locale. Under locale tr_TR.UTF-8, tolower('I') returns 'I' rather than
> > 'i', because
> > > lower case of 'I' defined in tr_TR is 'ı' (U+0131).
> >
> > I think at the time the bug was being diagnosed, sqlite3_strnicmp()
> > wasn't being exposed.  I think that does the right thing because it
> > uses the internal UpperToLower table.
> >
> > -scott
> >
> >
> > On Fri, Aug 6, 2010 at 11:11 AM, Richard Hipp <d...@sqlite.org> wrote:
> > > If "ch" is an unsigned char then how is the following unsafe:
> > >
> > > ch = (ch<0x80) ? tolower(ch) : ch
> > >
> > > And why does it need to be changed to
> > >
> > >ch = (ch>='A' && ch<='Z') ? ch - 'A' + 'a' : ch;
> > >
> > > There is only one such instance of code remaining in FTS3 (at
> > > fts3_tokenizer1.c:196) but I want to understand what the issue is
> before
> > I
> > > change it.
> > >
> > > On Fri, Aug 6, 2010 at 1:30 PM, Paweł Hajdan, Jr.
> > > <phajdan...@chromium.org>wrote:
> > >
> > >> On Wed, Aug 4, 2010 at 15:23, Paweł Hajdan, Jr. <
> > phajdan...@chromium.org
> > >> >wrote:
> > >>
> > >> > I'm attaching a suggested patch to fix locale-unsafe usage of
> tolower
> > in
> > >> > FTS code. The goal is to make Chromium closer to the upstream, so if
> > you
> > >> > have a better solution, that's great.
> > >>
> > >>
> > >> Oh, I have just noticed that the mailing list removes all attachments.
> > What
> > >> is the best way to send patches then?
> > >>
> > >> By the way, any suggestions about the Chromium patch I linked to (
> > >>
> > >>
> >
> http://src.chromium.org/viewvc/chrome/trunk/src/third_party/sqlite/safe-tolower.patch?view=markup
> > >> )?
> > >> It seems that it has somehow been fixed in fts3 code. I'm not yet very
> > >> familiar with the SQLite codebase though, so could you point me to the
> > >> fixes?
> > >> ___
> > >> sqlite-users mailing list
> > >> sqlite-users@sqlite.org
> > >> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
> > >>
> > >
> > >
> > >
> > > --
> > > -
> > > D. Richard Hipp
> > > d...@sqlite.org
> > > ___
> > > sqlite-users mailing list
> > > sqlite-users@sqlite.org
> > > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
> > >
> > ___
> > sqlite-users mailing list
> > sqlite-users@sqlite.org
> > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
> >
>
>
>
> --
> -
> D. Richard Hipp
> d...@sqlite.org
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] [PATCH] Fix locale-unsafe usage of tolower (upstreaming Chromium change)

2010-08-06 Thread Paweł Hajdan , Jr . 
On Wed, Aug 4, 2010 at 15:23, Paweł Hajdan, Jr. <phajdan...@chromium.org>wrote:

> I'm attaching a suggested patch to fix locale-unsafe usage of tolower in
> FTS code. The goal is to make Chromium closer to the upstream, so if you
> have a better solution, that's great.


Oh, I have just noticed that the mailing list removes all attachments. What
is the best way to send patches then?

By the way, any suggestions about the Chromium patch I linked to (
http://src.chromium.org/viewvc/chrome/trunk/src/third_party/sqlite/safe-tolower.patch?view=markup)?
It seems that it has somehow been fixed in fts3 code. I'm not yet very
familiar with the SQLite codebase though, so could you point me to the
fixes?
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] [PATCH] Fix locale-unsafe usage of tolower (upstreaming Chromium change)

2010-08-04 Thread Paweł Hajdan , Jr . 
On Wed, Aug 4, 2010 at 16:30, Scott Hess  wrote:

> You should probably pull the current SQLite code and make sure the
> patch even applies, and if not, check to make sure that the problem
> hasn't already been fixed.  ext/fts3 should no longer have the flaw in
> question, as that code was heavily rewritten.


Yeah, the fts3 parts no longer apply and are not included in the patch sent
here. I'm working on the latest (fossil) SQLite codebase.


> The fts1/2 changes would probably be better recast in the mode of the
> fts3 changes.  The relevant comment where things are coming clear was:
>   http://code.google.com/p/chromium/issues/detail?id=15261#c20


Yes, that would be a good direction. However, I'm not yet very familiar with
the SQLite codebase, so I sent something that is in our codebase, doesn't
break tests, and asked for suggestions.

We could then see how SQLite fixed the problem in fts3, and backport that to
Chromium's copy.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] [PATCH] Fix locale-unsafe usage of tolower (upstreaming Chromium change)

2010-08-04 Thread Paweł Hajdan , Jr . 
I'm attaching a suggested patch to fix locale-unsafe usage of tolower in FTS
code. The goal is to make Chromium closer to the upstream, so if you have a
better solution, that's great.

This is upstreaming a Chromium patch
http://src.chromium.org/viewvc/chrome/trunk/src/third_party/sqlite/safe-tolower.patch?view=markup

The issue has been reported, see
http://www.sqlite.org/src/tktview/991789d9f3136a0460dc83a33e815c1aa9757c26

What do you think?
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] [PATCH] Verify number of arguments in icuRegexpFunc

2010-08-03 Thread Paweł Hajdan , Jr . 
On Thu, Jul 29, 2010 at 22:53, Dan Kennedy  wrote:

> Looks like it was fixed for 3.6.22:
>
>   http://www.sqlite.org/src/ci/c34cf23efb


Thank you. I have changed chromium's copy of sqlite to contain the above
version of the code, so we're closer to upstream now.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] [PATCH] Verify number of arguments in icuRegexpFunc

2010-07-29 Thread Paweł Hajdan , Jr . 
I'm attaching a suggested patch to verify number of arguments
in icuRegexpFunc. Please review it.

This is upstreaming of
http://src.chromium.org/viewvc/chrome/trunk/src/third_party/sqlite/icu-regexp.patch?revision=34807=markup

According to the commit message from above, it has been reported earlier to
you.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] problems with running test suite

2010-07-29 Thread Paweł Hajdan , Jr . 
On Thu, Jul 29, 2010 at 16:34, Richard Hipp  wrote:

> The test suite now requires Tcl 8.5.  You appear to be running Tcl 8.4.
>

That's right. After upgrading from tcl8.4 to tcl8.5 the tests run just fine.

Thanks!
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] problems with running test suite

2010-07-29 Thread Paweł Hajdan , Jr . 
I'm planning to contribute some patches (upstreaming patches Chromium
project applies to its local copy of sqlite).

I've checked out the fossil repository, configured and compiled sqlite, and
tried running "make test" (I didn't make any changes).

I'm pasting below the test result and some fossil info. Please let me know
if you need more. This is on Ubuntu Lucid. Do you have any ideas what makes
these tests fail? Should I be doing it some other way?

wal2-10.2.1... Ok
wal2-10.2.2... Ok
wal2-10.2.3... Ok
wal2-11.0... Ok
wal2-11.1.1... Ok
./testfixture: bad field specifier "t"
while executing
"binary scan $I t* L"
(file "./test/wal2.test" line 987)
invoked from within
"source ./test/wal2.test"
invoked from within
"interp eval tinterp $script"
(procedure "slave_test_script" line 24)
invoked from within
"slave_test_script [list source $zFile] "
invoked from within
"time { slave_test_script [list source $zFile] }"
(procedure "slave_test_file" line 14)
invoked from within
"slave_test_file $::testdir/$file"
(procedure "run_tests" line 12)
invoked from within
"run_tests veryquick -presql {} -files {shared3.test bigfile.test
where9.test tkt3419.test sync.test fts1o.test fts2f.test misc2.test
tkt3541.test type..."
("uplevel" body line 1)
invoked from within
"uplevel run_tests $name $::testspec($name)"
(procedure "run_test_suite" line 5)
invoked from within
"run_test_suite veryquick"
(file "./test/veryquick.test" line 16)
make: *** [test] Error 1

real 12m54.106s
user 1m21.060s
sys 0m52.890s

$ fossil status
repository:   /home/phajdan/sqlite/sqlite.fossil
local-root:   /home/phajdan/sqlite/
server-code:  56e02465c398e67515e94b1e1be0a6a41f357093
checkout: 33b1e862ffa7109480cf4a77ceae8aebe98d3eee 2010-07-28 19:17:51
UTC
parent:   a3401d9ee540828f3efd26d89f6b771e0ecb2777 2010-07-28 18:51:27
UTC
tags: trunk

$ fossil timeline
=== 2010-07-28 ===
13:57:49 [4202cdf359] New ticket [ce7c133ea6] Foreign key constraint
fails
 when it should succeed.. (user: drh)
12:17:51 [33b1e862ff] *CURRENT* Get SQLITE_OMIT_VIRTUALTABLE working again
 after being broken by recent changes. (user: drh tags: trunk)
11:51:27 [a3401d9ee5] Fix the dbstatus.test script so that it works
correctly
 on 64-bit machines. (user: drh tags: trunk)
11:35:50 [aa81900153] *MERGE* Merge trunk changes into experimental branch.
 (user: dan tags: experimental)
10:36:12 [ae89777e7f] Improve the accuracy of the Pager heap usage estimate.
 (user: drh tags: trunk)
10:16:41 [2f2fa7dd80] Adjust the shell test script "shell4.test" to account
 for changes in the ".stat on" display format from the previous
 checkin. (user: drh tags: trunk)
10:01:24 [f9adf66ad5] Record the pcache allocation size statistics even for
 pcache overflow allocations. Adjust the wording on one of the stat
 output lines in the shell. (user: drh tags: trunk)
09:05:35 [419ce0ed89] Modify CLI to optionally display "stats". (user:
shaneh
 tags: trunk)
08:52:09 [07abfd5268] Lookaside memory is not used to store schemas. Change
 the SQLITE_DBSATUS_SCHEMA_USED documentation to reflect this fact.
 (user: drh tags: trunk)
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users