Re: [sqlite] SQLite 3.7.17 preview - 2x faster?

[Richard Hipp]

On Mon, Apr 8, 2013 at 8:20 AM, Max Vlasov  wrote:

> Richard,
>
> It makes sense, but I see here some possibility of future confusion.
> Correct me if I'm wrong. Currently if I have a vfs that requires special
> preparations (for example, decompression), I have two choices, either
> provide V2 interface or emulate memory-mapping by allocating my own blocks
> of memory in xFetch and deallocating in xUnfetch. If future V4 IO routines
>

You an implement xFetch to always return NULL:

int xFetch(sqlie3_file *pNotUsed1, i64 notUsed2, int notUsed3, void
**pp){
  *pp = 0;
  return SQLITE_OK;
}

Then SQLite will always fallback to doing plain old xRead and xWrite.



> introduce something new, one will not have the first option. So anyone in
> the future should be aware that there are two points where data can be
> needed and since one expects filling previously allocated block and another
> expects pointer to the data, the complexity of understanding will grow. Or
> is there a simple way to disable xFetch/xUnfetch on the VFS level?
>
> Max
>
>
>
>
>
> On Mon, Apr 8, 2013 at 3:33 PM, Richard Hipp  wrote:
>
> > On Mon, Apr 8, 2013 at 6:12 AM, Max Vlasov  wrote:
> >
> > > But I also noticed that if I provide
> > > version 2 of vfs, I won't get benefits of file-mapping
> > >
> >
> > That's how we implement backwards compatibility to legacy VFSes.
> >
> > --
> > D. Richard Hipp
> > d...@sqlite.org
> > ___
> > sqlite-users mailing list
> > sqlite-users@sqlite.org
> > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
> >
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQLite 3.7.17 preview - 2x faster?

[Max Vlasov]

Richard,

It makes sense, but I see here some possibility of future confusion.
Correct me if I'm wrong. Currently if I have a vfs that requires special
preparations (for example, decompression), I have two choices, either
provide V2 interface or emulate memory-mapping by allocating my own blocks
of memory in xFetch and deallocating in xUnfetch. If future V4 IO routines
introduce something new, one will not have the first option. So anyone in
the future should be aware that there are two points where data can be
needed and since one expects filling previously allocated block and another
expects pointer to the data, the complexity of understanding will grow. Or
is there a simple way to disable xFetch/xUnfetch on the VFS level?

Max





On Mon, Apr 8, 2013 at 3:33 PM, Richard Hipp  wrote:

> On Mon, Apr 8, 2013 at 6:12 AM, Max Vlasov  wrote:
>
> > But I also noticed that if I provide
> > version 2 of vfs, I won't get benefits of file-mapping
> >
>
> That's how we implement backwards compatibility to legacy VFSes.
>
> --
> D. Richard Hipp
> d...@sqlite.org
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] 64bit compatibility warnings

[Richard Hipp]

On Sun, Apr 7, 2013 at 1:06 PM, Alexandr Němec  wrote:

>
> Line 6766   u.bc.r.flags = (u16)(UNPACKED_INCRKEY * (1 & (u.bc.oc -
> OP_SeekLt)));  WARNING: conversion from 'u16' to 'u8', possible
> loss of data
> Line 71133 iBuf = p->iReadOff % p->nBuffer;
> WARNING: conversion from 'i64' to 'int', possible loss of data
> Line 71209 iBuf = p->iReadOff % p->nBuffer;
>  WARNING: conversion from 'i64' to 'int', possible loss of data
> Line 71286 iBuf = iStart % nBuf;
> WARNING: conversion from 'i64' to 'int', possible loss of data
> Line 71574 p->iBufEnd = p->iBufStart = (iStart % nBuf); WARNING:
> conversion from 'i64' to 'int', possible loss of data
>

The first warning is harmless and results from a prior datatype change.
Dan has already fixed that one.  The other four appear to be due to an MSVC
compiler bug, since every (i64%int) operation will always yield a value
that can fit in an int, no?
-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQLite 3.7.17 preview - 2x faster?

[Richard Hipp]

On Mon, Apr 8, 2013 at 6:12 AM, Max Vlasov  wrote:

> But I also noticed that if I provide
> version 2 of vfs, I won't get benefits of file-mapping
>

That's how we implement backwards compatibility to legacy VFSes.

-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] 64bit compatibility warnings

[Roland Hughes]

On Mon, 2013-04-08 at 09:31 +0200, Kees Nuyt wrote:

> On Mon, 08 Apr 2013 08:39:49 +0200, Alexandr N?mec 
> wrote:
> 
> > Hi all,
> > 
> > thanks for your replies, but unfortunately they did not answer
> > my original question whether these warnings are harmless and
> > can be ignored or not. These warnings reported by the VS C++
> > compiler are about "possible loss of data", so it is a
> > situation when a "int64" expression result is assigned to an
> > "int" variable for example. In such cases these warnings are
> > very legitimate. If such an assignment is the real intention
> > of the programmer, an explicit (int) typecast should be added,
> > because it will
> > 
> >- tell to the rest of the world, that the programmer knows
> >  what he is doing, ie. he really wants to "truncate" the result,
> 
> The programmers know what they are doing.
> As  tells, they are harmless if all
> tests scripts succeed. The test scripts are run before every SQLite
> release. SQLite is not released if a test fails. So, the warnings can be
> ignored.
> 


Or, it could be the tests simply don't exercise those possibilities.
I've been in IT far too long to trust that a test suite is "complete".


-- 
Roland Hughes, President
Logikal Solutions
(630)-205-1593

http://www.theminimumyouneedtoknow.com
http://www.infiniteexposure.net

No U.S. troops have ever lost their lives defending our ethanol
reserves.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQLite 3.7.17 preview - 2x faster?

[Max Vlasov]

Dan, I see, seems like this works. But I also noticed that if I provide
version 2 of vfs, I won't get benefits of file-mapping (I assume because my
query that worked 9 seconds for file-mapping now back 27 seconds). I'm not
sure this is right, but calling sqlite original xRead very abstract by its
nature, so nobody ever expected particular implementation from sqlite and
if file mapping is on, sqlite could use file-mapping even if user provided
ver.2 of io routines.

In other word, if sqlite doesn't expect special knowledge of file mapping
existence for vfs-free client then it should not expect this from client
that uses vfs

Max


On Mon, Apr 8, 2013 at 1:56 PM, Dan Kennedy  wrote:

> On 04/08/2013 04:40 PM, Max Vlasov wrote:
>
>> On Mon, Apr 8, 2013 at 1:23 PM, Dan Kennedy 
>> wrote:
>>
>>
>>> Right. But a VFS is not obliged to support the new xFetch() and
>>> xUnfetch() methods (used to read data from a memory mapped file).
>>> And if it doesn't, SQLite will use xRead() exclusively.
>>>
>>> It always uses xWrite() to write - whether mmap is enabled or not.
>>>
>>>
>>>  Great, what is the correct way of not providing fetch procedures? Maybe
>> I
>> did something wrong?
>> - If I call original xFetch/xUnfetch from mine - no xRead called,
>> - if I provide Nil as the function address, then I get Access violation
>> () so probably sqlite tries to call it anyway.
>> - If return SQLITE_Error from xFetch, xUnfetch, I get sqlite logic error.
>>
>
> Set the iVersion field of your sqlite3_io_methods struct to 2 (not 3).
>
> The idea is that if you have an existing VFS, it should keep working
> as is without any modifications. If you find this is not the case, it
> may be a bug.
>
> Dan.
>
>
> __**_
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-**bin/mailman/listinfo/sqlite-**users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQLite 3.7.17 preview - 2x faster?

[Dan Kennedy]

On 04/08/2013 04:40 PM, Max Vlasov wrote:

On Mon, Apr 8, 2013 at 1:23 PM, Dan Kennedy  wrote:



Right. But a VFS is not obliged to support the new xFetch() and
xUnfetch() methods (used to read data from a memory mapped file).
And if it doesn't, SQLite will use xRead() exclusively.

It always uses xWrite() to write - whether mmap is enabled or not.



Great, what is the correct way of not providing fetch procedures? Maybe I
did something wrong?
- If I call original xFetch/xUnfetch from mine - no xRead called,
- if I provide Nil as the function address, then I get Access violation
() so probably sqlite tries to call it anyway.
- If return SQLITE_Error from xFetch, xUnfetch, I get sqlite logic error.


Set the iVersion field of your sqlite3_io_methods struct to 2 (not 3).

The idea is that if you have an existing VFS, it should keep working
as is without any modifications. If you find this is not the case, it
may be a bug.

Dan.

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQLite 3.7.17 preview - 2x faster?

[Max Vlasov]

On Mon, Apr 8, 2013 at 1:23 PM, Dan Kennedy  wrote:

>
> Right. But a VFS is not obliged to support the new xFetch() and
> xUnfetch() methods (used to read data from a memory mapped file).
> And if it doesn't, SQLite will use xRead() exclusively.
>
> It always uses xWrite() to write - whether mmap is enabled or not.
>
>
Great, what is the correct way of not providing fetch procedures? Maybe I
did something wrong?
- If I call original xFetch/xUnfetch from mine - no xRead called,
- if I provide Nil as the function address, then I get Access violation
() so probably sqlite tries to call it anyway.
- If return SQLITE_Error from xFetch, xUnfetch, I get sqlite logic error.

Max
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQLite 3.7.17 preview - 2x faster?

[Dan Kennedy]

On 04/08/2013 03:22 PM, Max Vlasov wrote:

On Sun, Apr 7, 2013 at 2:12 PM, Max Vlasov  wrote:





On Thu, Apr 4, 2013 at 4:02 PM, Richard Hipp  wrote:


We would like to encourage people to try out the new code and
report both success and failure.




Not particulary about this draft version, but about my experience with
memory mapped files on Windows If you don't mind .
...

I don't know whether such scenario is possible with sqlite.



Finally I did some tests and didn't not notice anything like that with
creating tables, probably because memory-mapping is not currently for
inserting and updating, so the problem I described seems like not actual.

As for general queries, I have mixed feeling. At least one of my queries
worked 9 seconds on 3.7.17 ddraft instead of 27 seconds with 3.7.16.1. So
the speed progress can be very noticeable in some cases. But as I see the
VFS stopped working transparently in this case. Shouldn't it be so that
xRead and probably xWrite still be in the chain of callings, just doing
memcpy from file-mapping regions instead of calling file routines?
Otherwise many existing vfs filtering solutions (encryption, compression)
won't longer work when memory-mapping is on.


Right. But a VFS is not obliged to support the new xFetch() and
xUnfetch() methods (used to read data from a memory mapped file).
And if it doesn't, SQLite will use xRead() exclusively.

It always uses xWrite() to write - whether mmap is enabled or not.





___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] SQLite 3.7.17 preview - 2x faster?

[Max Vlasov]

On Sun, Apr 7, 2013 at 2:12 PM, Max Vlasov  wrote:

>
>
>
> On Thu, Apr 4, 2013 at 4:02 PM, Richard Hipp  wrote:
>
>> We would like to encourage people to try out the new code and
>> report both success and failure.
>>
>
>
> Not particulary about this draft version, but about my experience with
> memory mapped files on Windows If you don't mind .
> ...
>
> I don't know whether such scenario is possible with sqlite.
>
>
Finally I did some tests and didn't not notice anything like that with
creating tables, probably because memory-mapping is not currently for
inserting and updating, so the problem I described seems like not actual.

As for general queries, I have mixed feeling. At least one of my queries
worked 9 seconds on 3.7.17 ddraft instead of 27 seconds with 3.7.16.1. So
the speed progress can be very noticeable in some cases. But as I see the
VFS stopped working transparently in this case. Shouldn't it be so that
xRead and probably xWrite still be in the chain of callings, just doing
memcpy from file-mapping regions instead of calling file routines?
Otherwise many existing vfs filtering solutions (encryption, compression)
won't longer work when memory-mapping is on.

Max
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] 64bit compatibility warnings

[Kees Nuyt]

On Mon, 08 Apr 2013 08:39:49 +0200, Alexandr N?mec 
wrote:

> Hi all,
> 
> thanks for your replies, but unfortunately they did not answer
> my original question whether these warnings are harmless and
> can be ignored or not. These warnings reported by the VS C++
> compiler are about "possible loss of data", so it is a
> situation when a "int64" expression result is assigned to an
> "int" variable for example. In such cases these warnings are
> very legitimate. If such an assignment is the real intention
> of the programmer, an explicit (int) typecast should be added,
> because it will
> 
>- tell to the rest of the world, that the programmer knows
>  what he is doing, ie. he really wants to "truncate" the result,

The programmers know what they are doing.
As  tells, they are harmless if all
tests scripts succeed. The test scripts are run before every SQLite
release. SQLite is not released if a test fails. So, the warnings can be
ignored.

>- eliminate compiler warnings of this type.
>
> There are only 5 warning of this type in the entire code base,
> so that should be an easy fix.

Warnings are fixed eventually
( e.g.  ), 
but with a lower priority than making sure all tests succeed.
 
>Alex

-- 
Groet, Cordialement, Pozdrawiam, Regards,

Kees Nuyt

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] 64bit compatibility warnings

[Alexandr Němec]

Hi all,
 
thanks for your replies, but unfortunately they did not answer my original question whether these warnings 
are harmless and can be ignored or not. These warnings reported by the VS C++ compiler are about 
"possible loss of data", so it is a situation when a "int64" expression result is 
assigned to an "int" variable for example. In such cases these warnings are very legitimate. If 
such an assignment is the real intention of the programmer, an explicit (int) typecast should be added, 
because it will
 
- tell to the rest of the world, that the programmer knows what he is doing, ie. he 
really wants to "truncate" the result,
- eliminate compiler warnings of this type.
 
There are only 5 warning of this type in the entire code base, so that should 
be an easy fix.
 
Alex
__

Od: "Simon Slavin" 
Komu: , General Discussion of SQLite Database 

Datum: 07.04.2013 20:05
Předmět: Re: [sqlite] 64bit compatibility warnings



On 7 Apr 2013, at 6:26pm, f...@cetussoft.com wrote:


I think that in general it might be a good idea to update the code to
not produce any 64 bit portability warnings, so that we know for sure,
that compiling 64 bit does not introduce any 64 bit side effects or possible 
bugs.


...as long as doing so does not break 32-bit code...


One of the problems with compiler warnings is that different compilers generate warnings about 
different things.  So it's not "Get rid of 64-bit warnings" it's "Try to get rid of 
64--bit warnings in GCC without creating more of them in LCC and Visual C+, and try to get rid of 
them with all the different directives most people use most of the time.".  And it turns out 
that this is very difficult:

>

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users 


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users