Re: [sqlite] Enabling MMAP in Android

[advancenOO]

OK, I know it is not safe when use mmap in Android 7 fuse. 
And do you mean the bug is in fuse kernel framework as you mentioned Android
FUSE 
filesystem driver? Which corresponding to Linux 3.10?
 
By the way, do you have any possible repro steps especially in Android
Sqlite?



--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Enabling MMAP in Android

[advancenOO]

Yeah. Thanks for your suggestion about -DSQLITE_ENABLE_BATCH_ATOMIC_WRITE.
But the version of my sqlite is 3.16.2, so I guess I may use it at a later
time int the future.



--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Enabling MMAP in Android

[advancenOO]

Thanks for the notification about the bugs in MacOS.
So the most important reason for disable SQLITE_MMAP_READWRITE is the bugs
in OS, not the stray pointer in upper applications. Is it right?



--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Enabling MMAP in Android

[Howard Chu]

J Decker wrote:

On Tue, Dec 12, 2017 at 4:35 PM, Howard Chu  wrote:


Martin Raiber wrote:


On 12.12.2017 19:47 Simon Slavin wrote:


On 12 Dec 2017, at 6:27pm, Jens Alfke  wrote:

On Dec 12, 2017, at 5:46 AM, Simon Slavin  wrote:



Before you answer that question, you should know that both Windows and
macOS have been proved to have serious bugs in their memory mapping code.


This has been brought up several times recently, but I’ve never seen
any details given about exactly what was wrong with macOS’s mmap
implementation. Does anyone have a pointer to authoritative information
about this?


See this thread:



This is the thread which led to memory mapping being disabled for
writing on macOS, as discussed here:





There might also be a mmap bug in the Android 7.0 fuse layer:
https://www.mail-archive.com/openldap-its@openldap.org/msg10970.html



There is definitely a bug in Android 7 fuse/mmap. The bug is definitely
not present when bypassing fuse, but only rooted devices can bypass...

My experience was in majority only rooted devices could use fuse.  since

/dev/fuse was rw--- .  (although not all.)


The /sdcard partition is always mounted through fuse because it's a vfat/exfat 
filesystem that doesn't support owner/permission bits, and the fuse driver 
imposes the Android security model on top of it. Many android devices no 
longer include a physical SDcard slot, but still have an internal storage 
partition that's labeled /sdcard and it still behaves this way.


--
  -- Howard Chu
  CTO, Symas Corp.   http://www.symas.com
  Director, Highland Sun http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Enabling MMAP in Android

[J Decker]

On Tue, Dec 12, 2017 at 4:35 PM, Howard Chu  wrote:

> Martin Raiber wrote:
>
>> On 12.12.2017 19:47 Simon Slavin wrote:
>>
>>> On 12 Dec 2017, at 6:27pm, Jens Alfke  wrote:
>>>
>>> On Dec 12, 2017, at 5:46 AM, Simon Slavin  wrote:

> Before you answer that question, you should know that both Windows and
> macOS have been proved to have serious bugs in their memory mapping code.
>
 This has been brought up several times recently, but I’ve never seen
 any details given about exactly what was wrong with macOS’s mmap
 implementation. Does anyone have a pointer to authoritative information
 about this?

>>> See this thread:
>>>
>>> >> itable-catalog-corruption-td85620.html>
>>>
>>> This is the thread which led to memory mapping being disabled for
>>> writing on macOS, as discussed here:
>>>
>>> >> and-PRAGMA-fullfsync-on-macOS-td95366i20.html>
>>>
>>
>> There might also be a mmap bug in the Android 7.0 fuse layer:
>> https://www.mail-archive.com/openldap-its@openldap.org/msg10970.html
>>
>
> There is definitely a bug in Android 7 fuse/mmap. The bug is definitely
> not present when bypassing fuse, but only rooted devices can bypass...
>
> My experience was in majority only rooted devices could use fuse.  since
/dev/fuse was rw--- .  (although not all.)

> --
>   -- Howard Chu
>   CTO, Symas Corp.   http://www.symas.com
>   Director, Highland Sun http://highlandsun.com/hyc/
>   Chief Architect, OpenLDAP  http://www.openldap.org/project/
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Enabling MMAP in Android

[Howard Chu]

Martin Raiber wrote:

On 12.12.2017 19:47 Simon Slavin wrote:

On 12 Dec 2017, at 6:27pm, Jens Alfke  wrote:


On Dec 12, 2017, at 5:46 AM, Simon Slavin  wrote:

Before you answer that question, you should know that both Windows and macOS 
have been proved to have serious bugs in their memory mapping code.

This has been brought up several times recently, but I’ve never seen any 
details given about exactly what was wrong with macOS’s mmap implementation. 
Does anyone have a pointer to authoritative information about this?

See this thread:



This is the thread which led to memory mapping being disabled for writing on 
macOS, as discussed here:




There might also be a mmap bug in the Android 7.0 fuse layer:
https://www.mail-archive.com/openldap-its@openldap.org/msg10970.html


There is definitely a bug in Android 7 fuse/mmap. The bug is definitely not 
present when bypassing fuse, but only rooted devices can bypass...


--
  -- Howard Chu
  CTO, Symas Corp.   http://www.symas.com
  Director, Highland Sun http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Enabling MMAP in Android

[Howard Chu]

Jens Alfke wrote:


I’m skeptical about mmap being broken on Macs, since there are other production 
databases such as Realm* that use it heavily. (Though I am not sure whether 
Realm uses writeable mappings.)

—Jens

* and LMDB, but I am not sure if LMDB is in use on macOS.


LMDB is in common use on MacOS, no issues have been reported. On iOS there's 
this nagging problem that the virtual address space is still limited to 4GB, 
even on 64bit systems.


--
  -- Howard Chu
  CTO, Symas Corp.   http://www.symas.com
  Director, Highland Sun http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Enabling MMAP in Android

[Jens Alfke]

> On Dec 12, 2017, at 10:47 AM, Simon Slavin  wrote:
> 
> See this thread:
> 
>   
> >

That thread reported SQLite database corruption when writeable mmap was 
enabled, but didn't identify what caused it. There was also the following 
observation by Török Edwin, which no one ever replied to:

>> I notice that SQLite doesn't use msync, and msync(2) says 'Without use of 
>> this call there is no guarantee that changes are written back before 
>> munmap(2) is called'. 
>> Is f(data)sync enough to ensure changed mmap pages are written to the disk? 


That sounds pretty suspicious to me; maybe the problem was just that sqlite 
needed to call msync?

> This is the thread which led to memory mapping being disabled for writing on 
> macOS, as discussed here:
> 
>   
> >

Yup, and I was heavily involved in that thread :) Again, there was only 
circumstantial evidence that mmap was causing problems; no one seems to have 
gotten to the bottom of it.

I’m skeptical about mmap being broken on Macs, since there are other production 
databases such as Realm* that use it heavily. (Though I am not sure whether 
Realm uses writeable mappings.)

—Jens

* and LMDB, but I am not sure if LMDB is in use on macOS.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Enabling MMAP in Android

[Martin Raiber]

On 12.12.2017 19:47 Simon Slavin wrote:
> On 12 Dec 2017, at 6:27pm, Jens Alfke  wrote:
>
>> On Dec 12, 2017, at 5:46 AM, Simon Slavin  wrote:
>>> Before you answer that question, you should know that both Windows and 
>>> macOS have been proved to have serious bugs in their memory mapping code.
>> This has been brought up several times recently, but I’ve never seen any 
>> details given about exactly what was wrong with macOS’s mmap implementation. 
>> Does anyone have a pointer to authoritative information about this?
> See this thread:
>
> 
>
> This is the thread which led to memory mapping being disabled for writing on 
> macOS, as discussed here:
>
> 

There might also be a mmap bug in the Android 7.0 fuse layer:
https://www.mail-archive.com/openldap-its@openldap.org/msg10970.html


___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Enabling MMAP in Android

[Simon Slavin]

On 12 Dec 2017, at 6:27pm, Jens Alfke  wrote:

> On Dec 12, 2017, at 5:46 AM, Simon Slavin  wrote:

>> Before you answer that question, you should know that both Windows and macOS 
>> have been proved to have serious bugs in their memory mapping code.
> 
> This has been brought up several times recently, but I’ve never seen any 
> details given about exactly what was wrong with macOS’s mmap implementation. 
> Does anyone have a pointer to authoritative information about this?

See this thread:



This is the thread which led to memory mapping being disabled for writing on 
macOS, as discussed here:



Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Enabling MMAP in Android

[Jens Alfke]

> On Dec 12, 2017, at 5:46 AM, Simon Slavin  wrote:
> 
> The stray pointer or array overrun is not in the SQLite code.  It’s in the 
> implementation of memory mapping built into the OS.  So you need to ask 
> yourself whether you trust your OS.

This argument (which is a standard one against using writeable mmap) is about 
_application_ code issuing stray writes. This wasn’t a topic about OS bugs.

I’m not sure I buy the argument anyway. It’s already possible for app code to 
write into SQLite’s buffer cache, which can then be written back to disk, 
corrupting the file. And the buffer cache is located in the malloc heap, which 
is close to application heap blocks, while a mmap’ed database file is likely to 
be somewhere way far away in address space.

> Before you answer that question, you should know that both Windows and macOS 
> have been proved to have serious bugs in their memory mapping code.

This has been brought up several times recently, but I’ve never seen any 
details given about exactly what was wrong with macOS’s mmap implementation. 
Does anyone have a pointer to authoritative information about this?

—Jens
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Enabling MMAP in Android

[Richard Hipp]

On 12/12/17, advancenOO  wrote:
> Um, stray pointer or array overrun in upper applications will be treated as
> BUG in my system. And I believe all these bugs could be fixed before using
> sqlite.

You have a very optimistic view of application software :-)

Look - if you are interested in maximizing performance, you need to
use the latest Linux kernel with the latest F2FS filesystem, then
compile SQLite with the -DSQLITE_ENABLE_BATCH_ATOMIC_WRITE.  That
combination will get you way more performance boost than you will ever
find by playing games with mmap().
-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Enabling MMAP in Android

[Simon Slavin]

On 12 Dec 2017, at 1:32pm, advancenOO  wrote:

> Um, stray pointer or array overrun in upper applications will be treated as
> BUG in my system. And I believe all these bugs could be fixed before using
> sqlite.
> So can I safely enable SQLITE_MMAP_READWRITE directly if I can ignore the
> possibility of the BUG?

The stray pointer or array overrun is not in the SQLite code.  It’s in the 
implementation of memory mapping built into the OS.  So you need to ask 
yourself whether you trust your OS.

Before you answer that question, you should know that both Windows and macOS 
have been proved to have serious bugs in their memory mapping code.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Enabling MMAP in Android

[advancenOO]

I am a developer of system, not application. So I have nothing to do with the
SQL or indices even though I know these may cut orders of magnitude from
execution times. 

And that is why I use speedtest1.c of Sqlite to evaluate the performance of
my system.



--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Enabling MMAP in Android

[advancenOO]

Um, stray pointer or array overrun in upper applications will be treated as
BUG in my system. And I believe all these bugs could be fixed before using
sqlite.
So can I safely enable SQLITE_MMAP_READWRITE directly if I can ignore the
possibility of the BUG?

But I notice that Sqlite may not call msync() in mmap, calls fsync instead. 
So I am wondering if it is safe to use SQLITE_MMAP_READWRITE to write to
mmapped memory as msync() is not called.



--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Enabling MMAP in Android

[Richard Hipp]

On 12/11/17, advancenOO  wrote:
>
> At the same time, Mmap() is only used for reading by default, not writing.
> But why?

If the database is memory-mapped writable, then a stray pointer or
array overrun in the application (NB: not in SQLite - in the
application that uses SQLite) could easily overwrite part of the
database file and corrupt the database file.

It is also possible for this to happen without the use of writable
memory-maps, if the application corrupts internal SQLite data
structures and a subsequent COMMIT occurs.  But that is much less
likely, as SQLite would probably detect the situation before the
COMMIT completes, thus leaving the database intact.

-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Enabling MMAP in Android

[Graham Holden]

Have you actually got speed issues, or is this just a case of "this should be 
faster"? If you DO have speed issues, my experience from reading this list is 
that most can be improved with better SQL, better indices, or better interplay 
between readers and writers. Only if those have been exhausted should you be 
looking at compile options.
Graham.


Sent from my Samsung Galaxy S7 - powered by Three
 Original message From: advancenOO 
<haveagoodtime2...@gmail.com> Date: 11/12/2017  14:04  (GMT+00:00) To: 
sqlite-users@mailinglists.sqlite.org Subject: [sqlite] Enabling MMAP in Android 
I find MMAP is able to be used for db files(getPageMMap()) but is not enabled
by default.
I wonder if it is recommended to use MMAP in Android? As I think MMAP is one
way to improve the speed performance(I use speedtest1.c to evaluate).

At the same time, Mmap() is only used for reading by default, not writing.
But why? And in what situation I can compile with the
-DSQLITE_MMAP_READWRITE option?

By the way, is there any other recommended compile-options? I find some
references in  http://www.sqlite.org/compile.html
<http://www.sqlite.org/compile.html>  , but I am wondering if there is any
other method to enhance performance?

Thanks for your help.



--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Enabling MMAP in Android

[advancenOO]

I find MMAP is able to be used for db files(getPageMMap()) but is not enabled
by default.
I wonder if it is recommended to use MMAP in Android? As I think MMAP is one
way to improve the speed performance(I use speedtest1.c to evaluate).

At the same time, Mmap() is only used for reading by default, not writing.
But why? And in what situation I can compile with the
-DSQLITE_MMAP_READWRITE option?

By the way, is there any other recommended compile-options? I find some
references in  http://www.sqlite.org/compile.html
  , but I am wondering if there is any
other method to enhance performance?

Thanks for your help.



--
Sent from: http://sqlite.1065341.n5.nabble.com/
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users