Re: [sqlite] Why MMAP return ENOMEM in SQLite?

[Scott Robison]

On Aug 31, 2016 5:29 AM, "sanhua.zh"  wrote:
>
> BTW, what do you think if I mapseparatly instead of the whole db file, ...

I suspect that it wouldn't really help you much, if any.

One, there is overhead in making that many system calls to map a bunch of 4
MiB buffers.

Two, once you've mapped that many buffers, you now have to determine where
each part of your file is in memory before accessing it. You've lost the
benefit of easy address calculations and now have to perform an indirect
lookup, first finding the correct page, next computing the offset into that
page. It would be potentially worse if a piece of data every spanned two
pages, but I doubt that would be an issue for SQLite.

Three, I have no idea what limits exist for mapping pages at the OS level
per process, but it wouldn't surprise me if maybe this exceeded something
like that.

Four, the one potential benefit is that mapping the pages would avoid the
penalty of copying read data from a kernel buffer to your user space
buffer. I think the additional complexity and my other reasons above make
it a less than ideal solution.

If your problem space requires higher speed access to the data than SQLite
is capable of delivering, it seems to me that you'd be better off with a
data storage solution tailored to your requirements. I don't make that
suggestion lightly, and would have to be really desperate for performance
to do it myself, but a more specialized solution can gain performance at
the expense of not being as generally useful. Even then, it might be
difficult to improve on SQLite, a very optimized library for data storage,
manipulation, and access.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Why MMAP return ENOMEM in SQLite?

[Simon Slavin]

On 31 Aug 2016, at 12:28pm, sanhua.zh  wrote:

> BTW, what do you think if I mapseparatly instead of the whole db file, which 
> is the way I mentioned before ?

Sorry but I have no experience with that.  I hope other people can advise you.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Why MMAP return ENOMEM in SQLite?

[sanhua.zh]

Unless your users have complained about speed, this does not matter. There is 
no need to make everything happen as fast /as possible/. Try using just 
standard SQLite and find out if it is fast /enough/. Then you don't have to 
spend lots of programming time learning tiny little details about one small 
piece of a software library.
I have my monitor system to check the lag and delay. I can make sure that I 
must make it faster.



PRAGMA mmap_size=0x7fff
I am using SQLite for a long time, and be familiar with the most of the source 
code. So you have no need to worry about the basic things.


BTW, what do you think if I mapseparatly instead of the whole db file, which is 
the way I mentioned before ?




原始邮件
发件人:Simon slavinslav...@bigfraud.org
收件人:SQLite mailing listsqlite-us...@mailinglists.sqlite.org
发送时间:2016年8月31日(周三) 18:04
主题:Re: [sqlite] Why MMAP return ENOMEM in SQLite?


On 31 Aug 2016, at 9:39am, sanhua.zh sanhua...@foxmail.com wrote:  Yes. even on 
iDevice, [mmap] can get faster than sequential I/O. Sometime it will be twice 
faster. Unless your users have complained about speed, this does not matter. 
There is no need to make everything happen as fast /as possible/. Try using 
just standard SQLite and find out if it is fast /enough/. Then you don't have 
to spend lots of programming time learning tiny little details about one small 
piece of a software library.  Simon Won't the standard SQLite API do the job 
well enough ?  I did use the standard SQLite API. It maps the whole db file 
into address space. It does this only if you have told it to. According to 
https://www.sqlite.org/mmap.html "To activate memory-mapped I/O, use the 
mmap_size pragma and set the mmap_size to some large number" So do not do this 
and SQLite will stop trying to memory map the file. But it is possible that you 
are using a SQLite library which has a default memory map size. Can you execute 
the command "PRAGMA mmap_size" and tell what it outputs ? If the result is not 
zero or blank, you can disable memory mapping using the command "PRAGMA 
mmap_size=0" after you have opened the database. Please try this and see 
whether your application still crashes. Simon. 
___ sqlite-users mailing list 
sqlite-users@mailinglists.sqlite.org 
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Why MMAP return ENOMEM in SQLite?

[Simon Slavin]

On 31 Aug 2016, at 9:39am, sanhua.zh  wrote:

> Yes. even on iDevice, [mmap] can get faster than sequential I/O. Sometime it 
> will be twice faster.

Unless your users have complained about speed, this does not matter.  There is 
no need to make everything happen as fast /as possible/.  Try using just 
standard SQLite and find out if it is fast /enough/.  Then you don't have to 
spend lots of programming time learning tiny little details about one small 
piece of a software library.

> Simon Won't the standard SQLite API do the job well enough ?
> I did use the standard SQLite API. It maps the whole db file into address 
> space.

It does this only if you have told it to.  According to



"To activate memory-mapped I/O, use the mmap_size pragma and set the mmap_size 
to some large number"

So do not do this and SQLite will stop trying to memory map the file.

But it is possible that you are using a SQLite library which has a default 
memory map size.  Can you execute the command "PRAGMA mmap_size" and tell what 
it outputs ?

If the result is not zero or blank, you can disable memory mapping using the 
command "PRAGMA mmap_size=0" after you have opened the database.  Please try 
this and see whether your application still crashes.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Why MMAP return ENOMEM in SQLite?

[sanhua.zh]

OK, I get your idea.
Thanks for your advice. :)


原始邮件
发件人:Stephan bealsgb...@googlemail.com
收件人:SQLite mailing listsqlite-us...@mailinglists.sqlite.org
发送时间:2016年8月31日(周三) 17:09
主题:Re: [sqlite] Why MMAP return ENOMEM in SQLite?


On Wed, Aug 31, 2016 at 11:03 AM, sanhua.zh sanhua...@foxmail.com wrote:  Why 
do you think it will corrupt the database?  Can you give me more explainations 
or examples?  It's only my intuition - i don't have a concrete example. sqlite 
and ios are "well-oiled machines." They do their jobs and they do it well. If 
you start interfering with that, trying to take over or abuse their 
responsibilities because you think you can do it better, you will _eventually_ 
run into problems. In my experience, the chances of a back-fire when trying to 
push software beyond what it's designed to do are very high. You explicitly 
want to add complexity to an already complex system. Additional complexity 
almost always comes with a higher bug rate. A telephone is _not_ a 
high-performance computing platform, but a _convenience_ platform. Whether a db 
operation takes 10ms or 800ms should, for such platforms, be irrelevant. i 
_suspect_ that you are overestimating the impact of your perceived performance 
problem on the end users. But that's all just my opinion based on experience - 
i have no facts or statistics to back it up. Maybe it will work well for you. 
-- - stephan beal http://wanderinghorse.net/home/stephan/ "Freedom is 
sloppy. But since tyranny's the only guaranteed byproduct of those who insist 
on a perfect world, freedom will have to do." -- Bigby Wolf 
___ sqlite-users mailing list 
sqlite-users@mailinglists.sqlite.org 
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Why MMAP return ENOMEM in SQLite?

[Stephan Beal]

On Wed, Aug 31, 2016 at 11:03 AM, sanhua.zh  wrote:

> Why do you think it will corrupt the database?
> Can you give me more explainations or examples?
>

It's only my intuition - i don't have a concrete example. sqlite and ios
are "well-oiled machines." They do their jobs and they do it well. If you
start interfering with that, trying to take over or abuse their
responsibilities because you think you can do it better, you will
_eventually_ run into problems. In my experience, the chances of a
back-fire when trying to push software beyond what it's designed to do are
very high.

You explicitly want to add complexity to an already complex system.
Additional complexity almost always comes with a higher bug rate. A
telephone is _not_ a high-performance computing platform, but a
_convenience_ platform. Whether a db operation takes 10ms or 800ms should,
for such platforms, be irrelevant. i _suspect_ that you are overestimating
the impact of your perceived performance problem on the end users.

But that's all just my opinion based on experience - i have no facts or
statistics to back it up. Maybe it will work well for you.

-- 
- stephan beal
http://wanderinghorse.net/home/stephan/
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Why MMAP return ENOMEM in SQLite?

[sanhua.zh]

Why do you think it will corrupt the database?
Can you give me more explainations or examples?




原始邮件
发件人:Stephan bealsgb...@googlemail.com
收件人:SQLite mailing listsqlite-us...@mailinglists.sqlite.org
发送时间:2016年8月31日(周三) 16:57
主题:Re: [sqlite] Why MMAP return ENOMEM in SQLite?


On Wed, Aug 31, 2016 at 10:55 AM, sanhua.zh sanhua...@foxmail.com wrote:  Yes, 
[mmap] can be larger than physical memory.  Indeed, my mistake.  And what do 
you think about the new mapping way I mentioned ?  i think it's a "huge can of 
worms" - it's asking for more, bigger problems than the perceived performance 
problems you have right now. The chances that something breaks with that 
approach is, i suspect, very high. You will eventually corrupt a database and 
then write back to the list to ask why that approach corrupted it. -- - 
stephan beal http://wanderinghorse.net/home/stephan/ "Freedom is sloppy. But 
since tyranny's the only guaranteed byproduct of those who insist on a perfect 
world, freedom will have to do." -- Bigby Wolf 
___ sqlite-users mailing list 
sqlite-users@mailinglists.sqlite.org 
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Why MMAP return ENOMEM in SQLite?

[Stephan Beal]

On Wed, Aug 31, 2016 at 10:55 AM, sanhua.zh  wrote:

> Yes, [mmap] can be larger than physical memory.
>

Indeed, my mistake.


> And what do you think about the new mapping way I mentioned ?
>

i think it's a "huge can of worms" - it's asking for more, bigger problems
than the perceived performance problems you have right now. The chances
that something breaks with that approach is, i suspect, very high. You will
eventually corrupt a database and then write back to the list to ask why
that approach corrupted it.

-- 
- stephan beal
http://wanderinghorse.net/home/stephan/
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Why MMAP return ENOMEM in SQLite?

[sanhua.zh]

Yes, [mmap] can be larger than physical memory.
And what do you think about the new mapping way I mentioned ?




原始邮件
发件人:Stephan bealsgb...@googlemail.com
收件人:SQLite mailing listsqlite-us...@mailinglists.sqlite.org
发送时间:2016年8月31日(周三) 16:53
主题:Re: [sqlite] Why MMAP return ENOMEM in SQLite?


On Wed, Aug 31, 2016 at 10:43 AM, Stephan Beal sgb...@googlemail.com wrote:  On 
Wed, Aug 31, 2016 at 10:39 AM, sanhua.zh sanhua...@foxmail.com wrote:   In my 
testcase, I can only [mmap] a db file at most 1.4GB size. But in  this new way, 
I can map a file at most 3.2GB. The test device is iPhone 6S.According to 
google, the iPhone 6s only has 2GB of RAM, so you can't  memmap 3.2G.  
Correction: you can map larger than physical memory, up to limits set by the OS 
environment. In your case, see: 
http://stackoverflow.com/questions/9184773/is-there-a-practical-limit-on-the-number-of-memory-mapped-files-in-ios
  In any case, as Simone said, _other apps_ require memory of their own as  
well.  Correction 2: Simon, not Simone (my roommate's name, so i type the 'e' 
out of habit) -- - stephan beal http://wanderinghorse.net/home/stephan/ 
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of those 
who insist on a perfect world, freedom will have to do." -- Bigby Wolf 
___ sqlite-users mailing list 
sqlite-users@mailinglists.sqlite.org 
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Why MMAP return ENOMEM in SQLite?

[Stephan Beal]

On Wed, Aug 31, 2016 at 10:43 AM, Stephan Beal 
wrote:

> On Wed, Aug 31, 2016 at 10:39 AM, sanhua.zh  wrote:
>
>> In my testcase, I can only [mmap] a db file at most 1.4GB size. But in
>> this new way, I can map a file at most 3.2GB. The test device is iPhone 6S.
>>
>
> According to google, the iPhone 6s only has 2GB of RAM, so you can't
> memmap 3.2G.
>

Correction: you can map larger than physical memory, up to limits set by
the OS environment. In your case, see:

http://stackoverflow.com/questions/9184773/is-there-a-practical-limit-on-the-number-of-memory-mapped-files-in-ios


> In any case, as Simone said, _other apps_ require memory of their own as
> well.
>

Correction 2: Simon, not Simone (my roommate's name, so i type the 'e' out
of habit)

-- 
- stephan beal
http://wanderinghorse.net/home/stephan/
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Why MMAP return ENOMEM in SQLite?

[Stephan Beal]

On Wed, Aug 31, 2016 at 10:39 AM, sanhua.zh  wrote:

> In my testcase, I can only [mmap] a db file at most 1.4GB size. But in
> this new way, I can map a file at most 3.2GB. The test device is iPhone 6S.
>

According to google, the iPhone 6s only has 2GB of RAM, so you can't memmap
3.2G. In any case, as Simone said, _other apps_ require memory of their own
as well. Your app does not have that whole 2G to itself.

-- 
- stephan beal
http://wanderinghorse.net/home/stephan/
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Why MMAP return ENOMEM in SQLite?

[sanhua.zh]

Thanks for your answer. Simon Why are you trying to do memory-mapping on a 
portable device ? Is it for speed ? Do you absolutely need it ? Yes. even on 
iDevice, [mmap] can get faster than sequential I/O. Sometimes it will be twice 
faster. Simon Won't the standard SQLite API do the job well enough ? I did use 
the standard SQLite API. It maps the whole db file into address space. When the 
db file is too big, the [mmap] will failed and turn back to sequential I/O 
method. I have another way to solve this problem. The address space might 
be“scttered”. There is not enough space to fit a BIG file. But there are a lot 
of fragment space to fit multiple small regions. So I can map multiple regions 
of file, instead of mapping the whole db file. For example, mapping 256 
4MB-regions, instead of mapping the whole 1GB file. This solution should modify 
the [unixRemapFile] method in the source code of SQLite, also, I should remap 
the exact region into the pMapRegion variable. In my testcase, I can only 
[mmap] a db file at most 1.4GB size. But in this new way, I can map a file at 
most 3.2GB. The test device is iPhone 6S.




原始邮件
发件人:sanhua.zhsanhua...@foxmail.com
收件人:SQLite mailing listsqlite-us...@mailinglists.sqlite.org
发送时间:2016年8月31日(周三) 16:39
主题:Re: [sqlite] Why MMAP return ENOMEM in SQLite?


Thanks for your answer. Simon Why are you trying to do memory-mapping on a 
portable device ? Is it for speed ? Do you absolutely need it ? Yes. even on 
iDevice, [mmap] can get faster than sequential I/O. Sometime it will be twice 
faster. Simon Won't the standard SQLite API do the job well enough ? I did use 
the standard SQLite API. It maps the whole db file into address space. When the 
db file is too big, the [mmap] will failed and back tosequential I/O method. I 
have another way to solve this problem. The address space might be“scttered”. 
There is not enough space to fit a BIG file. But there are a lot of fragment 
space to fit multiple small region. So I can map multiple regions of file, 
instead of mapping the whole db file. For example, mapping 128 4MB-regions, 
instead of mapping the whole 1GB file. This solution should modify the 
[unixRemapFile] method in the source code of SQLite, also, I should remap the 
exact region into the pMapRegion variable. In my testcase, I can only [mmap] a 
db file at most 1.4GB size. But in this new way, I can map a file at most 
3.2GB. The test device is iPhone 6S. 原始邮件 发件人:Simon slavinslav...@bigfraud.org 
收件人:SQLite mailing listsqlite-us...@mailinglists.sqlite.org 发送时间:2016年8月31日(周三) 
14:38 主题:Re: [sqlite] Why MMAP return ENOMEM in SQLite? On 31 Aug 2016, at 
4:40am, sanhua.zh sanhua...@foxmail.com wrote: Is it caused by mmap file too 
big that the address space is not enough? Yes. Under iOS, memory maps can only 
be as big as the available physical memory. You're working on a tiny 
multi-tasking device and memory is at a premium. You can't afford to hog 
memory. Nor can you do data-handling so fast that it runs down the battery or 
overheats the device. Also, iDevices before the iPhone 5S in 2013 ran a 32-bit 
version of iOS, limiting them to 4GB of memory per App. If yes, then how can I 
get the allowed mmap file size? This figure would be useless since available 
physical memory varies depending on things your application cannot prevent. For 
instance, you may find out how much free memory there is, but then there may be 
in incoming phonecall. iOS will then allocate some of that memory to the 
process dealing with the phonecall. And, of course you cannot do anything to 
interfere with a phonecall because that is more important than an App. Why are 
you trying to do memory-mapping on a portable device ? Is it for speed ? Do you 
absolutely need it ? Won't the standard SQLite API do the job well enough ? 
Simon. ___ sqlite-users mailing 
list sqlite-users@mailinglists.sqlite.org 
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users 
___ sqlite-users mailing list 
sqlite-users@mailinglists.sqlite.org 
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Why MMAP return ENOMEM in SQLite?

[sanhua.zh]

Thanks for your answer.


Simon Why are you trying to do memory-mapping on a portable device ? Is it for 
speed ? Do you absolutely need it ? 
Yes. even on iDevice, [mmap] can get faster than sequential I/O. Sometime it 
will be twice faster.


Simon Won't the standard SQLite API do the job well enough ?
I did use the standard SQLite API. It maps the whole db file into address 
space. When the db file is too big, the [mmap] will failed and back 
tosequential I/O method.


I have another way to solve this problem.
The address space might be“scttered”. There is not enough space to fit a BIG 
file. But there are a lot of fragment space to fit multiple small region.
So I can map multiple regions of file, instead of mapping the whole db file. 
For example, mapping 128 4MB-regions, instead of mapping the whole 1GB file.
This solution should modify the [unixRemapFile] method in the source code of 
SQLite, also, I should remap the exact region into the pMapRegion variable.
In my testcase, I can only [mmap] a db file at most 1.4GB size. But in this new 
way, I can map a file at most 3.2GB. The test device is iPhone 6S.




原始邮件
发件人:Simon slavinslav...@bigfraud.org
收件人:SQLite mailing listsqlite-us...@mailinglists.sqlite.org
发送时间:2016年8月31日(周三) 14:38
主题:Re: [sqlite] Why MMAP return ENOMEM in SQLite?


On 31 Aug 2016, at 4:40am, sanhua.zh sanhua...@foxmail.com wrote:  Is it caused 
by mmap file too big that the address space is not enough? Yes. Under iOS, 
memory maps can only be as big as the available physical memory. You're working 
on a tiny multi-tasking device and memory is at a premium. You can't afford to 
hog memory. Nor can you do data-handling so fast that it runs down the battery 
or overheats the device. Also, iDevices before the iPhone 5S in 2013 ran a 
32-bit version of iOS, limiting them to 4GB of memory per App.  If yes, then 
how can I get the allowed mmap file size? This figure would be useless since 
available physical memory varies depending on things your application cannot 
prevent. For instance, you may find out how much free memory there is, but then 
there may be in incoming phonecall. iOS will then allocate some of that memory 
to the process dealing with the phonecall. And, of course you cannot do 
anything to interfere with a phonecall because that is more important than an 
App. Why are you trying to do memory-mapping on a portable device ? Is it for 
speed ? Do you absolutely need it ? Won't the standard SQLite API do the job 
well enough ? Simon. ___ 
sqlite-users mailing list sqlite-users@mailinglists.sqlite.org 
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Why MMAP return ENOMEM in SQLite?

[Simon Slavin]

On 31 Aug 2016, at 4:40am, sanhua.zh  wrote:

> Is it caused by mmap file too big that the address space is not enough?

Yes.  Under iOS, memory maps can only be as big as the available physical 
memory.  You're working on a tiny multi-tasking device and memory is at a 
premium.  You can't afford to hog memory.  Nor can you do data-handling so fast 
that it runs down the battery or overheats the device.

Also, iDevices before the iPhone 5S in 2013 ran a 32-bit version of iOS, 
limiting them to 4GB of memory per App.

> If yes, then how can I get the allowed mmap file size?

This figure would be useless since available physical memory varies depending 
on things your application cannot prevent.  For instance, you may find out how 
much free memory there is, but then there may be in incoming phonecall.  iOS 
will then allocate some of that memory to the process dealing with the 
phonecall.  And, of course you cannot do anything to interfere with a phonecall 
because that is more important than an App.

Why are you trying to do memory-mapping on a portable device ?  Is it for speed 
?  Do you absolutely need it ?  Won't the standard SQLite API do the job well 
enough ?

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Why MMAP return ENOMEM in SQLite?

[sanhua.zh]

I try to mmap a BIG file which is around 1.8GB size, on iOS.
Then I found that [sqlite3OSFetch] doesn’t work at all, which means the mmap is 
disable.
After debug, I find that [osMmap] in [unixRemapFile] return an error. The error 
code is 12, named ENOMEM, which means cannot allocate memory.


My question is:
Is it caused by mmap file too big that the address space is not enough?
If yes, then how can I get the allowed mmap file size?
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users