Re: [sqlite] Changing file descriptor of database file

[Black, Michael (IS)]

The select() limit has nothing to do with sqlite.  You already noted it's a 
limit on sockets -- it's really an OS limit.  Do you have any idea what your 
max is or what you think you need?
 
I see some solutions and upcoming problems (by the way, you forgot to mention 
what OS you're on).
 
#1 You're apparently handling your socket connections all from the same 
executing binary.  Just give each client their own executable and you won't hit 
that limit.  Two ways to do that are to fork() and also to spawn the executable 
uniquely for each client (so you don't run a server process -- you let xinetd 
do it for you so you get one process per socket.
 
#2 Use poll() instead of select().  poll() doesnt have the 1024 limit.
 
 
 
Michael D. Black
Senior Scientist
Northrop Grumman Mission Systems
 



From: sqlite-users-boun...@sqlite.org on behalf of Pavel Ivanov
Sent: Fri 5/7/2010 3:43 PM
To: General Discussion of SQLite Database
Subject: [sqlite] Changing file descriptor of database file



Hi, all!

I'm writing a server application that uses SQLite database as a
storage. In fact for various reasons I use a lot of SQLite database
files at once. This application can serve hundreds of clients spread
all over the network. Communication with those clients goes over
non-blocking sockets and currently select() is used to listen for
events in those sockets. select() has limitation that it can be used
only with file descriptors less than 1024. On a heavy loaded server my
application can easily run out of those 1024 file descriptors which
will mean that it will no longer accept connections from new clients.
To avoid such situation or at least to make possibility of such event
less I'd like to force SQLite to open databases with file descriptors
greater than 1024. Is there a way how I can do that without patching
SQLite sources and maintaining this patch over SQLite updates?


Pavel
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Changing file descriptor of database file

[Andrew Davison]

For people wanting to re-do their networking code I would reccommend 
Asio (or Boost::Asio), see:

   http://think-async.com/Asio/

Ad.

On 8/05/2010 9:14 AM, Roger Binns wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> On 05/07/2010 01:43 PM, Pavel Ivanov wrote:
>> select() has limitation that it can be used
>> only with file descriptors less than 1024.
>
> This is rarely true and I only know of one crappy operating system that had
> that limitation 13 years ago.  You can simply recompile with
> - -DFD_SETSIZE=1 in all files that use fd_set to have a different "limit".
>
> However select() is a horrendous interface, especially beyond a trivial
> number of file descriptors and there are lots of better ones out there with
> a variety of attributes and support.  I'd recommend using libevent which
> abstracts all this stuff away.  Heck, it even works on Windows!
>
>http://en.wikipedia.org/wiki/Libevent
>
> Roger
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.10 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAkvknu8ACgkQmOOfHg372QRbTQCeKx8ZA++bGsvNCybLlg3P3uwl
> eXkAoML1M+wGVvaoCWZaRT7OnwvpgPdD
> =WLrY
> -END PGP SIGNATURE-
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Changing file descriptor of database file

[Nicolas Williams]

On Fri, May 07, 2010 at 05:24:58PM -0400, Mikhail Terekhov wrote:
> IMHO it would be better to switch to poll/epoll from select instead of
> fighting file descriptor numbers.

Better: use libevent.

(select()'s limit of 1024 fildes is... odd and derives mainly from the
FD*() macros and their semantics.  Not all operating systems enforce a
limit of 1024 fildes as the obvious thing to do is to treat the fildes
sets as bitmasks of nfds bits rounded up to a multiple of NFDBITS.
Solaris does enfore a 1024 limit for select(), at run-time.)

Nico
-- 
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Changing file descriptor of database file

[Roger Binns]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 05/07/2010 01:43 PM, Pavel Ivanov wrote:
> select() has limitation that it can be used
> only with file descriptors less than 1024.

This is rarely true and I only know of one crappy operating system that had
that limitation 13 years ago.  You can simply recompile with
- -DFD_SETSIZE=1 in all files that use fd_set to have a different "limit".

However select() is a horrendous interface, especially beyond a trivial
number of file descriptors and there are lots of better ones out there with
a variety of attributes and support.  I'd recommend using libevent which
abstracts all this stuff away.  Heck, it even works on Windows!

  http://en.wikipedia.org/wiki/Libevent

Roger
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkvknu8ACgkQmOOfHg372QRbTQCeKx8ZA++bGsvNCybLlg3P3uwl
eXkAoML1M+wGVvaoCWZaRT7OnwvpgPdD
=WLrY
-END PGP SIGNATURE-
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Changing file descriptor of database file

[Pavel Ivanov]

> IMHO it would be better to switch to poll/epoll from select instead of
> fighting file descriptor numbers.

Sure, that's my opinion too. It's just some significant amount of work
which looks like less prioritized at this moment than other
application improvements. So I'm looking for some solution for the
short term.

Pavel

On Fri, May 7, 2010 at 5:24 PM, Mikhail Terekhov  wrote:
> On Fri, May 7, 2010 at 4:43 PM, Pavel Ivanov  wrote:
>
>> Hi, all!
>>
>> I'm writing a server application that uses SQLite database as a
>> storage. In fact for various reasons I use a lot of SQLite database
>> files at once. This application can serve hundreds of clients spread
>> all over the network. Communication with those clients goes over
>> non-blocking sockets and currently select() is used to listen for
>> events in those sockets. select() has limitation that it can be used
>> only with file descriptors less than 1024. On a heavy loaded server my
>> application can easily run out of those 1024 file descriptors which
>> will mean that it will no longer accept connections from new clients.
>> To avoid such situation or at least to make possibility of such event
>> less I'd like to force SQLite to open databases with file descriptors
>> greater than 1024. Is there a way how I can do that without patching
>> SQLite sources and maintaining this patch over SQLite updates?
>>
>>
> IMHO it would be better to switch to poll/epoll from select instead of
> fighting file descriptor numbers.
>
>
> Regards,
> --
> Mikhail Terekhov
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Changing file descriptor of database file

[Mikhail Terekhov]

On Fri, May 7, 2010 at 4:43 PM, Pavel Ivanov  wrote:

> Hi, all!
>
> I'm writing a server application that uses SQLite database as a
> storage. In fact for various reasons I use a lot of SQLite database
> files at once. This application can serve hundreds of clients spread
> all over the network. Communication with those clients goes over
> non-blocking sockets and currently select() is used to listen for
> events in those sockets. select() has limitation that it can be used
> only with file descriptors less than 1024. On a heavy loaded server my
> application can easily run out of those 1024 file descriptors which
> will mean that it will no longer accept connections from new clients.
> To avoid such situation or at least to make possibility of such event
> less I'd like to force SQLite to open databases with file descriptors
> greater than 1024. Is there a way how I can do that without patching
> SQLite sources and maintaining this patch over SQLite updates?
>
>
IMHO it would be better to switch to poll/epoll from select instead of
fighting file descriptor numbers.


Regards,
-- 
Mikhail Terekhov
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Changing file descriptor of database file

[Pavel Ivanov]

Hi, all!

I'm writing a server application that uses SQLite database as a
storage. In fact for various reasons I use a lot of SQLite database
files at once. This application can serve hundreds of clients spread
all over the network. Communication with those clients goes over
non-blocking sockets and currently select() is used to listen for
events in those sockets. select() has limitation that it can be used
only with file descriptors less than 1024. On a heavy loaded server my
application can easily run out of those 1024 file descriptors which
will mean that it will no longer accept connections from new clients.
To avoid such situation or at least to make possibility of such event
less I'd like to force SQLite to open databases with file descriptors
greater than 1024. Is there a way how I can do that without patching
SQLite sources and maintaining this patch over SQLite updates?


Pavel
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users