Le samedi 8 juin 2013 19:06:58, Tom Lane a écrit : > Recently we had a gripe about how you can't read very many files > concurrently with contrib/file_fdw: > http://www.postgresql.org/message-id/OF419B5767.8A3C9ADB-ON85257B79.005491E > 9-85257b79.0054f...@isn.rtss.qc.ca > > The reason for this is that file_fdw goes through the COPY code, which > uses AllocateFile(), which has a wired-in assumption that not very many > files need to be open concurrently. I thought for a bit about trying to > get COPY to use a "virtual" file descriptor such as is provided by the > rest of fd.c, but that didn't look too easy, and in any case probably > nobody would be excited about adding additional overhead to the COPY > code path. What seems more practical is to relax the hard-coded limit > on the number of files concurrently open through AllocateFile(). Now, > we could certainly turn that array into some open-ended list structure, > but that still wouldn't let us have an arbitrary number of files open, > because at some point we'd run into platform-specific EMFILES or ENFILES > limits on the number of open file descriptors. In practice we want to > keep it under the max_safe_fds limit that fd.c goes to a lot of trouble > to determine. So it seems like the most useful compromise is to keep > the allocatedDescs[] array data structure as-is, but allow its size to > depend on max_safe_fds. In the attached proposed patch I limit it to > max_safe_fds / 2, so that there's still a reasonable number of FDs > available for fd.c's main pool of virtual FDs. On typical modern > platforms that should be at least a couple of hundred, thus making the > effective limit about an order of magnitude more than it is currently > (32). > > Barring objections or better ideas, I'd like to back-patch this as far > as 9.1 where file_fdw was introduced.
I just wonder about this statement that you removed: * Since we don't want to encourage heavy use of those functions, * it seems OK to put a pretty small maximum limit on the number of * simultaneously allocated descs. Is it now encouraged to use those functions, or at least that it seems less 'scary' than in the past ? -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/ PostgreSQL: Support 24x7 - Développement, Expertise et Formation
signature.asc
Description: This is a digitally signed message part.
