Patches item #1607087, was opened at 2006-12-01 15:05
Message generated for change (Tracker Item Submitted) made by Item Submitter
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1607087&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Library (Lib)
Group: Python 2.5
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: John L. Allen (johnlallen)
Assigned to: Nobody/Anonymous (nobody)
Summary: popen() slow on AIX due to large FOPEN_MAX value
Initial Comment:
Experimentation revealed that the os.popen[234]() family of methods was at
least 10 times slower on AIX than it was on Solaris. It turns out that this is
because of the _run_child method in popen2.py, which has this definition:
def _run_child(self, cmd):
if isinstance(cmd, basestring):
cmd = ['/bin/sh', '-c', cmd]
for i in xrange(3, MAXFD):
try:
os.close(i)
except OSError:
pass
try:
os.execvp(cmd[0], cmd)
finally:
os._exit(1)
MAXFD is set as follows at the top of popen2.py:
try:
MAXFD = os.sysconf('SC_OPEN_MAX')
except (AttributeError, ValueError):
MAXFD = 256
On AIX, SC_OPEN_MAX is 32767, whereas on Solaris, it is not defined, and we get
the default value of 256. So, on AIX the python code for loop is being used
to close 32763 file descriptors, but only 253 on Solaris. The slowness of this
python loop is the source of the problem.
Several solutions are possible. AIX provides a much faster way to close all
file descriptors above a given one using the fcntl F_CLOSEM option. In this
case, it would be: fcntl(3, F_CLOSEM, 0). Other OSes, like Solaris and BSD,
have the closefrom() function instead. I think ideally, we would want to have
an os.closefrom() method defined in posixmodule.c, and always available on
every OS, and have popen2.py call that instead of doing the loop. The
closefrom() function would be defined something like this:
--------------------
PyDoc_STRVAR(posix_closefrom__doc__,
"closefrom(fd)\n\n\
Close all file descriptors greater than or equal to fd (for low level IO).");
static PyObject *
posix_closefrom(PyObject *self, PyObject *args)
{
int fd, maxfd, res;
if (!PyArg_ParseTuple(args, "i:closefrom", &fd))
return NULL;
Py_BEGIN_ALLOW_THREADS
#ifdef HAVE_CLOSEFROM
res = closefrom(fd);
#else
#ifdef F_CLOSEM
res = fcntl(3, F_CLOSEM, 0)
#else
#if defined( HAVE_SYSCONF ) && defined( _SC_OPEN_MAX )
# define PY_OPEN_MAX sysconf(_SC_OPEN_MAX)
#else
# ifdef FOPEN_MAX
# define PY_OPEN_MAX FOPEN_MAX
# else
# ifdef OPEN_MAX
# define PY_OPEN_MAX OPEN_MAX
# else
# ifdef _NFILE
# define PY_OPEN_MAX _NFILE
# else
# define PY_OPEN_MAX 256
# endif
# endif
# endif
#endif
maxfd = PY_OPEN_MAX;
while (fd < maxfd) {
res = close(fd);
fd++;
}
#endif
#endif
Py_END_ALLOW_THREADS
if (res < 0)
return posix_error();
Py_INCREF(Py_None);
return Py_None;
}
---------------------
While this is probably (close to) the ideal solution (since it would benefit
all OSes by avoiding the close loop if possible or if not, moving it to C
code), adding os.closefrom() probably needs to be discussed further before
being accepted by the Python community.
Instead, I will provide a simpler patch that only benefits AIX. It adds the
F_CLOSEM attribute to the fcntl class in fcntlmodule.c, if defined, and
modifies popen2.py to check for it and use it if possible, instead of doing the
close() loop. See the attached patch (against the 2.5 source). I don't
believe that any documentation has to change.
John Allen
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1607087&group_id=5470
_______________________________________________
Patches mailing list
Patches@python.org
http://mail.python.org/mailman/listinfo/patches
