[issue21826] Performance issue (+fix) AIX ctypes.util with no /sbin/ldconfig present

[Berker Peksag]

Change by Berker Peksag :


--
resolution:  -> wont fix
stage: test needed -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21826] Performance issue (+fix) AIX ctypes.util with no /sbin/ldconfig present

[Michael Felt]

Michael Felt  added the comment:

as this is fixed is Python3.7 (see issue26439) and has been stated several 
times that it will not be fixed in Python2.7 I suspect this issue may also be 
closed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21826] Performance issue (+fix) AIX ctypes.util with no /sbin/ldconfig present

[Michael Felt]

Michael Felt added the comment:

re: issue 26439 and issue 27435 would like to show:

without patch, find_library() is consistently slow, and in default situations, 
returns nothing.

root@x064:[/data/prj/aixtools/python/python-2.7.10]./python -m timeit -n 100 
'import ctypes.util; ctypes.util.find_library("crypt")'
100 loops, best of 3: 29.6 msec per loop
root@x064:[/data/prj/aixtools/python/python-2.7.10]./python -m timeit -n 100 
'import ctypes.util; ctypes.util.find_library("crypt")'
100 loops, best of 3: 28.8 msec per loop
root@x064:[/data/prj/aixtools/python/python-2.7.10]./python -m timeit -n 100 
'import ctypes.util; ctypes.util.find_library("c")'
100 loops, best of 3: 29.4 msec per loop
root@x064:[/data/prj/aixtools/python/python-2.7.10]./python -m timeit -n 100 
'import ctypes.util; ctypes.util.find_library("m")'
100 loops, best of 3: 29.8 msec per loop
root@x064:[/data/prj/aixtools/python/python-2.7.10]cd -
/data/prj/aixtools/python/python-2.7.12.1
root@x064:[/data/prj/aixtools/python/python-2.7.12.1]./python -m timeit -n 100 
'import ctypes.util; ctypes.util.find_library("crypt")'
100 loops, best of 3: 13.5 msec per loop
root@x064:[/data/prj/aixtools/python/python-2.7.12.1]./python -m timeit -n 100 
'import ctypes.util; ctypes.util.find_library("c")'
100 loops, best of 3: 21.4 msec per loop
root@x064:[/data/prj/aixtools/python/python-2.7.12.1]./python -m timeit -n 100 
'import ctypes.util; ctypes.util.find_library("m")'
100 loops, best of 3: 26.1 msec per loop

The results can be explained as follows:
no patch - all fail, nothing is ever found in a default environment.

with patch: crypt - the answer includes the archive member libcrypt.so, so it 
is found "earlier" than for "c" which must look for legacy names, which is 
faster than "m" which does not exist.

What I consider "frightening" - and a good reason to stay with Python2 as long 
as possible is the follow difference
Python3.6a2 - unpatched, versus Python3.6a4 - ctypes.util patched for AIX:

root@x064:[/data/prj/aixtools/python/python-3.6.0.162]./python -m timeit -n 100 
'import ctypes.util; ctypes.util.find_library("crypt")'
100 loops, best of 3: 84.1 msec per loop
root@x064:[/data/prj/aixtools/python/python-3.6.0.162]cd ../*164
root@x064:[/data/prj/aixtools/python/python-3.6.0.164]./python -m timeit -n 100 
'import ctypes.util; ctypes.util.find_library("crypt")'
100 loops, best of 3: 66.6 msec per loop

FYI: this is on a POWER6 (so h/w is 'quite old', msec should be seen as 
relative, not as accurate for newer hardware)

--
nosy: +Michael.Felt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21826] Performance issue (+fix) AIX ctypes.util with no /sbin/ldconfig present

[Martin Panter]

Martin Panter added the comment:

For the record, if you wanted to actually load the library function on AIX, I 
understand the code might look a bit like:

if sys.platform.startswith("aix"):
if sys.maxsize > 2**32:
lib = "libc.a(shr_64.o)"
else:
lib = "libc.a(shr.o)"
RTLD_MEMBER = 0x0004
lib = ctypes.CDLL(lib, ctypes.DEFAULT_MODE | RTLD_MEMBER)
_uuid_generate_time = lib.uuid_generate_time
else:
# The uuid_generate_* routines are provided by libuuid on at least
# Linux and FreeBSD, and provided by libc on Mac OS X.
...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21826] Performance issue (+fix) AIX ctypes.util with no /sbin/ldconfig present

[Martin Panter]

Martin Panter added the comment:

It looks like Bert’s patch adjusted two ldconfig calls; the first one was in a 
BSD-specific branch. Was this intended? Only the second call (default “else:” 
branch) seems applicable to AIX.

Perhaps the performance is improved now that the popen() shell calls have been 
replaced with more direct subprocess.Popen() calls; see revision a09ae70f3489 
and Issue 22636. If the 300 ms slowdown was due to the shell, that should no 
longer be a problem.

But if the slowdown is say inherent in fork(), I am not sure it is worth adding 
the proposed os.path.exists() check. Either try a way of spawning a child 
process without fork(), like Issue 20104. Or avoid calling find_library() in 
the first place, like I suggested at 
.

--
stage:  -> test needed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21826] Performance issue (+fix) AIX ctypes.util with no /sbin/ldconfig present

[Martin Panter]

Martin Panter added the comment:

See also Issue 26439 about find_library() AIX support

--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21826] Performance issue (+fix) AIX ctypes.util with no /sbin/ldconfig present

[tw.bert]

tw.bert added the comment:

Hi David, thank you for looking into this.
Issue 11063 starts with "When the uuid.py module is simply imported it has the 
side effect of forking a subprocess (/sbin/ldconfig) and doing a lot of stuff 
find a uuid implementation by ctypes.".

My fix is specific about solving the AIX performance problem I encounter in the 
exact same condition as above. My guess is, that if 11063 is picked up (not 
using external tools), the AIX performance problem will be addressed 
automatically, rendering the new issue 21826 obsolete.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21826] Performance issue (+fix) AIX ctypes.util with no /sbin/ldconfig present

[R. David Murray]

R. David Murray added the comment:

How does this interact with issue 11063?

--
nosy: +r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21826] Performance issue (+fix) AIX ctypes.util with no /sbin/ldconfig present

[tw.bert]

New submission from tw.bert:

Preample: This is my first post to the python issue tracker, I included a fix.

This issue is probably related to http://bugs.python.org/issue11063 .

The problem:

After upgrading a package on AIX 7.1 x64 that started using the uuid module, we 
experienced serious performance issues.

The culprit (found after a day of debugging) is here:

File: ctypes/util.py
Note: The file /sbin/ldconfig does *not* exist, so no useful information is 
collected here.
The statement: 

`f = os.popen('/sbin/ldconfig -p 2>/dev/null')`

To be more specific about the performace at popen(), the performance 
degradation happens in it's close() method. It takes 300 ms, which is 
unacceptable. In a larger scope, statements that took 200ms now take 1400ms 
(because the above is called several times.

If I simply check for os.path.exists before the popen, the performance is fine 
again. See the included simple patch. It's a simple unix diff, we don't have 
git on that machine. Git can handle those diffs easily to my knowledge.

More information:

Small scope, culprit identified:

import os, time, traceback
print os.__file__
print time.clock(),'pre'
f = None
try:
  #if os.path.exists('/sbin/ldconfig'):
  f = os.popen('/sbin/ldconfig -p 2>/dev/null')
except:
  print traceback.format_exc()
finally:
  print time.clock(),'post close'
  if f: f.close()
  print time.clock(),'post finally'

This takes 300ms (post finally) without the check for exists.

Large scope, before patch:
time python -c "import hiredis;import redis;print 'redis-py version: %s , 
hiredis-py version: %s' %(redis.VERSION,hiredis.__ver
sion__,)"
redis-py version: (2, 10, 1) , hiredis-py version: 0.1.3

real0m1.409s
user0m0.416s
sys 0m0.129s

Large scope, after patch:
time python -c "import hiredis;import redis;print 'redis-py version: %s , 
hiredis-py version: %s' %(redis.VERSION,hiredis.__ver
sion__,)"
redis-py version: (2, 10, 1) , hiredis-py version: 0.1.3

real0m0.192s
user0m0.056s
sys 0m0.050s

--
components: ctypes
files: patch_ctypes_util_py.diff
keywords: patch
messages: 221266
nosy: tw.bert
priority: normal
severity: normal
status: open
title: Performance issue (+fix) AIX ctypes.util with no /sbin/ldconfig present
versions: Python 2.7
Added file: http://bugs.python.org/file35726/patch_ctypes_util_py.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com