Re: Only Bytecode, No .py Files
On Jul 27, 8:56 am, Thomas Rachel nutznetz-0c1b6768-bfa9-48d5- a470-7603bd3aa...@spamschutz.glglgl.de wrote: Am 27.07.2011 14:18 schrieb John Roth: Two comments. First, your trace isn't showing attempts to open .py files, it's showing attempts to open the Curses library in the bash directory. Of course. My goal was to show that the OP's problem is not a python one, but occurs all over several programs. Maybe you also have a problem with the .py files, Why should I? It's also not showing the program that's causing the failing open. It is irrelevant, IMO. It just shows that it seems to be common, even for the bin loader, to search for libraries to be linked to by trying to open them from several places. So every program call must be full of failures shown by the audit program. Second, the audit program is an idiot. There are lots of programs which use the easier to ask forgiveness pattern and test for the existence of optional files by trying to open them. This may be what Bash is doing. ACK. That is exactly what I wanted to show. (With the difference that this is probably nt the bash, but the linux loader trying to link a .so, but for the problem, it doesn't make any difference.) Thomas Sorry. I thought what you posted was from the OP. I guess I don't really expect someone to post a completely irrelevant trace in a thread started by someone who has a problem. John Roth -- http://mail.python.org/mailman/listinfo/python-list
Re: Only Bytecode, No .py Files
John Roth wrote: ACK. That is exactly what I wanted to show. (With the difference that this is probably nt the bash, but the linux loader trying to link a .so, but for the problem, it doesn't make any difference.) Sorry. I thought what you posted was from the OP. I guess I don't really expect someone to post a completely irrelevant trace in a thread started by someone who has a problem. The trace was showing the same results in an effort to demonstrate that the problem was just normal operations. This is very relevant. ~Ethan~ -- http://mail.python.org/mailman/listinfo/python-list
Re: Only Bytecode, No .py Files
On Fri, Jul 29, 2011 at 8:51 AM, John Roth johnro...@gmail.com wrote: Sorry. I thought what you posted was from the OP. I guess I don't really expect someone to post a completely irrelevant trace in a thread started by someone who has a problem. I'm not sure why you would think that that post was by the original poster, or that it was irrelevant. It seems to me that it demonstrated exactly the behavior Eldon was complaining about as a normal part of the operation of lots of normal unix programs. Thomas even said that in the very first line of his post (after the quoted bit). -- Jerry -- http://mail.python.org/mailman/listinfo/python-list
Re: Only Bytecode, No .py Files
Am 29.07.2011 14:51 schrieb John Roth: Sorry. I thought what you posted was from the OP. I guess I don't really expect someone to post a completely irrelevant trace in a thread started by someone who has a problem. In what way do you find the trace irrelevant? It clearly shows that it is not a good idea for an audit tool to report all failing open() calls and that the OP's problem is none, at least not one related to Python. No more, no less. Thomas -- http://mail.python.org/mailman/listinfo/python-list
Re: Only Bytecode, No .py Files
On 07/26/2011 11:19 AM, Eldon Ziegler wrote: Is there a way to have the Python processor look only for bytecode files, not .py files? We are seeing huge numbers of Linux audit messages on production system on which only bytecode files are stored. The audit subsystem is recording each open failure. Thanks, Eldon Ziegler Here are two suggestions: 1. Put a dummy python file in the directory that does very little, perhaps something like: print(%s: You should not be seeing this % sys.argv[0]) Give it the name of one of the .pyc, e.g., if foo.pyc, then foo.py. Give the bytecode file a later date, e.g., touch foo.pyc (assuming you can do that without messing up other aspects of your system.) If that works, then copy the dummy file to each of the other required .py names in the directory. You'll then have to touch *.pyc to ensure that the pyc's have later dates. After that, you won't have to do anything when you replace a .pyc file. When you add a new .pyc you'll need to do the copy and touch again. 2. Use a log truncator to truncate the audit log. I don't remember the name of one at the moment, but I believe there is at least one open source program that will monitor named files and truncate them from the beginning to a fixed maximum size. I don't like this method as much as the first because it might result in something important being truncated. Alan -- http://mail.python.org/mailman/listinfo/python-list
Re: Only Bytecode, No .py Files
Am 26.07.2011 17:19 schrieb Eldon Ziegler: Is there a way to have the Python processor look only for bytecode files, not .py files? We are seeing huge numbers of Linux audit messages on production system on which only bytecode files are stored. The audit subsystem is recording each open failure. Is that really a problem? AFAIK there are many failing open() calls on the start of every program. E.g. open(/lib/bash/4.1/tls/i686/sse2/libncurses.so.5, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1/tls/i686/sse2, 0xbfd3a350) = -1 ENOENT (No such file or directory) open(/lib/bash/4.1/tls/i686/libncurses.so.5, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1/tls/i686, 0xbfd3a350) = -1 ENOENT (No such file or directory) open(/lib/bash/4.1/tls/sse2/libncurses.so.5, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1/tls/sse2, 0xbfd3a350) = -1 ENOENT (No such file or directory) open(/lib/bash/4.1/tls/libncurses.so.5, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1/tls, 0xbfd3a350) = -1 ENOENT (No such file or directory) open(/lib/bash/4.1/i686/sse2/libncurses.so.5, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1/i686/sse2, 0xbfd3a350) = -1 ENOENT (No such file or directory) open(/lib/bash/4.1/i686/libncurses.so.5, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1/i686, 0xbfd3a350) = -1 ENOENT (No such file or directory) open(/lib/bash/4.1/sse2/libncurses.so.5, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1/sse2, 0xbfd3a350) = -1 ENOENT (No such file or directory) open(/lib/bash/4.1/libncurses.so.5, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1, 0xbfd3a350) = -1 ENOENT (No such file or directory) open(/lib/libncurses.so.5, O_RDONLY) = 3 is a part of what happens if I start the MySQL client, Even starting the bash results in open(/lib/bash/4.1/tls/i686/sse2/libreadline.so.6, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1/tls/i686/sse2, 0xbfe0c4d0) = -1 ENOENT (No such file or directory) open(/lib/bash/4.1/tls/i686/libreadline.so.6, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1/tls/i686, 0xbfe0c4d0) = -1 ENOENT (No such file or directory) open(/lib/bash/4.1/tls/sse2/libreadline.so.6, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1/tls/sse2, 0xbfe0c4d0) = -1 ENOENT (No such file or directory) open(/lib/bash/4.1/tls/libreadline.so.6, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1/tls, 0xbfe0c4d0) = -1 ENOENT (No such file or directory) open(/lib/bash/4.1/i686/sse2/libreadline.so.6, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1/i686/sse2, 0xbfe0c4d0) = -1 ENOENT (No such file or directory) open(/lib/bash/4.1/i686/libreadline.so.6, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1/i686, 0xbfe0c4d0) = -1 ENOENT (No such file or directory) open(/lib/bash/4.1/sse2/libreadline.so.6, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1/sse2, 0xbfe0c4d0) = -1 ENOENT (No such file or directory) open(/lib/bash/4.1/libreadline.so.6, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1, 0xbfe0c4d0) = -1 ENOENT (No such file or directory) open(/etc/ld.so.cache, O_RDONLY) = 3 So can it really be such a huge problem? Thomas -- http://mail.python.org/mailman/listinfo/python-list
Re: Only Bytecode, No .py Files
Am 27.07.2011 03:32, schrieb harrismh777: Christian Heimes wrote: The first four bytes of a pyc file contain the magic header. It must match the magic of the current Python version. The next four bytes contain the pyc_mtime. It must match the mtime of the corresponding .py files as returned by fstat().st_mtime. If the magic doesn't match or the mtime header doesn't match the mtime of the .py file, the pyc is ignored. ... so recompile is required to fix. It's not required, you can fake a .py file with a trick: import os, struct, datetime First get the magic and mtime from the pyc file with open(test.pyc, rb) as f: ... header = f.read(8) ... magic, mtime = struct.unpack(ii, header) magic, mtime (168686339, 1311717735) Verify it's a good date datetime.datetime.fromtimestamp(mtime) datetime.datetime(2011, 7, 27, 0, 2, 15) Now create an empty test.py open(test.py, w).close() Set its mtime os.utime(test.py, (mtime, mtime)) Now the test.py has the same mtime as test.pyc and Python won't recompile the .pyc file from the .py file as long as the magic header (168686339) is correct. Christian -- http://mail.python.org/mailman/listinfo/python-list
Re: Only Bytecode, No .py Files
On Jul 27, 1:10 am, Thomas Rachel nutznetz-0c1b6768-bfa9-48d5- a470-7603bd3aa...@spamschutz.glglgl.de wrote: Am 26.07.2011 17:19 schrieb Eldon Ziegler: Is there a way to have the Python processor look only for bytecode files, not .py files? We are seeing huge numbers of Linux audit messages on production system on which only bytecode files are stored. The audit subsystem is recording each open failure. Is that really a problem? AFAIK there are many failing open() calls on the start of every program. E.g. open(/lib/bash/4.1/tls/i686/sse2/libncurses.so.5, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1/tls/i686/sse2, 0xbfd3a350) = -1 ENOENT (No such file or directory) open(/lib/bash/4.1/tls/i686/libncurses.so.5, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1/tls/i686, 0xbfd3a350) = -1 ENOENT (No such file or directory) open(/lib/bash/4.1/tls/sse2/libncurses.so.5, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1/tls/sse2, 0xbfd3a350) = -1 ENOENT (No such file or directory) open(/lib/bash/4.1/tls/libncurses.so.5, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1/tls, 0xbfd3a350) = -1 ENOENT (No such file or directory) open(/lib/bash/4.1/i686/sse2/libncurses.so.5, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1/i686/sse2, 0xbfd3a350) = -1 ENOENT (No such file or directory) open(/lib/bash/4.1/i686/libncurses.so.5, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1/i686, 0xbfd3a350) = -1 ENOENT (No such file or directory) open(/lib/bash/4.1/sse2/libncurses.so.5, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1/sse2, 0xbfd3a350) = -1 ENOENT (No such file or directory) open(/lib/bash/4.1/libncurses.so.5, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1, 0xbfd3a350) = -1 ENOENT (No such file or directory) open(/lib/libncurses.so.5, O_RDONLY) = 3 is a part of what happens if I start the MySQL client, Even starting the bash results in open(/lib/bash/4.1/tls/i686/sse2/libreadline.so.6, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1/tls/i686/sse2, 0xbfe0c4d0) = -1 ENOENT (No such file or directory) open(/lib/bash/4.1/tls/i686/libreadline.so.6, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1/tls/i686, 0xbfe0c4d0) = -1 ENOENT (No such file or directory) open(/lib/bash/4.1/tls/sse2/libreadline.so.6, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1/tls/sse2, 0xbfe0c4d0) = -1 ENOENT (No such file or directory) open(/lib/bash/4.1/tls/libreadline.so.6, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1/tls, 0xbfe0c4d0) = -1 ENOENT (No such file or directory) open(/lib/bash/4.1/i686/sse2/libreadline.so.6, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1/i686/sse2, 0xbfe0c4d0) = -1 ENOENT (No such file or directory) open(/lib/bash/4.1/i686/libreadline.so.6, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1/i686, 0xbfe0c4d0) = -1 ENOENT (No such file or directory) open(/lib/bash/4.1/sse2/libreadline.so.6, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1/sse2, 0xbfe0c4d0) = -1 ENOENT (No such file or directory) open(/lib/bash/4.1/libreadline.so.6, O_RDONLY) = -1 ENOENT (No such file or directory) stat64(/lib/bash/4.1, 0xbfe0c4d0) = -1 ENOENT (No such file or directory) open(/etc/ld.so.cache, O_RDONLY) = 3 So can it really be such a huge problem? Thomas Two comments. First, your trace isn't showing attempts to open .py files, it's showing attempts to open the Curses library in the bash directory. Maybe you also have a problem with the .py files, but it isn't documented in this trace. It's also not showing the program that's causing the failing open. Second, the audit program is an idiot. There are lots of programs which use the easier to ask forgiveness pattern and test for the existence of optional files by trying to open them. This may be what Bash is doing. John Roth -- http://mail.python.org/mailman/listinfo/python-list
Re: Only Bytecode, No .py Files
Am 27.07.2011 14:18 schrieb John Roth: Two comments. First, your trace isn't showing attempts to open .py files, it's showing attempts to open the Curses library in the bash directory. Of course. My goal was to show that the OP's problem is not a python one, but occurs all over several programs. Maybe you also have a problem with the .py files, Why should I? It's also not showing the program that's causing the failing open. It is irrelevant, IMO. It just shows that it seems to be common, even for the bin loader, to search for libraries to be linked to by trying to open them from several places. So every program call must be full of failures shown by the audit program. Second, the audit program is an idiot. There are lots of programs which use the easier to ask forgiveness pattern and test for the existence of optional files by trying to open them. This may be what Bash is doing. ACK. That is exactly what I wanted to show. (With the difference that this is probably nt the bash, but the linux loader trying to link a .so, but for the problem, it doesn't make any difference.) Thomas -- http://mail.python.org/mailman/listinfo/python-list
Re: Only Bytecode, No .py Files
Christian Heimes wrote: Now the test.py has the same mtime as test.pyc and Python won't recompile the .pyc file from the .py file as long as the magic header (168686339) is correct. ~very cool. -- http://mail.python.org/mailman/listinfo/python-list
Re: Only Bytecode, No .py Files
On 07/26/2011 11:19 AM, Eldon Ziegler wrote: Is there a way to have the Python processor look only for bytecode files, not .py files? We are seeing huge numbers of Linux audit messages on production system on which only bytecode files are stored. The audit subsystem is recording each open failure. Thanks, Eldon Ziegler How are you opening your files? -- Bill -- http://mail.python.org/mailman/listinfo/python-list
Re: Only Bytecode, No .py Files
*Maybe* you can cook something with import hooks (see PEP 302). However I think the easier option will be to distribute the .py files as well. -- http://mail.python.org/mailman/listinfo/python-list
Re: Only Bytecode, No .py Files
Another possibility: You could probably create a bunch of zero-length .py's that are older than the corresponding .pyc's. On Tue, Jul 26, 2011 at 8:19 AM, Eldon Ziegler eld...@atlanticdb.comwrote: Is there a way to have the Python processor look only for bytecode files, not .py files? We are seeing huge numbers of Linux audit messages on production system on which only bytecode files are stored. The audit subsystem is recording each open failure. Thanks, Eldon Ziegler -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Only Bytecode, No .py Files
That seemed like a good idea but the .py file was complied even though it was dated 2001-01-01 00:00:00. Python must be checking something beside the date. On Tue, 2011-07-26 at 12:32 -0700, Dan Stromberg wrote: Another possibility: You could probably create a bunch of zero-length .py's that are older than the corresponding .pyc's. On Tue, Jul 26, 2011 at 8:19 AM, Eldon Ziegler eld...@atlanticdb.com wrote: Is there a way to have the Python processor look only for bytecode files, not .py files? We are seeing huge numbers of Linux audit messages on production system on which only bytecode files are stored. The audit subsystem is recording each open failure. Thanks, Eldon Ziegler -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Only Bytecode, No .py Files
On 7/26/2011 3:32 PM, Dan Stromberg wrote: Another possibility: You could probably create a bunch of zero-length .py's that are older than the corresponding .pyc's. On Tue, Jul 26, 2011 at 8:19 AM, Eldon Ziegler eld...@atlanticdb.com mailto:eld...@atlanticdb.com wrote: Is there a way to have the Python processor look only for bytecode files, not .py files? We are seeing huge numbers of Linux audit messages on production system on which only bytecode files are stored. The audit subsystem is recording each open failure. Try packaging the app into a zip file so there is only one open, which succeeds. -- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: Only Bytecode, No .py Files
Am 26.07.2011 23:20, schrieb Eldon Ziegler: That seemed like a good idea but the .py file was complied even though it was dated 2001-01-01 00:00:00. Python must be checking something beside the date. The first four bytes of a pyc file contain the magic header. It must match the magic of the current Python version. The next four bytes contain the pyc_mtime. It must match the mtime of the corresponding .py files as returned by fstat().st_mtime. If the magic doesn't match or the mtime header doesn't match the mtime of the .py file, the pyc is ignored. Hint: If you run python with the -v option, you can get information why a pyc file is ignored. $ touch test.py $ pycompile test.py $ python -v -c import test ... # test.pyc matches test.py import test # precompiled from test.pyc ... $ touch test.py $ python -v -c import test ... # test.pyc has bad mtime import test # from test.py ... Christian -- http://mail.python.org/mailman/listinfo/python-list
Re: Only Bytecode, No .py Files
Christian Heimes wrote: The first four bytes of a pyc file contain the magic header. It must match the magic of the current Python version. The next four bytes contain the pyc_mtime. It must match the mtime of the corresponding .py files as returned by fstat().st_mtime. If the magic doesn't match or the mtime header doesn't match the mtime of the .py file, the pyc is ignored. ... so recompile is required to fix. -- http://mail.python.org/mailman/listinfo/python-list
