Re: [Python-Dev] Hash collision security issue (now public)
On Mon, Jan 2, 2012 at 7:18 PM, Christian Heimes li...@cheimes.de wrote: Am 02.01.2012 06:55, schrieb Paul McMillan: I think Ruby uses FNV-1 with a salt, making it less vulnerable to this. FNV is otherwise similar to our existing hash function. For the record, cryptographically strong hash functions are in the neighborhood of 400% slower than our existing hash function. I've pushed a new patch http://hg.python.org/features/randomhash/rev/0a65d2462e0c It seems for 32-bit version you are using pid for the two constants. Also, it's unclear why you even need to use a random constant for the final pass, you already use random constant as an initial h1, and it should be enough, no need to use for k1. Same for 128-bit: k1, k2, k3, k4 should be initialized to zero, these are key data, they don't need to be mixed with anything. Also, I'm not sure how portable is the always_inline attribute, is it supported on all compilers and all platforms? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
On Mon, Jan 2, 2012 at 10:53 PM, Alexey Borzenkov sna...@gmail.com wrote: On Mon, Jan 2, 2012 at 7:18 PM, Christian Heimes li...@cheimes.de wrote: Am 02.01.2012 06:55, schrieb Paul McMillan: I think Ruby uses FNV-1 with a salt, making it less vulnerable to this. FNV is otherwise similar to our existing hash function. For the record, cryptographically strong hash functions are in the neighborhood of 400% slower than our existing hash function. I've pushed a new patch http://hg.python.org/features/randomhash/rev/0a65d2462e0c It seems for 32-bit version you are using pid for the two constants. Also, it's unclear why you even need to use a random constant for the final pass, you already use random constant as an initial h1, and it should be enough, no need to use for k1. Same for 128-bit: k1, k2, k3, k4 should be initialized to zero, these are key data, they don't need to be mixed with anything. Sorry, sent too soon. What I mean is that you're initializing a pretty big array of values when you only need a 32-bit value. Pid, in my opinion might be too predictable, it would be a lot better to simply hash pid and gettimeofday bytes to produce this single 32-bit value and use it for h1, h2, h3 and h4 in both 32-bit and 128-bit versions. Also, I'm not sure how portable is the always_inline attribute, is it supported on all compilers and all platforms? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compiler used to build Python for Windows
On Thu, Mar 6, 2008 at 3:52 AM, Steve Holden [EMAIL PROTECTED] wrote: Mingw tends to be rather more stable (though not itself without the occasional library compatibility issue), and more freely available. Not all extensions can be built using mingw (pywin32 comes to mind immediately). And while you can almost easily (but not very easily, since you need to patch libmoldname yourself and then update your spec file) change runtime used by mingw, you cannot do so with Visual Studio. Besides, this would require spec-editing on both, build machines, as well as user machines (otherwise you get extensions that depend on both msvcrt.dll and msvcrXX.dll), this is a complication too. Runtime compatibility could be a potential problem (or maybe not, if WITH_PYMALLOC maybe fixes that?), so official mingw port (as much as I'd like for one to be there) does not seem feasible to me in any future, Python is already too far on the needle of Visual Studio. :( The biggest problem would also be with some extensions' setup.py. Many extensions I've seen hardwire to distutils.msvccompiler in one way or another. If official python would shift to mingw, they would suddenly break. I think the best lesson here is Tcl. Because it uses stubs mechanism, you don't need to depend on tclXX.dll, you don't deal with really direct implementation details, you don't care about runtimes, everything is much easier. Maybe it's possible (and not too late) for Python to somehow embrace such mechanism? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] distutils.cygwinccompiler: invalid executable for interpreters
Hi everyone, Some time ago I've stumbled upon a problem with compiling py2exe with mingw: it resulted in invalid executable run.exe. At first I dismissed it as some very strange problem, but recently I decided to look into it again and found that the reason for this is a .def file, supplied during compilation, that causes linker to produce executable with IMAGE_FILE_DLL in its PE Header Characteristics field, and operating system refuses to execute it. I traced this problem to the following block in distutils/cygwinccompiler.py: # handle export symbols by creating a def-file # with executables this only works with gcc/ld as linker if ((export_symbols is not None) and (target_desc != self.EXECUTABLE or self.linker_dll == gcc)): Removing 'or self.linker_dll == gcc' obviously helps. To me this sounds like a bug (introduced in revision 17747 on trunk, more that 7 years old!), but I was wondering if there's any logic behind generating a .def file for the executable (sidenote: look at distutils/emxccompiler.py where .def file is generated for executables only). To me it's very unlikely that anyone *ever* needs to export symbols from an executable (I'm not even sure if it can work as intended on Windows). Even then, if there's anyone at all who needs such a machinery (again, what for?), at the very least it shouldn't produce LIBRARY line. As minimal change as this: # Generate .def file contents = [ LIBRARY %s % os.path.basename(output_filename), EXPORTS] if target_desc == self.EXECUTABLE: contents.pop(0) Did the trick (generated executable runs fine). Is anyone interested in a bug report for this? Also, is there any chance that if there is interest it could end up in release25-maint? Best regards, Alexey. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Strange behavior of subprocess.Popen._get_handles under Windows
For a long time I was surprised why if I have a script testme.py: import subprocess subprocess.call(echo Something, shell=True) and I try to execute it like this: python testme.py testme.txt I get the output: The handle is invalid. Strange failures randomly happened with different programs, so I thought maybe this was intended (mis)behavior, and I found that I can workaround this by explicitly specifying std* handles: subprocess.call(, stdin=sys.stdin, stdout=sys.stdout, stderr=sys.stderr) I lived with this until I understood that if something substitutes sys.std* with a proxy (examples: running under IDLE or wxPython, or when there is a custom logger on stdout/stderr, etc), this will no longer work: File C:\Programs\Python25\Lib\subprocess.py, line 698, in _get_handles p2cread = msvcrt.get_osfhandle(stdin.fileno()) AttributeError: fileno Now I finally found that my problem are these two lines in subprocess.py: if stdin is None and stdout is None and stderr is None: return (None, None, None, None, None, None) These lines add an interesting quirk: if I explicitly specify any single channel (i.e. stdout=1) the problem suddenly disappears, and if I just comment these lines altogether my problem vanishes completely! (i.e. output redirection works absolutely well) Further investigations showed that it seems to be some strange OS quirk/bug, i.e. this behavior is caused when STARTF_USESTDHANDLES is not used, and crt's spawnl* behaves incorrectly too: os.spawnl(rC:\WINDOWS\system32\cmd.exe, rC:\WINDOWS\system32\cmd.exe /c echo Something) So the question is should these two lines be removed to workaround this OS bug? To my thinking, this will not change behavior when any std* arguments are passed to Popen, and the only case when it kicks in is when no std* arguments are specified and the resulting side effects are previously failing cases start working correctly. Or am I wrong here, are there any side effects of using STARTF_USESTDHANDLES that I'm missing here? Best regards, Alexey. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Strange behavior of subprocess.Popen._get_handles under Windows
On 8/23/07, Mark Hammond [EMAIL PROTECTED] wrote:
Further investigations showed that it seems to be some strange OS
quirk/bug,
I'm not quite with you here - what strange OS bug do you think you have
found? I expect that such a bug would be well documented somewhere, even if
not directly by MS.
[...]
MSDN documents that without that flag, hStdInput will be the keyboard
buffer while the output handles will default to the console window's
buffer - which sounds significantly different to your expectation that the
existing handles will be used as the default (unless I misunderstand).
Sadly, your mail isn't clear enough for me to be sure about what semantics
you are seeing and exactly what you expect, and I don't have time to
experiment. Maybe a clear indication of the OS bug you are referring to,
and some complete examples which demonstrate the problems you are having
would help.
Hmm, sorry if I wasn't clear. Here's a better example:
1.c:
#include stdio.h
#include windows.h
int main(int argc, char** argv)
{
HANDLE hStdOut = GetStdHandle(STD_OUTPUT_HANDLE);
fprintf(stderr, hStdOut: %08X\n, hStdOut);
DWORD dwWritten;
WriteFile(hStdOut, Something, 9, dwWritten, NULL);
return 0;
}
1.py:
import subprocess
subprocess.call([1.exe])
in cmd.exe shell:
C:\1.exe
hStdOut: 0007
Something
C:\1.exe1.txt
hStdOut: 06E0
(1.txt file now contains string Something)
C:\1.py
hStdOut: 0007
Something
C:\1.py1.txt
hStdOut: 0004
(1.txt file is now completely empty, i.e. hStdOut is invalid)
If what you say was true and stdout would default to console window's
buffer, then I would at least see Something in the console. On the
contrary output just stops working at all.
And what I expect (and always expected) is output redirection to be
inherited by child processes. The best example is when I'm writing a
batch file:
dosomething.exe /something
dosomething2.exe /somethingmore
and then execute it as batchfile.cmdsomefile.txt I always get this
behavior, i.e. all standard output (except standard error) is
redirected to somefile.txt. I'm sure on *nixes it's all the same as
well. So what documentation for STARTUPINFO specifies is already very
suspicious and uncommon, while in reality it doesn't even work as
specified!
Best regards,
Alexey.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] zipfile and unicode filenames
Also note that I'm trying to ask if zipfile should be improved, how it should be improved, and this possible improvement is not even for me (because now I know how zipfile behaves and I will work correctly with it, but someone else might stumble upon this very unexpectedly). If you want to come up with a patch: sure. The zipfile module should handle Unicode strings, encoding them in the encoding that the ZIP specification defines (both the formal one, and the informal-defined-by-pkwares-implementation). I don't think always encoding them to utf-8 (and using bit 11 of flag_bits) is a good idea, since there's a chance to create archives that won't be correctly readable by programs not supporting this bit (it's no secret that currently some programs just assume that filenames are encoded using one of system encodings). This is too complex and hazy to implement. Even if I know what is the situation on Windows (i.e. using OEM, also called DOS encoding, but I'm not sure how to determine its codec name from within python apart from calling GetConsoleCP), I'm totally unaware of the situation on other operating systems. The tricky question is what to do when reading in zipfiles with non-ASCII characters (and yes, I understand that in your case there were only ASCII characters in the file names). I don't think it should be changed. Ok, now I understand. If filename is a Unicode string, header is converted using the system encoding; depending on the exact value of header and depending on the system encoding, this may cause a decoding error. This bug has been reported as http://bugs.python.org/1170311 I see. Well, that's all easier now then, as I can just create a patch for an already existing bug. Because that's not supposed to work sanely when self.filename is unicode I'm asking if the right behavior would be to a) disallow unicode filenames in zipfile.ZipInfo, b) automatically convert filename to str in zipfile.ZipInfo, c) leave everything as it is. The correct behavior would be b); the difficult details are what encoding to use. Current zipfile seems to officially support ascii filenames only anyway, so the patch can be as simple as this: Index: Lib/zipfile.py === --- Lib/zipfile.py (revision 55850) +++ Lib/zipfile.py (working copy) @@ -252,12 +252,13 @@ self.extract_version = max(45, self.extract_version) self.create_version = max(45, self.extract_version) +filename = str(self.filename) header = struct.pack(structFileHeader, stringFileHeader, self.extract_version, self.reserved, self.flag_bits, self.compress_type, dostime, dosdate, CRC, compress_size, file_size, - len(self.filename), len(extra)) -return header + self.filename + extra + len(filename), len(extra)) +return header + filename + extra def _decodeExtra(self): # Try to decode the extra field. This doesn't introduce new features, just enforces filenames to be ascii (or whatever default encoding is) encodable. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] zipfile and unicode filenames
Current zipfile seems to officially support ascii filenames only anyway, so the patch can be as simple as this: Submitted patch and test case as http://python.org/sf/1734346 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] zipfile and unicode filenames
On 6/10/07, Martin v. Löwis [EMAIL PROTECTED] wrote: I don't think always encoding them to utf-8 (and using bit 11 of flag_bits) is a good idea, since there's a chance to create archives that won't be correctly readable by programs not supporting this bit (it's no secret that currently some programs just assume that filenames are encoded using one of system encodings). I think it is also fairly uniformly agreed that these programs are incorrect; the official encoding of file names in a zip file is Windows/DOS code page 437. Before replying to you I actually did some quick tests. I packed a file with localized filename and then opened it using explorer and also viewed it using the hexeditor: 7-Zip: directory cp866, header cp866: explorer sees correct filename. zipfile: directory cp1251, header cp1251: explorer sees incorrect filename. pkzip25.exe: directory cp866, header cp1251: explorer sees correct filenames, zipfile complains that filenames differ. zip.exe: directory cp1251, header cp1251: explorer sees incorrect filenames. Also note, that modifying filename in directory with a hex editor to cp866 made explorer see correct filenames. Another experiment with pkzip25 showed that modifying filename in directory makes it extract files with that filenam, i.e. it ignores header filename. The same behavior is showed by 7-Zip. So the general idea is that at least directory filename has some sort of convention of using oem (dos, console) encoding on Windows, cp866 in my case. Header filenames have different encodings, and seem to be ignored. I don't think that the situation on Windows is that the OEM code page should be used. Instead, CP 437 should be used, independent of the OEM code page. And on the contrary, pkzip25 made by PKWARE Inc. themselves behaves otherwise. +filename = str(self.filename) That would be incorrect, as it relies on the system encoding, which shouldn't be relied upon. Well, as I've seen in numerous examples above, system (or actually dos) encoding is actually what is used by at least by three major programs: 7-zip, pkzip25 and explorer, at least on windows. Plus, it would allow arbitrary non-string things as filenames. Hmm... why is that bad? What it should do instead (IMO) is to encode in CP437. Bonus points if it falls back to the UTF-8 feature of zip files if encoding as CP437 fails. And encoding to cp437 would be incorrect, as no currently existing program would correctly work on non-english Windows OSes. I think that letting the user deciding on the encoding is the right way to go here, as you can't know what user actually wants these days, it's all too hazy to me. And in case unicode is passed it just converts it using ascii (or default) codec. One can specify ascii codec there explicitly, if using system encoding is really an issue. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] zipfile and unicode filenames
On 6/10/07, Martin v. Löwis [EMAIL PROTECTED] wrote:
So the general idea is that at least directory filename has some sort
of convention of using oem (dos, console) encoding on Windows, cp866
in my case. Header filenames have different encodings, and seem to be
ignored.
Ok, then this is what the zipfile module should implement.
But this is only on Windows! I have no clue what's the common
situation on other OSes and don't even know how to sanely get OEM
codepage on Windows (the obvious way with ctypes.kernel32.GetOEMCP()
doesn't seem good to me).
So I guess that's bad idea anyway, maybe conforming to language bit is
better (ascii will stay ascii anyway).
What about this?
Index: Lib/zipfile.py
===
--- Lib/zipfile.py (revision 55850)
+++ Lib/zipfile.py (working copy)
@@ -252,6 +252,7 @@
self.extract_version = max(45, self.extract_version)
self.create_version = max(45, self.extract_version)
+self._encodeFilename()
header = struct.pack(structFileHeader, stringFileHeader,
self.extract_version, self.reserved, self.flag_bits,
self.compress_type, dostime, dosdate, CRC,
@@ -259,6 +260,16 @@
len(self.filename), len(extra))
return header + self.filename + extra
+def _encodeFilename(self):
+if isinstance(self.filename, unicode):
+self.filename = self.filename.encode('utf-8')
+self.flag_bits = self.flag_bits | 0x800
+
+def _decodeFilename(self):
+if self.flag_bits 0x800:
+self.filename = self.filename.decode('utf-8')
+self.flag_bits = self.flag_bits ~0x800
+
def _decodeExtra(self):
# Try to decode the extra field.
extra = self.extra
@@ -683,6 +694,7 @@
t11, (t5)0x3F, (t0x1F) * 2 )
x._decodeExtra()
+x._decodeFilename()
x.header_offset = x.header_offset + concat
self.filelist.append(x)
self.NameToInfo[x.filename] = x
@@ -967,6 +979,7 @@
extract_version = zinfo.extract_version
create_version = zinfo.create_version
+zinfo._encodeFilename()
centdir = struct.pack(structCentralDir,
stringCentralDir, create_version,
zinfo.create_system, extract_version, zinfo.reserved,
Index: Lib/test/test_zipfile.py
===
--- Lib/test/test_zipfile.py(revision 55850)
+++ Lib/test/test_zipfile.py(working copy)
@@ -515,6 +515,11 @@
# and report that the first file in the archive was corrupt.
self.assertRaises(RuntimeError, zipf.testzip)
+def testUnicodeFilenames(self):
+zf = zipfile.ZipFile(TESTFN, w)
+zf.writestr(ufoo.txt, Test for unicode filename)
+zf.close()
+
def tearDown(self):
support.unlink(TESTFN)
support.unlink(TESTFN2)
The problem is that I don't know if anything actually supports bit 11
at the time and can't even tell if I did this correctly or not. :(
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] zipfile and unicode filenames
On 6/11/07, Martin v. Löwis [EMAIL PROTECTED] wrote:
For compatibility, I would propose to use UTF-8 only if the file
name is not ASCII. Even though the OEM code pages vary, they
are (mostly) ASCII supersets. So if the string can be encoded
in ASCII, there is no need to set the UTF-8 flag bit.
Done:
Index: Lib/zipfile.py
===
--- Lib/zipfile.py (revision 55850)
+++ Lib/zipfile.py (working copy)
@@ -252,13 +252,29 @@
self.extract_version = max(45, self.extract_version)
self.create_version = max(45, self.extract_version)
+filename, flag_bits = self._encodeFilenameFlags()
header = struct.pack(structFileHeader, stringFileHeader,
- self.extract_version, self.reserved, self.flag_bits,
+ self.extract_version, self.reserved, flag_bits,
self.compress_type, dostime, dosdate, CRC,
compress_size, file_size,
- len(self.filename), len(extra))
-return header + self.filename + extra
+ len(filename), len(extra))
+return header + filename + extra
+def _encodeFilenameFlags(self):
+if isinstance(self.filename, unicode):
+try:
+return self.filename.encode('ascii'), self.flag_bits
+except UnicodeEncodeError:
+return self.filename.encode('utf-8'), self.flag_bits | 0x800
+else:
+return self.filename, self.flag_bits
+
+def _decodeFilenameFlags(self):
+if self.flag_bits 0x800:
+return self.filename.decode('utf-8'), self.flag_bits ~0x800
+else:
+return self.filename, self.flag_bits
+
def _decodeExtra(self):
# Try to decode the extra field.
extra = self.extra
@@ -684,6 +700,7 @@
x._decodeExtra()
x.header_offset = x.header_offset + concat
+x.filename, x.flag_bits = x._decodeFilenameFlags()
self.filelist.append(x)
self.NameToInfo[x.filename] = x
if self.debug 2:
@@ -967,16 +984,17 @@
extract_version = zinfo.extract_version
create_version = zinfo.create_version
+filename, flag_bits = zinfo._encodeFilenameFlags()
centdir = struct.pack(structCentralDir,
stringCentralDir, create_version,
zinfo.create_system, extract_version, zinfo.reserved,
- zinfo.flag_bits, zinfo.compress_type, dostime, dosdate,
+ flag_bits, zinfo.compress_type, dostime, dosdate,
zinfo.CRC, compress_size, file_size,
- len(zinfo.filename), len(extra_data), len(zinfo.comment),
+ len(filename), len(extra_data), len(zinfo.comment),
0, zinfo.internal_attr, zinfo.external_attr,
header_offset)
self.fp.write(centdir)
-self.fp.write(zinfo.filename)
+self.fp.write(filename)
self.fp.write(extra_data)
self.fp.write(zinfo.comment)
Index: Lib/test/test_zipfile.py
===
--- Lib/test/test_zipfile.py(revision 55850)
+++ Lib/test/test_zipfile.py(working copy)
@@ -515,6 +515,12 @@
# and report that the first file in the archive was corrupt.
self.assertRaises(RuntimeError, zipf.testzip)
+def testUnicodeFilenames(self):
+zf = zipfile.ZipFile(TESTFN, w)
+zf.writestr(ufoo.txt, Test for unicode filename)
+assert isinstance(zf.infolist()[0].filename, unicode)
+zf.close()
+
def tearDown(self):
support.unlink(TESTFN)
support.unlink(TESTFN2)
What I also changed is to encode filenames only for writing to the
target file, without damaging ZipInfo. The reason for this is that if
user decides to enumerate infolist after she wrote files to ZipFile,
she would expect ZipInfo.filename to be what she passed to
ZipFile.write/ZipFile.writestr.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] zipfile and unicode filenames
On 6/11/07, Alexey Borzenkov [EMAIL PROTECTED] wrote: The problem is that I don't know if anything actually supports bit 11 at the time and can't even tell if I did this correctly or not. :( I downloaded the latest WinZip and can confirm that it parses utf-8 filenames correctly (although it seems to treat presence of bit 11 more like enabling autodetection mode, not strict utf-8, but it must be because it has to cope with lots of incorrect zip files), i.e. in the presence of bit 11 it understands filename to be utf-8, without presence of bit 11 it treats it just like oem. :) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] zipfile and unicode filenames
Hi everyone, Today I've stumbled upon a bug in my program that wasn't very straightforward to understand. The problem is that I was passing unicode filenames to zipfile.ZipFile.write and I had sys.setdefaultencoding() in effect, which resulted in a situation where most of the bytes generated in zipfile.ZipInfo.FileHeader would pass thru, except for a few, which caused codec error on another machine (where filenames got infectiously upgraded to unicode). The problem here is that it was absolutely unclear at first that I get unicode filenames passed to write, and it incorrectly accepted them silently. Is it worth to submit a bug report on this? The desired behavior here would be to either a) disallow unicode strings as arcname are raise an exception (since it is used in concatenation with raw data it is likely to cause problems because of auto upgrading raw data to unicode), or b) silently encode unicode strings to raw strings (something like if isinstance(filename, unicode): filename = filename.encode() in zipfile.ZipInfo constructor). So, should I submit a bug report, and which behavior would be actually correct? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adventures with x64, VS7 and VS8 on Windows
On 5/23/07, Kristján Valur Jónsson [EMAIL PROTECTED] wrote: Install in the ProgramFiles folder. Only over my dead body. *This* is silly. Bill doesn't think so. And he gets to decide. I mean we do want to play nice, don't we? Nothing installs itself in the root anymore, not since windows 3.1 Maybe installing in the root is not good, but installing to Program Files is just asking for trouble. All sorts of development tools might suddenly break because of that space in the middle of the path and requirement to use quotes around it. I thus usually install things to drive:\Programs. I'm not sure if any packages/programs will break because of that space, but what if some will? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 30XZ: Simplified Parsing
On 4/30/07, Jim Jewett [EMAIL PROTECTED] wrote: Python initially inherited its parsing from C. While this has been generally useful, there are some remnants which have been less useful for python, and should be eliminated. + Implicit String concatenation + Line continuation with \ I don't know if I can vote, but if I could I'd be -1 on this. Can't say I'm using continuation often, but there's one case when I'm using it and I'd like to continue using it: #!/usr/bin/env python \ Usage: some-tool.py [arguments...] Does this and that based on its arguments if condition: print __doc__ sys.exit(1) This way usage immediately stands out much better, without any unnecessary new lines. Best regards, Alexey. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.5.1
On 5/1/07, Martin v. Löwis [EMAIL PROTECTED] wrote:
After doing some research I found that it seems to be impossible to
use CreateFile for a file that doesn't have SHARE_READ. I played with
different combinations and with FLAG_BACKUP_SEMANTICS and nothing
helped. However on Windows there's still a possibility to read
attributes: use FindFirstFile. _WIN32_FIND_DATA structure seems to
have all the necessary fields (attributes, file times, size and
full/short filename), and FindFirstFile doesn't care about sharing at
all...
So what about GetFileAttributesEx? What are the conditions under which
I can successfully invoke it?
Seems to have the same problems as with CreateFile(...):
// 1.cc
#include windows.h
#include stdio.h
int main(int argc, char** argv)
{
WIN32_FILE_ATTRIBUTE_DATA data;
if(!GetFileAttributesEx(pagefile.sys, GetFileExInfoStandard,
(LPVOID)data))
{
char buffer[1024];
FormatMessage(FORMAT_MESSAGE_FROM_SYSTEM, NULL, GetLastError(),
0,
buffer, 1024, NULL);
printf(Error %d: %s\n, GetLastError(), buffer);
return 1;
}
printf(Success\n);
return 0;
}
// 2.cc
#include windows.h
#include stdio.h
int main(int argc, char** argv)
{
WIN32_FIND_DATA data;
if(INVALID_HANDLE_VALUE == FindFirstFile(pagefile.sys, data))
{
char buffer[1024];
FormatMessage(FORMAT_MESSAGE_FROM_SYSTEM, NULL, GetLastError(),
0,
buffer, 1024, NULL);
printf(Error %d: %s\n, GetLastError(), buffer);
return 1;
}
printf(Success\n);
return 0;
}
C:\C:\3\1.exe
Error 32: The process cannot access the file because it is being used
by another process.
C:\C:\3\2.exe
Success
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.5.1
On 5/1/07, Martin v. Löwis [EMAIL PROTECTED] wrote: That code only tests it for pagefile.sys. My question was about open handles in general. Both Calvin Spealman and I found that you cannot reproduce the problem when you, in Python 2.5.0, open a file, and then try to os.stat() it - even though, in Python 2.5.0, os.stat() will perform GetFileAttributesEx. So even though we opened the file with not passing any sharing flags, we could still do GetFileAttributesEx on it. I now studied the CRT sources, and it seems that if you use a regular open() call, the CRT will pass FILE_SHARE_READ | FILE_SHARE_WRITE to CreateFile. You would have to use _sopen in the CRT to create any kind of sharing conflict, and that isn't exposed in Python. Wow, I'm very sorry, I didn't realize how much special pagefile.sys and hiberfil.sys are. As it turns out, even if you create a file with no sharing allowed, you can still open it with backup semantics in other processes, and thus can use GetFileAttributesEx, GetFileTime, etc. The file pagefile.sys seems almost magical then, I don't understand how it's opened to behave like that. The difference is also immediately visible if you try to open Properties of pagefile.sys, you won't even see Security tab there (even when I create file something.txt and then remove all ACLs, including SYSTEM, I can't access the file, but I can see Security tab and can grant myself permissions back), it looks like all kinds of opening that file are denied. Maybe this is a special security feature, so that no process could access swapped pages (otherwise it could be possible with backup semantics). Thus you can't access the file itself, you can only access containing directory. So I guess we need continue using pagefile.sys as a test case. Seems to be true, it's just maybe it shouldn't be hardcoded to C:\ There's REG_MULTI_SZ PagingFiles in HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management, btw. The format seems to be filename minmbsize maxmbsize for every line. Best regards, Alexey. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.5.1
Hi Martin, On 4/29/07, Martin v. Löwis [EMAIL PROTECTED] wrote: The error Windows reports is ERROR_SHARING_VIOLATION. I never understood sharing fully, but it may be that if the file is opened in exclusive sharing, stat'ing it may fail. Sharing is actually very easy. If you didn't specify SHARE_READ/SHARE_WRITE/SHARE_DELETE when opening the file and you succeeded, then any process can't read/write/delete the file until you close the handle. I personally consider it a bug in Windows that you cannot get file attributes if some other process has opened it. Exclusive access should only restrict access to file contents, but not file attributes. After doing some research I found that it seems to be impossible to use CreateFile for a file that doesn't have SHARE_READ. I played with different combinations and with FLAG_BACKUP_SEMANTICS and nothing helped. However on Windows there's still a possibility to read attributes: use FindFirstFile. _WIN32_FIND_DATA structure seems to have all the necessary fields (attributes, file times, size and full/short filename), and FindFirstFile doesn't care about sharing at all... ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] concerns regarding callable() method
On 4/8/07, Paul Pogonyshev [EMAIL PROTECTED] wrote: assert hasattr(x, '__call__') I note that callable() was introduced before all callable objects had a __call__ attribute. This is no longer the case, so it's not needed. I just didn't think about that possibility. If that works the same way, callable() is just a sugar and not something unimplementable in other ways. Therefore, my objection is discarded. (But PEP 3100 should probably be update to mention this, otherwise you may get this complaint again ;) I whole-heartedly agree here, because people who start learning python (like me some time ago) usually learn that it's bad to test for __call__ (can't remember where I read about that though), and that we should always use callable() to be on a safe side. When I first heard that callable() was going to be removed I kinda panicked myself, it's good to know that at least there's still a way to be sure that object is callable or not... ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] concerns regarding callable() method
On 4/8/07, Guido van Rossum [EMAIL PROTECTED] wrote: On 4/8/07, Paul Pogonyshev [EMAIL PROTECTED] wrote: Guido van Rossum wrote: What if someone passes a callable that doesn't have the expected signature? Well, I don't know a way to catch such situations now, so removing callable() will not make it worse (even if you don't know about hasattr trick above.) My point is that it's futile to use callable() -- even if it passes, you have no assurance that you actually have a valid callback. So why bother with it at all? It's counter to the spirit of Python. If someone passes you a bad callback, they will see a traceback when you call it. Then they fix their program. That's how it's supposed to work. But what if you need to do different things based on argument is callable or not? Take for example Dependency Injection recipe: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/413268 It uses callable to differentiate whether it needs to use object as singleton or to instantiate it on each request. I'm sure there might be other uses for callable when it's really useful. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] splitext('.cshrc')
On 3/7/07, Josiah Carlson [EMAIL PROTECTED] wrote:
Martin v. Löwis [EMAIL PROTECTED] wrote:
Now it's becoming difficult: several people in favor, some opposed...
What about changing the semantics of splitext and creating a new
function (available on all platforms) that does what the Windows version
currently does?
I don't understand only one thing, why do people need new functions?
You can anticipate the change today, and write functions that do
exactly what you need no matter which way (current or proposed) python
implements:
def ensurenew(path):
a,b = os.path.splitext(path)
if a == '': # also possibly check if a contains dots only
return b,a
return a,b
def ensureold(path):
a,b = os.path.splitext(path)
if b == '' and a.startswith('.'): # also possibly check if a
starts with multiple dots
return b,a
return a,b
Best regards,
Alexey.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] splitext('.cshrc')
On 3/8/07, Alexey Borzenkov [EMAIL PROTECTED] wrote: On 3/7/07, Josiah Carlson [EMAIL PROTECTED] wrote: Martin v. Löwis [EMAIL PROTECTED] wrote: Now it's becoming difficult: several people in favor, some opposed... What about changing the semantics of splitext and creating a new function (available on all platforms) that does what the Windows version currently does? I don't understand only one thing, why do people need new functions? You can anticipate the change today, and write functions that do exactly what you need no matter which way (current or proposed) python implements: [...snip...] On second thought I completely forgot about path parts. Best regards, Alexey. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Floor division
On 1/24/07, Gareth McCaughan [EMAIL PROTECTED] wrote: complex(complex(1.0, 2.0), complex(10.0, 20.0)) (-19+12j) WTF? In any case, that's also what's destroying the sign of the imaginary part in complex(1.0, -0.0). It seems pretty clear what it thinks it's doing -- namely, defining complex(a,b) = a + ib even when a,b are complex. And half of why it does that is clear: you want complex(a)=a when a is complex. Why b should be allowed to be complex too, though, it's hard to imagine. I think that's the right thing to do, because that is mathematically correct. j is just an imaginary number with a property that j*j = -1. So (a+bj) + (c+dj)j = (a-d) + (b+c)j. Complex numbers are not just magic pairs with two numbers and have actual mathematical rules. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Floor division
On 1/24/07, Gareth McCaughan [EMAIL PROTECTED] wrote: [Alexey:] I think that's the right thing to do, because that is mathematically correct. j is just an imaginary number with a property that j*j = -1. So (a+bj) + (c+dj)j = (a-d) + (b+c)j. Yes, thanks, I know what j is, and I know how to multiply complex numbers. (All of which you could have deduced from reading what I wrote, as it happens.) The question is whether it makes sense to define complex(a,b) = a+ib for all a,b or whether the two-argument form is always in practice going to be used with real numbers[1]. If it is, which seems pretty plausible to me, then changing complex() to complain when passed two complex numbers would (1) notify users sooner when they have errors in their programs, (2) simplify the code, and (3) avoid the arguably broken behaviour Tim was remarking on, where complex(-0.0).real is +0 instead of -0. Haven't looked in source code for complex constructor yet, but I think that if it changes sign of -0.0 then it just does something wrong and needs fixing without change in behaviour. Maybe it could check if numbers it got on input are real or complex and proceed accordingly so that it never gets to computing -0.0-(+0.0), i.e. when second argument is not a complex number it could just add it to imaginary part of first argument, but skip substracting inexistant 0.0 from first argument's real part. Change of behaviour like ignoring imaginary part of second argument seems bad to me, and that's my only point. Besides, documentation (at least for Python 2.4) clearly states that second argument can be a complex number: *complex*( [real[, imag]]) Create a complex number with the value *real + imag*j* or convert a string or number to a complex number. If the first parameter is a string, it will be interpreted as a complex number and the function must be called without a second parameter. The second parameter can never be a string. Each argument may be *any numeric type* (*including complex*). If imag is omitted, it defaults to zero and the function serves as a numeric conversion function like int(), long() and float(). If both arguments are omitted, returns 0j. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] features i'd like [Python 3000?] ... #4: interpolated strings ala perl
On 12/4/06, Josiah Carlson [EMAIL PROTECTED] wrote: With the proper mapping, this is trivial... class namelookup: [...snip...] foo = foo() print %(foo.b)i + %(foo.a)i%namelookup(locals()) 2 + 1 It can even be simpler and more powerful: class evallookup: def __init__(self, nsg, nsl): self.nsg = nsg self.nsl = nsl def __getitem__(self, name): return eval(name, self.nsg, self.nsl) class foo: a = 1 b = 2 print %(foo.a)i + %(foo.b)i = %(foo.a + foo.b)i % evallookup(globals(), locals()) 1 + 2 = 3 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why spawnvp not implemented on Windows?
On 10/13/06, Fredrik Lundh [EMAIL PROTECTED] wrote: any reason you cannot just use the subprocess module instead, like everyone else? Oh! Wow! I just simply didn't know of its existance (I'm pretty much new to python), and both distutils and SCons (I was looking inside them because they are major build systems and surely had to execute compilers somehow), and upon seeing that each of them invented their own method of searching path created a delusion as if inventing custom workarounds was the only way... Sorry... x_x ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Why spawnvp not implemented on Windows?
Hi all, I've been looking at python 2.5 today and what I notices is absense of spawnvp with this comment in os.py: # At the moment, Windows doesn't implement spawnvp[e], # so it won't have spawnlp[e] either. I'm wondering, why so? Searching MSDN I can see that these functions are implemented in CRT: spawnvp: http://msdn2.microsoft.com/en-us/library/275khfab.aspx spawnvpe: http://msdn2.microsoft.com/en-us/library/h565xwht.aspx I can also see that spawnvp and spawnvpe are currently wrapped in posixmodule.c, but for some reason on OS/2 only. Forgive me if I'm wrong but shouldn't it work when #if defined(PYOS_OS2) is changed to #if defined(PYOS_OS2) || defined(MS_WINDOWS) around spawnvp and spawnvpe wrappers and in posix_methods? At least when I did it with my copy, nt.spawnvp seems to work fine... ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why spawnvp not implemented on Windows?
On 10/12/06, Alexey Borzenkov [EMAIL PROTECTED] wrote: At least when I did it with my copy, nt.spawnvp seems to work fine... Hi everyone again. I've created patch for spawn*p*, as well as for exec*p* against trunk, so that when possible it uses crt's execvp[e] (defined via HAVE_EXECVP, if there are other platforms that have it they will need to define HAVE_EXECVP and HAVE_SPAWNVP). Fix is in os.py and posixmodule.c: http://snaury.googlepages.com/python-win32-spawn_p_.patch Should I submit it to sourceforge as a patch, or someone can review it as is? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why spawnvp not implemented on Windows?
On 10/13/06, Martin v. Löwis [EMAIL PROTECTED] wrote: Please consider also exposing _wspawnvp, depending on whether path argument is a Unicode object or not. See PEP 277 for guidance. Since this would go into 2.6, support for Windows 95 isn't mandatory. Umm... do you mean that spawn*p* on python 2.5 is an absolute no? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why spawnvp not implemented on Windows?
Forgot to include python-dev... On 10/13/06, Martin v. Löwis [EMAIL PROTECTED] wrote: Umm... do you mean that spawn*p* on python 2.5 is an absolute no? Yes. No new features can be added to Python 2.5.x; Python 2.5 has already been released. Ugh... that's just not fair. Because of this there will be no spawn*p* in python for another two years. x_x I have a workaround for this, that tweaks os module: [...snip wrong code...] It should have been: if (not (hasattr(os, 'spawnvpe') or hasattr(os, 'spawnvp')) and hasattr(os, 'spawnve') and hasattr(os, 'spawnv')): def _os__spawnvpe(mode, file, args, env=None): import sys from errno import ENOENT, ENOTDIR from os import path, spawnve, spawnv, environ, defpath, pathsep, error if env is not None: func = spawnve argrest = (args, env) else: func = spawnv argrest = (args,) env = environ head, tail = path.split(file) if head: return func(mode, file, *argrest) if 'PATH' in env: envpath = env['PATH'] else: envpath = defpath PATH = envpath.split(pathsep) if os.name == 'nt' or os.name == 'os2': PATH.insert(0, '') saved_exc = None saved_tb = None for dir in PATH: fullname = path.join(dir, file) try: return func(mode, fullname, *argrest) except error, e: tb = sys.exc_info()[2] if (e.errno != ENOENT and e.errno != ENOTDIR and saved_exc is None): saved_exc = e saved_tb = tb if saved_exc: raise error, saved_exc, saved_tb raise error, e, tb def _os_spawnvp(mode, file, args): return os._spawnvpe(mode, file, args) def _os_spawnvpe(mode, file, args, env): return os._spawnvpe(mode, file, args, env) def _os_spawnlp(mode, file, *args): return os._spawnvpe(mode, file, args) def _os_spawnlpe(mode, file, *args): return os._spawnvpe(mode, file, args[:-1], args[-1]) os._spawnvpe = _os__spawnvpe os.spawnvp = _os_spawnvp os.spawnvpe = _os_spawnvpe os.spawnlp = _os_spawnlp os.spawnlpe = _os_spawnlpe os.__all__.extend([spawnvp, spawnvpe, spawnlp, spawnlpe]) But the fact that I have to use similar code anywhere I need to use spawnlp is not fair. Notice that _spawnvpe is simply a clone of _execvpe from os.py, maybe if the problem is new API in c source, this approach could be used in os.py? P.S. Although it's a bit stretching, one might also say that implementing spawn*p* on windows is not actually a new feature, and rather is a bugfix for misfeature. Why every other platform can benefit from spawn*p* and only Windows can't? This just makes os.spawn*p* useless: it becomes unreliable and can't be used in portable code at all. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
