[issue2810] _winreg.EnumValue fails when the registry data includes multibyte unicode characters
Daniel Stutzbach dan...@stutzbachenterprises.com added the comment: I just found a one line example of the problem: EnumValue(HKEY_PERFORMANCE_DATA, 0) Traceback (most recent call last): File stdin, line 1, in module WindowsError: [Error 234] More data is available Other functions are also affected: QueryValueEx(HKEY_PERFORMANCE_DATA, None) Traceback (most recent call last): File stdin, line 1, in module WindowsError: [Error 234] More data is available In a nutshell, the Python implementation of these functions works like this: 1) Query the API for the length of the value 2) Allocate a buffer of that size 3) Query the API for the value However, the API functions called in step #1 are not guaranteed to be accurate. It works *most* of the time, but if step #3 returns ERROR_MORE_DATA then we need to dynamically grow the buffer and try again until the data fits. I'll have a patch (with test cases) ready later today. -- versions: +Python 3.1, Python 3.2, Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2810 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2810] _winreg.EnumValue fails when the registry data includes multibyte unicode characters
anatoly techtonik techto...@gmail.com added the comment: Great analysis! -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2810 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2810] _winreg.EnumValue fails when the registry data includes multibyte unicode characters
Changes by Brian Curtin cur...@acm.org: -- assignee: - brian.curtin priority: - normal stage: - needs patch versions: -Python 2.4, Python 2.5, Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2810 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2810] _winreg.EnumValue fails when the registry data includes multibyte unicode characters
Changes by Brian Curtin cur...@acm.org: -- assignee: brian.curtin - ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2810 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2810] _winreg.EnumValue fails when the registry data includes multibyte unicode characters
Changes by Daniel Stutzbach dan...@stutzbachenterprises.com: Added file: http://bugs.python.org/file16802/winreg_test.pach ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2810 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2810] _winreg.EnumValue fails when the registry data includes multibyte unicode characters
Changes by Daniel Stutzbach dan...@stutzbachenterprises.com: -- keywords: +patch Added file: http://bugs.python.org/file16803/winreg.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2810 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2810] _winreg.EnumValue fails when the registry data includes multibyte unicode characters
Changes by Daniel Stutzbach dan...@stutzbachenterprises.com: Removed file: http://bugs.python.org/file16728/more_data_is_available.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2810 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2810] _winreg.EnumValue fails when the registry data includes multibyte unicode characters
Changes by Daniel Stutzbach dan...@stutzbachenterprises.com: Removed file: http://bugs.python.org/file16729/more_data_is_available2.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2810 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2810] _winreg.EnumValue fails when the registry data includes multibyte unicode characters
Daniel Stutzbach dan...@stutzbachenterprises.com added the comment: I've uploaded two patches (against trunk): one to add test cases that demonstrate this bug and another to fix the bug. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2810 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2810] _winreg.EnumValue fails when the registry data includes multibyte unicode characters
anatoly techtonik techto...@gmail.com added the comment: If this doesn't relate to multibyte strings anymore, but just to long strings then I'd open new bug. If even regedit fails to query then maybe its WinAPI flaw? Maybe it will worth to try ctypes. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2810 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2810] _winreg.EnumValue fails when the registry data includes multibyte unicode characters
Daniel Stutzbach dan...@stutzbachenterprises.com added the comment: It was never 100% clear that it ever related to multi-byte strings, so (as the original reporter) I'd prefer to continue using this bug and just update the title to: _winreg.EnumValue sometimes raises WindowsError (More data is available). The patch I have planned will fix the problem regardless of the cause, (except for impossible-to-fix causes, such as if there really is an API flaw). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2810 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2810] _winreg.EnumValue fails when the registry data includes multibyte unicode characters
Daniel Stutzbach dan...@stutzbachenterprises.com added the comment: I just wrote a C program that can read the long key name just fine, so it's not a Windows API bug. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2810 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2810] _winreg.EnumValue fails when the registry data includes multibyte unicode characters
anatoly techtonik techto...@gmail.com added the comment: I propose to close this as invalid, because the bug with _winreg.EnumValue can not be confirmed. However, it seems to be impossible to return unicode data from _winreg.EnumKey, and this deserves a new bug. -- nosy: +techtonik versions: +Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2810 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2810] _winreg.EnumValue fails when the registry data includes multibyte unicode characters
Changes by Brian Curtin cur...@acm.org: -- nosy: +brian.curtin ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2810 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2810] _winreg.EnumValue fails when the registry data includes multibyte unicode characters
Changes by anatoly techtonik techto...@gmail.com: Added file: http://bugs.python.org/file16726/issue2810_registry.png ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2810 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2810] _winreg.EnumValue fails when the registry data includes multibyte unicode characters
Daniel Stutzbach dan...@stutzbachenterprises.com added the comment: Attached is a file that reliably produces the More data is available error from _winreg.EnumValue in Python 2.6. The script triggers the error via a race condition, by modifying the value after PyEnumValue() calls RegQueryInfoKey() but before it calls RegEnumValue(). I don't know if that's what caused the original problem for my end-user, but it certainly is one way to trigger the problem. I can work on a patch next week for a proper test case and to fix PyEnumValue(). -- Added file: http://bugs.python.org/file16728/more_data_is_available.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2810 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2810] _winreg.EnumValue fails when the registry data includes multibyte unicode characters
Daniel Stutzbach dan...@stutzbachenterprises.com added the comment: Here's another script that causes a More data is available error. This one creates a key with a name that's exactly 256 characters long. (The Windows Registry Editor can't display the key either) I'm testing this on XP. Newer versions of Windows might handle long key names better, or have a different magic length that causes failures. -- Added file: http://bugs.python.org/file16729/more_data_is_available2.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2810 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2810] _winreg.EnumValue fails when the registry data includes multibyte unicode characters
Martin v. Löwis [EMAIL PROTECTED] added the comment: IIUC, no fix for this bug has been proposed, so the issue is no candidate for 2.5.3. -- versions: -Python 2.5.3 ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2810 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2810] _winreg.EnumValue fails when the registry data includes multibyte unicode characters
Changes by Daniel Stutzbach [EMAIL PROTECTED]: -- versions: +Python 2.5.3 ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2810 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2810] _winreg.EnumValue fails when the registry data includes multibyte unicode characters
Erik Nolte [EMAIL PROTECTED] added the comment: To reproduce on Windows XP with Python 2.4, 2.5, and 2.6: 1. Import issue_2810.reg (it creates two keys under HKCU\PythonTestUnicodeKeys: one with a one byte unicode char in the key, the other with multiple 2-byte unicode chars) 2. Run the following: import _winreg key = _winreg.OpenKey( _winreg.HKEY_CURRENT_USER, PythonTestUnicodeKeys ) one_byte_key = _winreg.EnumKey( key, 0 ) two_byte_key = _winreg.EnumKey( key, 1 ) _winreg.CloseKey( key ) try: unicode( one_byte_key ) except Exception, ex: print EnumKey didn't return a valid string:, ex print should be unicode, not str:, two_byte_key.__class__ for ch in two_byte_key: print ord( ch ), print -- nosy: +enolte versions: +Python 2.4, Python 2.6 -Python 3.0 Added file: http://bugs.python.org/file11681/issue_2810.reg ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2810 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2810] _winreg.EnumValue fails when the registry data includes multibyte unicode characters
Daniel Stutzbach [EMAIL PROTECTED] added the comment: After several failed attempts at making a test case, and stepping through C code with a debugger, I see that my initial diagnose is quite wrong. RegQueryInfoKey *does* return the sizes in units of bytes (even though the Microsoft documentation says otherwise). My apologies. I do still have a stack trace from an end-user of my python2.5-based product, showing that _winreg.EnumValue raises: WindowsError: [Error 234] More data is available The application reliably crashes on start-up for this user, when trying to read some registry entries written by another program and hitting the above exception. Unfortunately, I have been unable to reproduce the problem locally. I tried a variety of Unicode characters (including some that encode to 4 bytes), and that didn't raise an exception. I also tried putting some very long data strings (more than 64kb) into the registry, and that worked fine too (even though the Microsoft documentation says the ANSI version *should* return the above exception!). I'm going to try building a custom PyEnumValue that will dynamically grow the buffer size when that error occurs. I'll report back on how that works out for the end user. In the meantime, I'm open to other theories on what might cause RegEnumValue to fail with that error. The end user is running Vista, if it matters. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue2810 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2810] _winreg.EnumValue fails when the registry data includes multibyte unicode characters
New submission from Daniel Stutzbach [EMAIL PROTECTED]: _winreg.EnumValue raises a WindowsError (More data is available) if the registry data includes multibyte unicode characters. Inspecting PyEnumValue in _winreg.c, I believe I see the problem. The function uses RegQueryInfoKey to determine the maximum data and key name sizes to pass to RegEnumValue. Unfortunately, RegQueryInfoKey returns the size in number of unicode characters, while RegEnumValue expects a size in bytes. This is OK if all the values are ASCII, but it fails if there are any multibyte unicode characters. I believe it would be sufficient to multiply the sizes by 4, since that's the maximum width of a unicode character. The bug exists in at least Python 2.5 and Python 3.0 (based on source code inspection). References: RegEnumValue: http://msdn.microsoft.com/en-us/library/ms724865(VS.85).aspx RegQueryInfoKey: http://msdn.microsoft.com/en-us/library/ms724902(VS.85).aspx -- components: Windows messages: 66542 nosy: stutzbach severity: normal status: open title: _winreg.EnumValue fails when the registry data includes multibyte unicode characters type: behavior versions: Python 2.5, Python 3.0 __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue2810 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2810] _winreg.EnumValue fails when the registry data includes multibyte unicode characters
Martin v. Löwis [EMAIL PROTECTED] added the comment: Is that for Python 2.5 or 3.0? -- nosy: +loewis __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue2810 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2810] _winreg.EnumValue fails when the registry data includes multibyte unicode characters
Daniel Stutzbach [EMAIL PROTECTED] added the comment: The bug is in both. On Sat, May 10, 2008 at 12:52 PM, Martin v. Löwis [EMAIL PROTECTED] wrote: Martin v. Löwis [EMAIL PROTECTED] added the comment: Is that for Python 2.5 or 3.0? -- nosy: +loewis __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue2810 __ __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue2810 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2810] _winreg.EnumValue fails when the registry data includes multibyte unicode characters
Martin v. Löwis [EMAIL PROTECTED] added the comment: Can you please provide a test case then? The 3.0 code doesn't use RegQueryInfoKey, but RegQueryInfoKeyW. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue2810 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
