[issue17289] readline.set_completer_delims() doesn't play well with others
Bradley Froehle added the comment: Patch attached. I implemented this by adding a 'static char *' which holds the memory we allocate. I did not use the PyState machinery. -- keywords: +patch Added file: http://bugs.python.org/file30143/readline_completer_state.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17289 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17289] readline.set_completer_delims() doesn't play well with others
New submission from Bradley Froehle: The `readline.set_completer_delims` doesn't play well with others because it assumes that only it ever allocates or modifies the rl_completer_word_break_characters buffer. If other programs modify this value, for example changing it from heap allocated space to stack allocated space, the results can be catastrophic. To remind you, the function essentially works as: set_completer_delims(PyObject *self, PyObject *args) { // ... free((void*) rl_completer_word_break_characters; rl_completer_word_break_characters = strdup(break_chars); // ... } where `break_chars` is the user provided string. Take, for example, R as another programs which changes the readline completer strings. When an embedded R instance is initialized (say, using `r2py`) something similar to the following takes place:: static void set_rl_completer_word_break_characters(const char *new) { static char[201] buffer; strncpy(buffer, new, 200); rl_completer_word_break_characters = buffer; } static void initialize_embedded_R(...) { // ... set_rl_completer_word_break_characters(...); } As you might expect the next trip through `readline.set_completer_delims` after initializing R will be catastrophic when we attempt to free a stack allocate buffer. I think we should consider modifying the `readline.set_completer_delims` to store the allocated buffers in the module state:: set_completer_delims(PyObject *self, PyObject *args) { // ... free(_readlinestate_global-break_chars); rl_completer_word_break_characters = strdup(break_chars); _readlinestate_global-break_chars = rl_completer_word_break_characters; // ... } This would prevent the segfault and memory leaks, and would render weird hacks (like https://bitbucket.org/lgautier/rpy2/commits/408bae913653 in the r2py code) unnecessary. -- components: Extension Modules messages: 182882 nosy: bfroehle priority: normal severity: normal status: open title: readline.set_completer_delims() doesn't play well with others type: crash versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17289 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16612] Integrate Argument Clinic specialized preprocessor into CPython trunk
Bradley Froehle added the comment: As for the docstring: I would like it better if I could avoid typing the cumbersome \n\s. I agree with Stefan that the file is a lot more readable if the docstring is not repeated twice. It's unfortunate that C doesn't have a notion of a raw string (as opposed to C++11 with the new R(...) syntax) but I think it is something we'll have to live with. I would have expected that a good text editor would be able to convert a selected region into a C string, but I've never actually seen such a feature. In general I think we should aim for clarity in scope of the arguments in the DSL -- either by using curly-braces (a C construct) or indentation (a Python construct). To minimize the usage of vertical space, I'd like to see the DSL not require a blank line between arguments. In a project I worked on recently I ended up writing a parser to go through a list of C arguments and automatically produce the PyArg_ParseTuple / PyArg_ParseTupleAndKeywords lines. It's not as full featured as what we are looking for here, but it did have the benefit of minimizing the number of extra vertical lines. For example:: static PyObject * w_rktime(PyObject *self, PyObject *args, PyObject *kwargs) { /*[kwargs rktime]*/ darray u; meshdata msh; double dt; int nsteps=1; /*[/kwargs]*/ static char *_keywords[] = {u, msh, dt, nsteps, NULL}; if (!PyArg_ParseTupleAndKeywords(args, kwargs, OOd|i:rktime, _keywords, view_as_darray, u, DgMeshData_Converter, msh, dt, nsteps)) return NULL; /*[checksum=...]*/ ... } I could imagine extending such a syntax to allow custom converters and other extensions using comments:: path_t path = PATH_T_INITIALIZE(stat, 0, 1) /* converter = path_converter; default = None */; int dir_fd = DEFAULT_DIR_FD /* converter = OS_STAT_DIR_FD_CONVERTER */; The biggest downside to this approach would be that the parser could not inject C code directly into the global scope -- instead it would be limited to producing #define lines. -Brad -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16612 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17162] Py_LIMITED_API needs a PyType_GenericDealloc
New submission from Bradley Froehle: I tried to implement a custom extension type using PyType_FromSpec and Py_LIMITED_API but couldn't implement tp_dealloc: static void mytype_dealloc(mytypeobject *self) { // free some fields in mytypeobject Py_TYPE(self)-tp_free((PyObject *) self); // XXX } The line marked XXX will not compile in Py_LIMITED_API because there is no access to the fields of PyTypeObject. There doesn't seem to be any function in the API which just calls tp_free. I suggest the addition of an API function (to be included in Py_LIMITED_API): void PyType_GenericDealloc(PyObject *self) { Py_TYPE(self)-tp_free(self); } -- messages: 181689 nosy: bfroehle priority: normal severity: normal status: open title: Py_LIMITED_API needs a PyType_GenericDealloc type: enhancement versions: Python 3.4, Python 3.5 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17162 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17162] Py_LIMITED_API needs a PyType_GenericDealloc
Bradley Froehle added the comment: I should mention that essentially what I'm advocating is renaming and exposing `object_dealloc` in Objects/typeobject.c. The proper name is not obvious to me... should it be PyObject_GenericDealloc since it acts on objects? Or PyType_GenericDealloc since it will be stuck into one of the PyTypeObject slots? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17162 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5309] packaging doesn't parallelize extension module compilation
Changes by Bradley Froehle brad.froe...@gmail.com: -- nosy: +bfroehle ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5309 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16612] Integrate Argument Clinic specialized preprocessor into CPython trunk
Bradley Froehle added the comment: First off, thanks for all the work so far. This has proven incredibly useful to me in a personal project. However, I think there needs to be some additional discussion of how to handle situations where the arguments passed to PyArg_ParseTuple require additional cleanup. As a prototypical example, I'll consider filename arguments. The Python docs recommend that filename arguments be handled with `O` and PyUnicode_FSConverter. How can we handle this in clinic? 1. No special handling in clinic: /*[clinic] foo - None PyObject *filename [clinic]*/ ... foo_impl(PyObject *self, PyObject *filename) /*[clinic end:...*/ { char *c_filename; PyObject *b_filename; if (!PyUnicode_FSConverter(filename, b_filename)) return NULL; c_filename = PyBytes_AsString(b_filename); // ... Py_DECREF(b_filename); } This offloads all of the processing to the impl function and leaves us no better off. Unacceptable. 2. Use PyObject* and a converter option: /*[clinic] foo - None PyBytesObject *filename converter=PyUnicode_FSConverter [clinic]*/ ... foo_impl(PyObject *self, PyBytesObject *filename) /*[clinic end:...]*/ { char *c_filename = PyBytes_AsString(filename); ... Py_DECREF(filename); } This is much more convenient, but the `_impl` function now steals the filename reference, which is unexpected (and confusing). 3. The dream Ideally `foo_impl` would have a signature like: static PyObject * foo_impl(PyObject *self, char *filename); And `foo` would be automatically generated as: static PyObject * foo(PyObject *self, PyObject *args, PyObject *kwargs) { PyObject *_ret; PyObject *filename; static char *_keywords[] = {filename, NULL}; if (!PyArg_ParseTupleAndKeywords(args, kwargs, O:foo, _keywords, PyUnicode_FSConverter, filename)) return NULL; _ret = foo_impl(self, PyBytes_AsString(filename)); Py_DECREF(filename); return _ret; } It's not clear to me how one would extend the clinic syntax to support this. In particular, clinic would need to know: - The type of the intermediate object (i.e., PyObject * or PyBytesObject *). - The converter to use in PyArg_ParseTupleAndKeywords (i.e., PyUnicode_FSConverter) - The impl type (i.e, char *) - How to convert the intermediate object to the impl type (i.e., PyBytes_AsString(filename)). - How to cleanup in the end (i.e., Py_DECREF(filename)). This seems like too much data to encode in the clinic syntax. 4. Extend clinic to add a cleanup flag which would produce code like: /*[clinic] foo PyBytesObject *filename converter=PyUnicode_FSConverter cleanup=Py_DECREF [clinic]*/ ... static PyObject * foo(PyObject *self, PyObject *args, PyObject *kwargs) { PyObject *_ret; PyBytesObject *filename; static char *_keywords[] = {filename, NULL}; if (!PyArg_ParseTupleAndKeywords(args, kwargs, O:foo, _keywords, PyUnicode_FSConverter, filename)) return NULL; _ret = foo_impl(self, filename); Py_DECREF(filename); return _ret; } static PyObject * foo_impl(PyObject *self, PyBytesObject *filename) /*[clinic end:...]*/ { char *c_filename = PyBytes_AsString(filename); // ... } This seems like a relatively modest addition, which might also work for other cleanup functions like PyBuffer_Release. Additionally, there are a few other bugs I've noticed: - The s* and z* codes should be of type Py_buffer (and not Py_buffer *) - Since Py_buffer is a relatively large struct, zlib_decompress_impl should probably take a pointer to a Py_buffer. This, however, would likely require extending the clinic syntax. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16612 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16740] Types created with PyType_FromSpec lack a __module__ attribute.
New submission from Bradley Froehle: Types created using PyType_FromSpec do not have a __module__ attribute by default. This caught me off guard. $ python3 Python 3.2.3 (default, Oct 19 2012, 20:10:41) [GCC 4.6.3] on linux2 Type help, copyright, credits or license for more information. import xxlimited xxlimited.Null.__module__ Traceback (most recent call last): File stdin, line 1, in module AttributeError: __module__ Do we expect module authors to set the __module__ attribute immediately after calling PyType_FromSpec? To refresh your memory, non-heap types determine the module/name combo according to something like:: try: __module__, __name__ = tp_name.rsplit('.', 1) except ValueError: __module__, __name__ = 'builtins', tp_name whereas heap types use something like:: __name__ = tp_name @property def __module__(self): return self.__dict__['__module__'] I think this is unnecessarily confusing, and, as far as I know, not documented anywhere. I see from reading the commit logs that it was felt that by allowing users to set __name__ (and therefore tp_name), it could have an unintended impact on the value __module__. If so, why didn't we just disallow setting __name__? There are likely some unintended impacts of this decision, for example weird error message in other library functions: import inspect inspect.getfile(xxlimited.Null) Traceback (most recent call last): File stdin, line 1, in module File /usr/lib/python3.2/inspect.py, line 415, in getfile object = sys.modules.get(object.__module__) AttributeError: __module__ Is there anything we can do here? -- components: Interpreter Core messages: 177860 nosy: bfroehle priority: normal severity: normal status: open title: Types created with PyType_FromSpec lack a __module__ attribute. type: behavior versions: Python 3.2, Python 3.3, Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16740 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16740] Types created with PyType_FromSpec lack a __module__ attribute.
Bradley Froehle added the comment: For example, in PyType_FromSpec can we split the provided name into a module component and name component, and create the '__module__' entry in the dict? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16740 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16740] Types created with PyType_FromSpec lack a __module__ attribute.
Bradley Froehle added the comment: I see this has already been fixed in Python 3.3. My apologies. -- resolution: - invalid status: open - closed versions: -Python 3.3, Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16740 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16690] Reference leak with custom tp_dealloc in PyType_FromSpec
Changes by Bradley Froehle brad.froe...@gmail.com: -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16690 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16690] Reference leak with custom tp_dealloc in PyType_FromSpec
Bradley Froehle added the comment: The attached file `heaptype_refcnt_testcases.py` runs through several test cases (ssl.SSLError, a subclass of ssl.SSLError, and xxlimited.Xxo) seeing if references are leaked in each instance. Unfortunately `xxlimited.Xxo` isn't set to be a base type and I don't know of any other types in the default install which use PyType_FromSpec with a custom tp_dealloc. -- Added file: http://bugs.python.org/file28372/heaptype_refcnt_testcases.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16690 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16690] Reference leak with custom tp_dealloc in PyType_FromSpec
New submission from Bradley Froehle: There is a reference leak when using PyType_FromSpec with custom tp_dealloc. This was first noted in issue #15142, where a fix was given which only applies to types which do not override tp_dealloc. For example, the xxlimited.Xxo type suffers from this: Python 3.3.0 (default, Oct 26 2012, 11:06:17) [GCC 4.6.3] on linux Type help, copyright, credits or license for more information. import xxlimited import sys Xxo = type(xxlimited.new()) e = Xxo() sys.getrefcount(Xxo) 7 e = Xxo() sys.getrefcount(Xxo) 8 e = Xxo() sys.getrefcount(Xxo) 9 -- components: Interpreter Core messages: 177527 nosy: bfroehle priority: normal severity: normal status: open title: Reference leak with custom tp_dealloc in PyType_FromSpec type: resource usage versions: Python 3.2, Python 3.3, Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16690 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16690] Reference leak with custom tp_dealloc in PyType_FromSpec
Bradley Froehle added the comment: I see this issue came up in the course of #15653 as well. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16690 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16612] Integrate Argument Clinic specialized preprocessor into CPython trunk
Changes by Bradley Froehle brad.froe...@gmail.com: -- nosy: +bfroehle ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16612 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10712] 2to3 fixer for deprecated unittest method names
Bradley Froehle added the comment: Bikeshedding, but the fixer name of 'asserts' bugs me. I'd suggest 'unittest' or 'unittest_asserts'. -- nosy: +bfroehle ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10712 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5141] C API for appending to arrays
Changes by Bradley Froehle brad.froe...@gmail.com: -- nosy: +bfroehle ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5141 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16378] venv.EnvBuilder docstring inconsistencies
New submission from Bradley Froehle: Consider the docstring for venv.EnvBuilder:: By default, the builder makes the system (global) site-packages dir available to the created environment. By default, the creation process uses symlinks wherever possible. It seems to suggest that the `system_site_packages` and `symlink` parameters both default to True, even though the default values for each are False. The docstring for venv.create is similarly misleading. -- assignee: docs@python components: Documentation messages: 174388 nosy: bfroehle, docs@python priority: normal severity: normal status: open title: venv.EnvBuilder docstring inconsistencies versions: Python 3.3, Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16378 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14803] Add feature to allow code execution prior to __main__ invocation
Changes by Bradley Froehle brad.froe...@gmail.com: -- nosy: +bfroehle ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14803 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16268] dir(closure) does not find __dir__
Bradley Froehle added the comment: Your patch looks good to me, and I can verify that it properly tests for the issue in Python 3.3. On a related note, I've found that PyType_Ready(...) isn't called for a few other core PyTypeObjects... see #16369. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16268 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16369] Global PyTypeObjects not initialized with PyType_Ready(...)
New submission from Bradley Froehle: In Python 3.3.0, several global `PyTypeObject`s are not initialized using PyType_Ready(...). The list of uninitialized type objects:: PyCapsule_Type PyLongRangeIter_Type PyFieldNameIter_Type PyFormatterIter_Type PySTEntry_Type PyCell_Type PyInstanceMethod_Type PyClassMethodDescr_Type PyMethodDescr_Type PyCallIter_Type PySeqIter_Type PyDictDummy_Type Each of these can be verified using gdb:: $ gdb -quiet python3.3 Reading symbols from /opt/python/3.3.0/bin/python3.3...done. (gdb) break Py_Finalize Breakpoint 1 at 0x4a40a0: file Python/pythonrun.c, line 478. (gdb) run -c pass Starting program: /opt/python/3.3.0/bin/python3.3 -c pass [Thread debugging using libthread_db enabled] Using host libthread_db library /lib/x86_64-linux-gnu/libthread_db.so.1. Breakpoint 1, Py_Finalize () at Python/pythonrun.c:478 478 if (!initialized) (gdb) print PyCell_Type-tp_mro $1 = (PyObject *) 0x0 (The list of uninitialized types was built by searching info variables in gdb to produce a list of PyTypeObjects. Those with name-tp_mro != NULL were treated as properly initialized. The remainder were further inspected by hand to throw out additional false positives, like those from modules which had been statically compiled into the Python binary). I'm not sure if any of these actually need fixing, but for PyCell_Type this issue already resulted in one obscure bug: #16268. -- components: Interpreter Core messages: 174255 nosy: bfroehle priority: normal severity: normal status: open title: Global PyTypeObjects not initialized with PyType_Ready(...) type: behavior versions: Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16369 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16268] dir(closure) does not find __dir__
Bradley Froehle added the comment: This rather obscure bug seems to be caused by a failure to properly initialize PyCell_Type. Running with GDB, we see that _PyType_Lookup(class 'cell', __dir__) fails in: /* Look in tp_dict of types in MRO */ mro = type-tp_mro; /* If mro is NULL, the type is either not yet initialized by PyType_Ready(), or already cleared by type_clear(). Either way the safest thing to do is to return NULL. */ if (mro == NULL) return NULL; Since: (gdb) print PyCell_Type-tp_mro $9 = (PyObject *) 0x0 Searching the code base shows that we never call PyType_Ready(PyCell_Type). A patch is attached. -- keywords: +patch nosy: +bfroehle Added file: http://bugs.python.org/file27783/init_cell_type.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16268 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16268] dir(closure) does not find __dir__
Bradley Froehle added the comment: Note that we fail to initialize PyCell_Type in all versions of Python, even if there aren't any visible ramifications in earlier versions. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16268 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7897] Support parametrized tests in unittest
Changes by Bradley Froehle brad.froe...@gmail.com: -- nosy: +bfroehle ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7897 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14768] os.path.expanduser('~/a') doesn't works correctly when HOME is '/'
New submission from Bradley Froehle brad.froe...@gmail.com: When $HOME=/, os.path.expanduser('~/a') returns '//a' rather than '/a'. This regression was created by a partially incorrect resolution to issue #5471, and affects versions 2.7 and 3.2 (at least). $ HOME=/ python2.7 -c import os; print os.path.expanduser('~/a') //a $ HOME=/ python3.2 -c import os; print(os.path.expanduser('~/a')) //a In each case the expected result should be '/a'. -- components: Library (Lib) messages: 160321 nosy: bfroehle priority: normal severity: normal status: open title: os.path.expanduser('~/a') doesn't works correctly when HOME is '/' versions: Python 2.7, Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14768 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14768] os.path.expanduser('~/a') doesn't works correctly when HOME is '/'
Bradley Froehle brad.froe...@gmail.com added the comment: Patch (for version 2.7) attached. -- keywords: +patch Added file: http://bugs.python.org/file25514/issue_14768.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14768 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com