[issue35160] PyObjects initialized with PyObject_New have uninitialized pointers to set to 0x1

[Serafeim Mellos]

Serafeim Mellos  added the comment:

Ok, I understand your point about PyObject_New not initializing custom fields 
(although it looks like something is initializing the memory since it's not 
random, it's either 0x0 or 0x1).

However, if that's the expected behavior, it's not very clear why there's a 
difference between the default values when invoking the same commands in 
different ways or between different python versions, eg:

Python3.8
=

Python 3.8.0a0 (heads/master-dirty:f98c1623ec, Nov  4 2018, 17:16:57) 
[GCC 5.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import custom; custom.Custom()
weird pointer has value of: 0x0

>>> import custom;
>>> custom.Custom()
weird pointer has value of: 0x1
deallocating weird pointer

>>> 
deallocating weird pointer
zsh: segmentation fault  ../cpython/python

Python 3.6
==

Python 3.6.5 (default, Nov  4 2018, 18:20:36) 
[GCC 5.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import custom; custom.Custom()
weird pointer has value of: 0x0
deallocating weird pointer

>>> import custom  
>>> custom.Custom()
weird pointer has value of: 0x0
deallocating weird pointer



Is there any discussions or documentation I can refer to, in order to better 
understand this behavior or why it was changed since python 3.7? 

Thanks!

--

___
Python tracker 
<https://bugs.python.org/issue35160>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35160] PyObjects initialized with PyObject_New have uninitialized pointers to set to 0x1

[Benjamin Peterson]

Benjamin Peterson  added the comment:

PyObject_New is a low-level allocation function. It doesn't initialize anything 
but the type pointer and ref count. If you want a field to be NULL, set it to 
NULL.

--
nosy: +benjamin.peterson
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue35160>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35160] PyObjects initialized with PyObject_New have uninitialized pointers to set to 0x1

[Serafeim Mellos]

Serafeim Mellos  added the comment:

Actually, I'm not so sure it's related to memory management after all judging 
by this (but I don't really know how the compilation part is supposed to work 
:)):


(gdb) watch *0x77ec0680
Hardware watchpoint 1: *0x77ec0680
(gdb) b Custom_dealloc
Breakpoint 2 at 0x7fffef9c4c50: file custom.c, line 16.
(gdb) run
[...]
Hardware watchpoint 1: *0x77ec0680

Old value = 0
New value = 1
dfs (c=c@entry=0x7fffd440, b=b@entry=0x77ec0660, 
a=a@entry=0x7fffd3b0, end=2) at Python/compile.c:4974
4974a->a_postorder[--j] = b;
(gdb) bt
#0  dfs (c=c@entry=0x7fffd440, b=b@entry=0x77ec0660, 
a=a@entry=0x7fffd3b0, end=2) at Python/compile.c:4974
#1  0x004e9e86 in assemble (c=c@entry=0x7fffd440, 
addNone=) at Python/compile.c:5530
#2  0x004f45c6 in compiler_mod (mod=0x9a86a8, c=0x7fffd440) at 
Python/compile.c:1641
#3  PyAST_CompileObject (mod=mod@entry=0x9a86a8, 
filename=filename@entry=0x77e510d8, flags=flags@entry=0x7fffd5e0, 
optimize=optimize@entry=-1, arena=arena@entry=0x77fb81e0)
at Python/compile.c:345
#4  0x00520025 in run_mod (arena=0x77fb81e0, flags=0x7fffd5e0, 
locals=0x77f38d80, globals=0x77f38d80, filename=0x77e510d8, 
mod=0x9a86a8)
at Python/pythonrun.c:1028
#5  PyRun_InteractiveOneObjectEx (fp=fp@entry=0x7749b8a0 <_IO_2_1_stdin_>, 
filename=filename@entry=0x77e510d8, flags=flags@entry=0x7fffd5e0) at 
Python/pythonrun.c:256
#6  0x00520326 in PyRun_InteractiveLoopFlags 
(fp=fp@entry=0x7749b8a0 <_IO_2_1_stdin_>, 
filename_str=filename_str@entry=0x5f1345 "", 
flags=flags@entry=0x7fffd5e0)
at Python/pythonrun.c:120
#7  0x0052049c in PyRun_AnyFileExFlags (fp=0x7749b8a0 
<_IO_2_1_stdin_>, filename=0x5f1345 "", closeit=0, flags=0x7fffd5e0) 
at Python/pythonrun.c:78
#8  0x0042e0be in pymain_run_stdin (pymain=0x7fffd6f0, 
pymain=0x7fffd6f0, cf=0x7fffd5e0, config=0x922ea8) at 
Modules/main.c:1185
#9  pymain_run_python (interp=0x922e20, pymain=0x7fffd6f0) at 
Modules/main.c:1613
#10 pymain_main (pymain=0x7fffd6f0) at Modules/main.c:1758
#11 0x0042e309 in _Py_UnixMain (argc=, argv=) at Modules/main.c:1795
#12 0x7711b4db in __libc_start_main () from /lib64/libc.so.6
#13 0x0042ae6a in _start ()
(gdb) frame
#0  dfs (c=c@entry=0x7fffd440, b=b@entry=0x77ec0660, 
a=a@entry=0x7fffd3b0, end=2) at Python/compile.c:4974
4974a->a_postorder[--j] = b;
(gdb) c
Continuing.
weird pointer has value of: 0x1
>>> 

Breakpoint 2, Custom_dealloc (self=0x77ec0660) at custom.c:16
16  {
(gdb) print(self->test_field)
$1 = (PyObject *) 0x1
(gdb) print(>test_field)
$2 = (PyObject **) 0x77ec0680

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35160] PyObjects initialized with PyObject_New have uninitialized pointers to set to 0x1

[Serafeim Mellos]

Serafeim Mellos  added the comment:

Backtrace from segfault is below but I'm not very familiar with python's memory 
internals in order to figure out where is the value initialized at the first 
place 

(gdb) bt
#0  _Py_XDECREF (op=0x1) at 
/home/fim/.pyenv/versions/3.8-dev/include/python3.8m/object.h:896
#1  Custom_dealloc (self=0x77ec0660) at custom.c:20
#2  0x004685f2 in _Py_Dealloc_inline (op=) at 
./Include/object.h:806
#3  _Py_DECREF (filename=0x5f7460 "Objects/dictobject.c", lineno=1508, 
op=) at ./Include/object.h:833
#4  delitem_common (old_value=, ix=, 
hash=-2939463505650448616, mp=0x77f38d80) at Objects/dictobject.c:1508
#5  _PyDict_DelItem_KnownHash (hash=-2939463505650448616, key=0x77f39b90, 
op=0x77f38d80) at Objects/dictobject.c:1560
#6  PyDict_DelItem (op=0x77f38d80, key=0x77f39b90) at 
Objects/dictobject.c:1526
#7  0x00468aa5 in dict_ass_sub (mp=0x77f38d80, v=0x77f39b90, 
w=) at Objects/dictobject.c:2058
#8  0x004262e3 in _PyEval_EvalFrameDefault (f=, 
throwflag=) at Python/ceval.c:2044
#9  0x004e3ac0 in PyEval_EvalFrameEx (throwflag=0, f=0x77f5a970) at 
Python/ceval.c:531
#10 _PyEval_EvalCodeWithName (_co=_co@entry=0x77e97e40, 
globals=globals@entry=0x77f38d80, locals=locals@entry=0x77f38d80, 
args=args@entry=0x0, argcount=argcount@entry=0, 
kwnames=kwnames@entry=0x0, kwargs=0x0, kwcount=0, kwstep=2, defs=0x0, 
defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at 
Python/ceval.c:3946
#11 0x004e3bb3 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, 
defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0, args=0x0, 
locals=locals@entry=0x77f38d80, 
globals=globals@entry=0x77f38d80, _co=_co@entry=0x77e97e40) at 
Python/ceval.c:3975
#12 PyEval_EvalCode (co=co@entry=0x77e97e40, 
globals=globals@entry=0x77f38d80, locals=locals@entry=0x77f38d80) at 
Python/ceval.c:508
#13 0x0052009f in run_mod (arena=0x77fb81e0, flags=0x7fffd5e0, 
locals=0x77f38d80, globals=0x77f38d80, filename=0x77e510d8, 
mod=0x9a66c0)
at Python/pythonrun.c:1031
#14 PyRun_InteractiveOneObjectEx (fp=fp@entry=0x7749b8a0 <_IO_2_1_stdin_>, 
filename=filename@entry=0x77e510d8, flags=flags@entry=0x7fffd5e0) at 
Python/pythonrun.c:256
#15 0x00520386 in PyRun_InteractiveLoopFlags 
(fp=fp@entry=0x7749b8a0 <_IO_2_1_stdin_>, 
filename_str=filename_str@entry=0x5f13a5 "", 
flags=flags@entry=0x7fffd5e0)
at Python/pythonrun.c:120
#16 0x005204fc in PyRun_AnyFileExFlags (fp=0x7749b8a0 
<_IO_2_1_stdin_>, filename=0x5f13a5 "", closeit=0, flags=0x7fffd5e0) 
at Python/pythonrun.c:78
#17 0x0042e0fe in pymain_run_stdin (pymain=0x7fffd6f0, 
pymain=0x7fffd6f0, cf=0x7fffd5e0, config=0x922ea8) at 
Modules/main.c:1185
#18 pymain_run_python (interp=0x922e20, pymain=0x7fffd6f0) at 
Modules/main.c:1613
#19 pymain_main (pymain=0x7fffd6f0) at Modules/main.c:1758
#20 0x0042e349 in _Py_UnixMain (argc=, argv=) at Modules/main.c:1795
#21 0x7711b4db in __libc_start_main () from /lib64/libc.so.6
#22 0x0042aeaa in _start ()

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35160] PyObjects initialized with PyObject_New have uninitialized pointers to set to 0x1

[Serafeim Mellos]

New submission from Serafeim Mellos :

I looked but I couldn't find any mention of this either in the bug tracker or 
the docs so I assume this is not a desired behavior or a known proble but feel 
free to correct me if I'm wrong.

I have noticed a weird behavior in Python3.7 onwards with the way 
PyObject_New() works. I'm not sure what is the exact cause for the issue but 
even though the documentation mentions that it should initialize all fields 
described in the associated PyTypeObject, I think there's a problem that 
somehow causes uninitialized pointers to get initialized with strange values 
that break other functionality (eg Py_XDECREF).

In more detail, it seems like uninitialized pointers in PyObjects get 
initialized to 0x1 which can lead to SEGFAULTs when calling Py_XDECREF() on 
them since they are no longer valid pointers.

I have taken the example extension from the python docs 
(https://docs.python.org/3/extending/newtypes_tutorial.html) and modified 
lightly in order to surface the issue. You can find the sample extension in my 
github: https://github.com/fim/python_pyobjectnew_example

I have checked versions 3.5, 3,6 which are fine but 3.7.0, 3.7.1 and 3.8 seem 
to reproduce this issue.

You can see a simple transcript of how the issue manifests below:

$ python setup.py build
[...]
$ pip install . 
[...]
$ python -c 'import custom; custom.Custom()'
weird pointer has value of: 0x8ec480
deallocating weird pointer
$ python
Python 3.7.1 (default, Nov  3 2018, 09:33:27) 
[GCC 5.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import custom, gc
>>> custom.Custom()
weird pointer has value of: 0x1

>>> gc.collect()
deallocating weird pointer
zsh: segmentation fault  python

--
components: Extension Modules
messages: 329224
nosy: fim
priority: normal
severity: normal
status: open
title: PyObjects initialized with PyObject_New have uninitialized pointers to 
set to 0x1
type: crash
versions: Python 3.7, Python 3.8

___
Python tracker 
<https://bugs.python.org/issue35160>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: PyObject_New

[Gerhard Häring]

Jeremy Moles wrote:
 [...] What I'm trying
 now is the following:
 
   PyObject* obj = _PyObject_New(PyType_MyType);
   obj = PyObject_Init(obj, PyType_MyType);
   
   ...
 
   return obj;
 
 When obj gets back to the interpreter, Python sees it (or rather, it's
 __repr__) in accordance with what it should be. However, any attempt
 to USE the object results in a segfault. I feel my problem is simply
 that I'm not allocating obj correctly in the C++ function.
 
 If anyone has any advice, I would really appreciate it.

When I looked for info on this, I didn't find any good documentation, 
either.

I'm currently using code like the following in pysqlite 
(http://initd.org/tracker/pysqlite/browser/pysqlite/trunk/src/connection.c?rev=154):

126 if (factory == NULL) {
127 factory = (PyObject*)CursorType;
128 }
129 
130 cursor = PyObject_CallFunction(factory, O, self);

So, basically I'm just calling the type like any other callable object. 
I don't remember if there was a reason why I didn't use 
PyObject_CallObject instead, at the time ...

In any case, this approach worked fine for me.

-- Gerhard

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: PyObject_New

[Martin v. Löwis]

Jeremy Moles wrote:
   PyObject* obj = _PyObject_New(PyType_MyType);
   obj = PyObject_Init(obj, PyType_MyType);
   
   ...
 
   return obj;

The call to PyObject_Init is redundant: _PyObject_New
is malloc+init. However, this shouldn't cause any crashes (except in the
debug build). PyObject_Init is documented as

Initialize a newly-allocated object op with its type and initial 
reference. Returns the initialized object. If type  indicates that the 
object participates in the cyclic garbage detector, it is added to the 
detector's set of observed objects. Other fields of the object are not 
affected.

[I don't know where the mentioning of GC comes from - it appears to be
  incorrect]

 When obj gets back to the interpreter, Python sees it (or rather, it's
 __repr__) in accordance with what it should be. However, any attempt
 to USE the object results in a segfault. I feel my problem is simply
 that I'm not allocating obj correctly in the C++ function.

It doesn't crash because of the allocation - this code is correct.
However, it is also incomplete: none of the state of the new object
gets initialized in the fragment you are showing. So it likely crashes
because the members of the object are stray pointers or some such,
and accessing them causes a crash.

Regards,
Martin
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: PyObject_New

[Jeremy Moles]

I just noticed this response right as I sent my other message. For some
reason my news reader didn't thread it, so it was all by itself...
Please disregard the rant concerning creation of objects in C. :)

/me hugs Martin
/me ducks and hides!

On Fri, 2005-10-07 at 09:57 +0200, Martin v. Löwis wrote:
 Jeremy Moles wrote:
  PyObject* obj = _PyObject_New(PyType_MyType);
  obj = PyObject_Init(obj, PyType_MyType);
  
  ...
  
  return obj;
 
 The call to PyObject_Init is redundant: _PyObject_New
 is malloc+init. However, this shouldn't cause any crashes (except in the
 debug build). PyObject_Init is documented as
 
 Initialize a newly-allocated object op with its type and initial 
 reference. Returns the initialized object. If type  indicates that the 
 object participates in the cyclic garbage detector, it is added to the 
 detector's set of observed objects. Other fields of the object are not 
 affected.
 
 [I don't know where the mentioning of GC comes from - it appears to be
   incorrect]
 
  When obj gets back to the interpreter, Python sees it (or rather, it's
  __repr__) in accordance with what it should be. However, any attempt
  to USE the object results in a segfault. I feel my problem is simply
  that I'm not allocating obj correctly in the C++ function.
 
 It doesn't crash because of the allocation - this code is correct.
 However, it is also incomplete: none of the state of the new object
 gets initialized in the fragment you are showing. So it likely crashes
 because the members of the object are stray pointers or some such,
 and accessing them causes a crash.
 
 Regards,
 Martin

-- 
http://mail.python.org/mailman/listinfo/python-list

PyObject_New

[Jeremy Moles]

Hey guys, sorry to ask another question of this nature, but I can't find
the answer or a single example of it anywhere. I'm sure it's been asked
before, but my google-fu isn't strong enough to find anything.

I have the following:

struct MyType {
PyObject_HEAD
...
};

PyTypeObject PyType_MyType = { ... }

Now, elsewhere in the code I want to create an instance of this custom
type in C/C++ (rather than in the Python interpreter, where this all
seems to happen magically) and return it from a method. What I'm trying
now is the following:

PyObject* obj = _PyObject_New(PyType_MyType);
obj = PyObject_Init(obj, PyType_MyType);

...

return obj;

When obj gets back to the interpreter, Python sees it (or rather, it's
__repr__) in accordance with what it should be. However, any attempt
to USE the object results in a segfault. I feel my problem is simply
that I'm not allocating obj correctly in the C++ function.

If anyone has any advice, I would really appreciate it.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: PyObject_New not running tp_new for iterators?

[Gregory Bond]

I wrote:
[snip]
 What am I missing?
The fundamental problem is that this:
   if (!(ro = PyObject_New(MyIter, MyIterType)))
  return NULL;
is really only a malloc() - it doesn't call the tp_new function at all. 
 This is not really clear in the 2.3 version of the C API document that 
I was consulting - the 2.4 version is much clearer.

I have no idea how to do in C all the things that calling a class 
constructor in Python does.  Am I supposed to be calling tp_new  
tp_init directly?

In the end, this particular class really only needs to be created from a 
factory function, so I removed the tp_new member altogether and just 
wrote a C function to initialise the object, and called that from the C 
factory function.

[The above is not particular to the fact that my class is an iterator - 
all C classes have the same behaviour.  My test of a plain-old class 
probably looked like it worked due to lucky memory patterns.]

--
http://mail.python.org/mailman/listinfo/python-list