[issue29802] A possible null-pointer dereference in struct.s_unpack_internal()

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset 73c4708630f99b94c35476529748629fff1fc63e by Serhiy Storchaka in 
branch 'master':
Fix bytes warnings in test_struct (added in bpo-29802). (#4068)
https://github.com/python/cpython/commit/73c4708630f99b94c35476529748629fff1fc63e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29802] A possible null-pointer dereference in struct.s_unpack_internal()

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +4039

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29802] A possible null-pointer dereference in struct.s_unpack_internal()

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29802] A possible null-pointer dereference in struct.s_unpack_internal()

[Serhiy Storchaka]

Serhiy Storchaka added the comment:


New changeset 7bfd740e3d484e6fdf3f5c2d4640450957f9d89c by Serhiy Storchaka in 
branch 'master':
Remove unneeded Misc/NEWS entry for bpo-29802. (#1251)
https://github.com/python/cpython/commit/7bfd740e3d484e6fdf3f5c2d4640450957f9d89c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29802] A possible null-pointer dereference in struct.s_unpack_internal()

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
pull_requests: +1366

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29802] A possible null-pointer dereference in struct.s_unpack_internal()

[Serhiy Storchaka]

Serhiy Storchaka added the comment:


New changeset 17db4b99b4d300a9b024ba0efdaa46d05d4f4cd3 by Serhiy Storchaka in 
branch '3.5':
[3.5] bpo-29802: Fix the cleaning up issue in PyUnicode_FSDecoder(). (GH-1217) 
(#1219)
https://github.com/python/cpython/commit/17db4b99b4d300a9b024ba0efdaa46d05d4f4cd3


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29802] A possible null-pointer dereference in struct.s_unpack_internal()

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
pull_requests: +1342

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29802] A possible null-pointer dereference in struct.s_unpack_internal()

[Serhiy Storchaka]

Serhiy Storchaka added the comment:


New changeset 7a113a0cbf545588d61286fcc0e89141cf211735 by Serhiy Storchaka in 
branch '3.6':
bpo-29802: Fix the cleaning up issue in PyUnicode_FSDecoder(). (#1217)
https://github.com/python/cpython/commit/7a113a0cbf545588d61286fcc0e89141cf211735


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29802] A possible null-pointer dereference in struct.s_unpack_internal()

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
pull_requests: +1340

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29802] A possible null-pointer dereference in struct.s_unpack_internal()

[Serhiy Storchaka]

Serhiy Storchaka added the comment:


New changeset 40db90c1ce1a59d5f5f2894bb0ce3211bf27 by Serhiy Storchaka in 
branch 'master':
bpo-29802: Fix reference counting in module-level struct functions (#1213)
https://github.com/python/cpython/commit/40db90c1ce1a59d5f5f2894bb0ce3211bf27


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29802] A possible null-pointer dereference in struct.s_unpack_internal()

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Thank you for your report Artem.

But Modules/clinic/_struct.c.h is generated file, it shouldn't be manually 
edited. And clearing the cache doesn't solves the bug.

If _PyArg_ParseStack() failed it calls cache_struct_converter() for clearing 
s_object and jumps to the end of the function where s_object is decrefed second 
time. Yet one reference is left in the cache, but this is a hanging reference 
to deallocated object. Next call of struct.unpack() can retrieve that hanging 
reference and use it.

PR 1213 properly fixes this issue. It also fixes similar bug 
PyUnicode_FSDecoder(), but the latter is hardly reproducible since in the 
stdlib PyUnicode_FSDecoder() is used mostly for the last argument of a function.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29802] A possible null-pointer dereference in struct.s_unpack_internal()

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
pull_requests: +1336

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29802] A possible null-pointer dereference in struct.s_unpack_internal()

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
assignee:  -> serhiy.storchaka
components: +Extension Modules
nosy: +serhiy.storchaka
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29802] A possible null-pointer dereference in struct.s_unpack_internal()

[Artem Smotrakov]

Changes by Artem Smotrakov :


--
keywords: +patch
Added file: http://bugs.python.org/file46723/_struct_cache.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29802] A possible null-pointer dereference in struct.s_unpack_internal()

[Artem Smotrakov]

New submission from Artem Smotrakov:

Attached struct_unpack_crash.py results to a null-pointer dereference in 
s_unpack_internal() function of _struct module:

ASAN:SIGSEGV
=
==20245==ERROR: AddressSanitizer: SEGV on unknown address 0x (pc 
0x7facd2cea83a bp 0x sp 0x7ffd0250f860 T0)
   #0 0x7facd2cea839 in s_unpack_internal 
/home/artem/projects/python/src/cpython-asan/Modules/_struct.c:1515
   #1 0x7facd2ceab69 in Struct_unpack_impl 
/home/artem/projects/python/src/cpython-asan/Modules/_struct.c:1570
   #2 0x7facd2ceab69 in unpack_impl 
/home/artem/projects/python/src/cpython-asan/Modules/_struct.c:2192
   #3 0x7facd2ceab69 in unpack 
/home/artem/projects/python/src/cpython-asan/Modules/clinic/_struct.c.h:215
   #4 0x474397 in _PyMethodDef_RawFastCallKeywords Objects/call.c:618
   #5 0x474397 in _PyCFunction_FastCallKeywords Objects/call.c:690
   #6 0x42685f in call_function Python/ceval.c:4817
   #7 0x42685f in _PyEval_EvalFrameDefault Python/ceval.c:3298
   #8 0x54b164 in PyEval_EvalFrameEx Python/ceval.c:663
   #9 0x54b164 in _PyEval_EvalCodeWithName Python/ceval.c:4173
   #10 0x54b252 in PyEval_EvalCodeEx Python/ceval.c:4200
   #11 0x54b252 in PyEval_EvalCode Python/ceval.c:640
   #12 0x431e0e in run_mod Python/pythonrun.c:976
   #13 0x431e0e in PyRun_FileExFlags Python/pythonrun.c:929
   #14 0x43203b in PyRun_SimpleFileExFlags Python/pythonrun.c:392
   #15 0x446354 in run_file Modules/main.c:338
   #16 0x446354 in Py_Main Modules/main.c:809
   #17 0x41df71 in main Programs/python.c:69
   #18 0x7facd58ac82f in __libc_start_main 
(/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
   #19 0x428728 in _start 
(/home/artem/projects/python/build/cpython-asan/bin/python3.7+0x428728)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV 
/home/artem/projects/python/src/cpython-asan/Modules/_struct.c:1515 
s_unpack_internal
==20245==ABORTING


Looks like _struct implementation assumes that PyStructObject->s_codes cannot 
be null,
but it may happen if a bytearray was passed to unpack().
PyStructObject->s_codes becomes null in a couple of places in _struct.c, but 
that's not the case.
unpack() calls _PyArg_ParseStack() with cache_struct_converter() which 
maintains a cache.
Even if unpack() was called incorrectly with a string as second parameter (see 
below), this value is going to be cached anyway.
Next time, if the same format string is used, the value is going to be 
retrieved from the cache.
But PyStructObject->s_codes is still not null in cache_struct_converter() 
function.
If you watch "s_object" under gdb, you can see that "s_codes" becomes null here:

PyBuffer_FillInfo (view=0x7fffd700, obj=obj@entry=0x77e50730,
buf=0x8df478 <_PyByteArray_empty_string>, len=0, readonly=readonly@entry=0,
flags=0) at Objects/abstract.c:647
647 view->format = NULL;
(gdb) bt
#0  PyBuffer_FillInfo (view=0x7fffd700, obj=obj@entry=0x77e50730,
buf=0x8df478 <_PyByteArray_empty_string>, len=0, readonly=readonly@entry=0,
flags=0) at Objects/abstract.c:647
#1  0x0046020c in bytearray_getbuffer (obj=0x77e50730,
view=, flags=) at Objects/bytearrayobject.c:72
#2  0x00560b0a in getbuffer (errmsg=,
view=0x7fffd700, arg=0x77e50730) at Python/getargs.c:1380
#3  convertsimple (freelist=0x7fffd3b0, bufsize=256,
msgbuf=0x7fffd4c0 "must be bytes-like object, not str", flags=2,
p_va=0x0, p_format=, arg=0x77e50730)
at Python/getargs.c:938
#4  convertitem (arg=0x77e50730, p_format=p_format@entry=0x7fffd3a8,
p_va=p_va@entry=0x7fffd610, flags=flags@entry=2,
levels=levels@entry=0x7fffd3c0,
msgbuf=msgbuf@entry=0x7fffd4c0 "must be bytes-like object, not str",
bufsize=256, freelist=0x7fffd3b0) at Python/getargs.c:596
#5  0x00561d6f in vgetargs1_impl (compat_args=compat_args@entry=0x0,
stack=stack@entry=0x6164b520, nargs=2,
format=format@entry=0x735d5c88 "O*:unpack",
p_va=p_va@entry=0x7fffd610, flags=flags@entry=2) at Python/getargs.c:388
#6  0x005639b0 in _PyArg_ParseStack_SizeT (
args=args@entry=0x6164b520, nargs=,
format=format@entry=0x735d5c88 "O*:unpack") at Python/getargs.c:163
#7  0x735d2df8 in unpack (module=module@entry=0x77e523b8,
args=args@entry=0x6164b520, nargs=,
kwnames=kwnames@entry=0x0)
at 
/home/artem/projects/python/src/cpython-asan/Modules/clinic/_struct.c.h:207
#8  0x00474398 in _PyMethodDef_RawFastCallKeywords (kwnames=0x0,
nargs=140737352377272, args=0x6164b520, self=0x77e523b8,
method=0x737d94e0 ) at Objects/call.c:618
#9  _PyCFunction_FastCallKeywords (func=func@entry=0x77e53828,
args=args@entry=0x6164b520, nargs=nargs@entry=2,
kwnames=kwnames@entry=0x0) at Objects/call.c:690
#10 0x00426860 in call_function (kwnames=0x0,