[Tim Peters]
>> As I said before, I don't think we need to support this any more.
>> More, I think we should not -- the support code is excruciatingly
>> subtle, it wasted plenty of your time trying to keep it working, and
>> if we keep it in it's going to continue to waste time over the coming
>> years (for example, in the short term, it will waste my time reviewing
>> it).
[Evan Jones]
> I do not have nearly enough experience in the Python world to evaluate
> this decision. I've only been programming in Python for about two years
> now, and as I am sure you are aware, this is my first patch that I have
> submitted to Python. I don't really know my way around the Python
> internals, beyond writing basic extensions in C. Martin's opinion is
> clearly the opposite of yours.
? This is all I recall Martin saying about this:
http://mail.python.org/pipermail/python-dev/2005-January/051265.html
I'm not certain it is acceptable to make this assumption. Why is it
not possible to use the same approach that was previously used (i.e.
leak the arenas array)?
Do you have something else in mind? I'll talk with Martin about it if
he still wants to. Martin, this miserable code must die!
> Basically, the debate seems to boil down to maintaining backwards
> compatibility at the cost of making the code in obmalloc.c harder to
> understand.
The "let it leak to avoid thread problems" cruft is arguably the
single most obscure bit of coding in Python's code base. I created
it, so I get to say that <wink>. Even 100 lines of comments aren't
enough to make it clear, as you've discovered. I've lost track of how
many hours of my life have been pissed away explaining it, and its
consequences (like how come this or that memory-checking program
complains about the memory leak it causes), and the historical madness
that gave rise to it in the beginning. I've had enough of it -- the
only purpose this part ever had was to protect against C code that
wasn't playing by the rules anyway. BFD. There are many ways to
provoke segfaults with C code that breaks the rules, and there's just
not anything that special about this way _except_ that I added
objectionable (even at the time) hacks to preserve this kind of broken
C code until authors had time to fix it. Time's up.
> The particular case that is being supported could definitely be viewed
> as a "bug" in the code that using obmalloc. It also likely is quite rare.
> However, until now it has been supported, so it is hard to judge exactly
> how much code would be affected.
People spent many hours searching for affected code when it first went
in, and only found a few examples then, in obscure extension modules.
It's unlikely usage has grown. The hack was put it in for the dubious
benefit of the few examples that were found then.
> It would definitely be a minor barrier to moving to Python 2.5.
That's in part what python-dev is for. Of course nobody here has code
that will break -- but the majority of high-use extension modules are
maintained by people who read this list, so that's not as empty as it
sounds.
It's also what alpha and beta releases are for. Fear of change isn't
a good enough reason to maintain this code.
> Is there some sort of consensus that is possible on this issue?
Absolutely, provided it matches my view <0.5 wink>. Rip it out, and
if alpha/beta testing suggests that's a disaster, _maybe_ put it back
in.
...
> It turns out that basically the only thing that would change would be
> removing the "volatile" specifiers from two of the global variables,
> plus it would remove about 100 lines of comments. :) The "work" was
> basically just hurting my brain trying to reason about the concurrency
> issues, not changing code.
And the brain of everyone else who ever bumps into this. There's a
high probability that if this code actually doesn't work (can you
produce a formal proof of correctness for it? I can't -- and I
tried), nothing can be done to repair it; and code this outrageously
delicate has a decent chance of being buggy no matter how many people
stare at it (overlooking that you + me isn't that many). You also
mentioned before that removing the "volatile"s may have given a speed
boost, and that's believable. I mentioned above the unending costs in
explanations, and nuisance gripes from memory-integrity tools about
the deliberate leaks. There are many kinds of ongoing costs here, and
no _intended_ benefit anymore (it certainly wasn't my intent to cater
to buggy C code forever).
>> It was never legit to do #a without holding the GIL. It was clear as
>> mud whether it was legit to do #b without holding the GIL. If
>> PyMem_Del (etc) change to expand to "free" in a release build, then #b
>> can remain clear as mud without harming anyone. Nobody should be
>> doing #a anymore. If someone still is, "tough luck -- fix it, you've
>> had years of warning" is easy for me to live with at this stage.
> Hmm... The issue is that case #a may not be an easy problem to
> diagnose:
Many errors in C code are difficult to diagnose. That's life. Mixing
a PyObject call with a PyMem call is obvious now "by eyeball", so if
there is such code still out there, and it blows up, an experienced
eye has a good chance of spotting the error at once.
'
> Some implementations of free() will happily do nothing if
> they are passed a pointer they know nothing about. This would just
> result in a memory leak. Other implementations of free() can output a
> warning or crash in this case, which would make it trivial to locate.
I expect most implementations of free() would end up corrupting memory
state, leading to no symptoms or to disastrous symptoms, from 0 to a
googol cycles after the mistake was made. Errors in using malloc/free
are often nightmares to debug. We're not trying to make coding in C
pleasant here -- which is good, because that's unachievable <wink>.
>> I suppose the other consideration is that already-compiled extension
>> modules on non-Windows(*) systems will, if they're not recompiled,
>> continue to call PyObject_Free everywhere they had a
>> PyMem_Del/DEL/FREE call.
> Is it guaranteed that extension modules will be binary compatible with
> future Python releases? I didn't think this was the case.
Nope, that's not guarantfeed. There's a magic number
(PYTHON_API_VERSION) that changes whenever the Python C API undergoes
an incompatible change, and binary compatibility is guaranteed across
releases if that doesn't change. The then-current value of
PYTHON_API_VERSION gets compiled into extensions, by virtue of the
module-initialization macro their initialization function has to call.
The guts of that function are in the Python core (Py_InitModule4()),
which raises this warning if the passed-in version doesn't match the
current version:
"Python C API version mismatch for module %.100s:\
This Python has API version %d, module %.100s has version %d.";
This is _just_ a warning, though. Perhaps unfortunately for Python's
users, Guido learned long ago that most API mismatches don't actually
matter for his own code <wink>. For example, the C API officially
changed when the signature of PyFrame_New() changed in 2001 -- but
almost no extension modules call that function.
Similarly, if we change PyMem_Del (etc) to map to the system free(),
PYTHON_API_VERSION should be bumped for Python 2.5 -- but many people
will ignore the mismatch warning, and again it will probably make no
difference (if there's code still out there that calls PyMem_DEL (etc)
without holding the GIL, I don't know about it).
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
