Dennis Lee Bieber wrote:
greg.ew...@canterbury.ac.nz declaimed the following
So maybe we need to redesign the hardware.
Remember the iAPX-432?
http://en.wikipedia.org/wiki/Intel_iAPX_432#Garbage_collection
Not quite what I had in mind. That sounds like a conventional
GC algorithm
On 9/8/2010 6:20 PM, Paul Rubin wrote:
What tricks on tricks? Even the fanciest GC's are orders of magnitude
less complicated than any serious database, optimizing compiler, OS
kernel, file system, etc. Real-world software is complicated. Get used
to that fact, and look for ways to manage the
In message 7xeid9gtuq@ruckus.brouhaha.com, Paul Rubin wrote:
Lawrence D'Oliveiro l...@geek-central.gen.new_zealand writes:
That reinforces my point, about how easy it was to check the correctness
of the code. In this case one simple fix, like this ...
would render the code watertight.
Lawrence D'Oliveiro l...@geek-central.gen.new_zealand writes:
In that case, I can similarly discount your attempts to fix up issues with
garbage collectors after they’re pointed out, can I not?
Adapting GC algorithms to improve their performance or better use the
capabilities of new hardware
On Thu, 09 Sep 2010 12:41:20 +1200, Lawrence D'Oliveiro wrote:
Part of the problem is C itself.
And yet, what are these complicated garbage collectors, that you intend
relying on to work correctly with all their layers of tricks upon
tricks, written in? C.
Not necessarily.
Pascal, despite
Lawrence D'Oliveiro l...@geek-central.gen.new_zealand writes:
But you’ll notice that Free Software comes with no such
restrictions. In fact, it is contrary to commonly-accepted Free
Software guidelines to impose any sort of restrictions on areas of use.
Free software comes with an explicit
Paul Rubin wrote:
Now extrapolate that error rate from 30 lines to a program the size of
Firefox (something like 5 MLOC), and you should see how fraught with
danger that style of programming is.
But you don't write 5 MLOC of code using that programming style.
You use it to write a small core
Lawrence D'Oliveiro wrote:
But alone of all of these, garbage collection still remains just as costly
to implement as ever. That should tell you something about how poorly it
matches the characteristics of modern computing hardware.
So maybe we need to redesign the hardware.
Perhaps
John Nagle na...@animats.com writes:
I've argued for an approach in which only synchronized or immutable
objects can be shared between threads. Then, only synchronized objects
have refcounts. See
http://www.animats.com/papers/languages/pythonconcurrency.html;
I'm going to have to read
John Nagle na...@animats.com writes:
Unoptimized reference counting, which is what CPython does, isn't
all that great either. The four big bottlenecks in Python are boxed
numbers, attribute lookups, reference count updates, and the GIL.
The performance hit of having to lock the refcounts
On 9/4/2010 11:51 PM, Paul Rubin wrote:
John Naglena...@animats.com writes:
Unoptimized reference counting, which is what CPython does, isn't
all that great either. The four big bottlenecks in Python are boxed
numbers, attribute lookups, reference count updates, and the GIL.
The
Dennis Lee Bieber wlfr...@ix.netcom.com writes:
Not to mention having to ensure that one finds ALL the references to
the object so that they can be updated to the new address! Somehow I
don't see a C compiler being smart enough to find intermediary pointer
We're not talking about C
[gc]
In article 7x7hj2kyd6@ruckus.brouhaha.com,
Paul Rubin no.em...@nospam.invalid wrote:
A minimal naive implementation indeed doubles the memory requirements,
but from a Python perspective where every integer takes something like
24 bytes already, even that doesn't seem so terrible.
Lawrence D'Oliveiro l...@geek-central.gen.new_zealand writes:
That reinforces my point, about how easy it was to check the correctness of
the code. In this case one simple fix, like this ...
would render the code watertight. See how easy it is?
Well, no, it's irrelevant how easy it is to fix
On 8/28/2010 5:42 AM, Aahz wrote:
In article4c78572c$0$28655$c3e8...@news.astraweb.com,
Steven D'Apranost...@remove-this-cybersource.com.au wrote:
On Fri, 27 Aug 2010 09:16:52 -0700, Aahz wrote:
In articlemailman.1967.1281549328.1673.python-l...@python.org, MRAB
pyt...@mrabarnett.plus.com
In message mailman.434.1283568372.29448.python-l...@python.org, MRAB
wrote:
Lawrence D'Oliveirol...@geek-central.gen.new_zealand writes:
Wonder why Sun’s licence explicitly forbade its use in danger-critical
areas like nuclear power plants and the like, then?
I thought it was just that
In message 4c82b097$0$1661$742ec...@news.sonic.net, John Nagle wrote:
Personally, I'd like to have reference counting only, an enforced
prohibition on loops (backpointers must be weak pointers), RAII,
and reliably ordered finalization.
Is there a cheap way of checking at runtime for
In message 7x7hj2kyd6@ruckus.brouhaha.com, Paul Rubin wrote:
Lawrence D'Oliveiro l...@geek-central.gen.new_zealand writes:
In message 7xmxs2uez1@ruckus.brouhaha.com, Paul Rubin wrote:
GC's for large systems generally don't free (or even examine) individual
garbage objects. They
Lawrence D'Oliveiro l...@geek-central.gen.new_zealand writes:
A minimal naive implementation indeed doubles the memory requirements,
but from a Python perspective where every integer takes something like
24 bytes already, even that doesn't seem so terrible.
Doubling 24 is less terrible than
On 9/4/2010 6:44 PM, Lawrence D'Oliveiro wrote:
In message4c82b097$0$1661$742ec...@news.sonic.net, John Nagle wrote:
Personally, I'd like to have reference counting only, an enforced
prohibition on loops (backpointers must be weak pointers), RAII,
and reliably ordered finalization.
Is
In message 7xvd6sv0n4@ruckus.brouhaha.com, Paul Rubin wrote:
Lawrence D'Oliveiro l...@geek-central.gen.new_zealand writes:
AddrObj = PyTuple_GetItem(TheBufferInfo, 0);
LenObj = PyTuple_GetItem(TheBufferInfo, 1);
the first PyTuple_GetItem succeeds and the second one
In message 7xiq2que93@ruckus.brouhaha.com, Paul Rubin wrote:
Lawrence D'Oliveiro l...@geek-central.gen.new_zealand writes:
Refcounting is susceptable to the same pauses for reasons already
discussed.
Doesn’t seem to happen in the real world, though.
def f(n):
from time
In message 7xmxs2uez1@ruckus.brouhaha.com, Paul Rubin wrote:
Lawrence D'Oliveiro l...@geek-central.gen.new_zealand writes:
Whereas garbage collection will happen at some indeterminate time long
after the last access to the object, when it very likely will no longer
be in the cache, and
In message 7xr5heufhb@ruckus.brouhaha.com, Paul Rubin wrote:
Java has considerably greater reputation for reliability than C or C++.
Wonder why Sun’s licence explicitly forbade its use in danger-critical areas
like nuclear power plants and the like, then?
Ada is a different story, but
Lawrence D'Oliveiro l...@geek-central.gen.new_zealand writes:
Java has considerably greater reputation for reliability than C or C++.
Wonder why Sun’s licence explicitly forbade its use in danger-critical
areas like nuclear power plants and the like, then?
Probably because Sun lawyers
On 04/09/2010 03:21, Paul Rubin wrote:
Lawrence D'Oliveirol...@geek-central.gen.new_zealand writes:
Java has considerably greater reputation for reliability than C or C++.
Wonder why Sun’s licence explicitly forbade its use in danger-critical
areas like nuclear power plants and the like,
Lawrence D'Oliveiro l...@geek-central.gen.new_zealand writes:
GC's for large systems generally don't free (or even examine) individual
garbage objects. They copy the live objects to a new contiguous heap
without ever touching the garbage, and then they release the old heap.
And suddenly
On 8/30/2010 12:22 AM, Paul Rubin wrote:
I guess that is how the so-called smart pointers in the Boost C++
template library work. I haven't used them so I don't have personal
experience with how convenient or reliable they are, or what kinds of
constraints they imposed on programming style.
Dennis Lee Bieber wlfr...@ix.netcom.com writes:
GC's for large systems ... copy the live objects to a new contiguous heap
That sounds suspiciously like the original Macintosh OS, with its
handles... IE, double-indirection.
Nah, a double indirection on every access would be a terrible
Lawrence D'Oliveiro l...@geek-central.gen.new_zealand writes:
And yet Java code, for example, doesn’t have a reputation for greater
reliability compared to, say code written in Ada or C++, or even C. What is
the Java runtime written in? C. Why not use Java, if there is no speed
penalty in
Lawrence D'Oliveiro l...@geek-central.gen.new_zealand writes:
Whereas garbage collection will happen at some indeterminate time long after
the last access to the object, when it very likely will no longer be in the
cache, and have to be brought back in just to be freed,
GC's for large
Lawrence D'Oliveiro l...@geek-central.gen.new_zealand writes:
Refcounting is susceptable to the same pauses for reasons already
discussed.
Doesn’t seem to happen in the real world, though.
def f(n):
from time import time
a = [1] * n
t0 = time()
del a
In message 7xtymbzixt@ruckus.brouhaha.com, Paul Rubin wrote:
It's pretty well established by now that GC doesn't have any significant
speed penalty compared with manual allocation. It does consume more
memory, which is acceptable a lot of the time. It certainly leads to
more reliable
In message 7x39tz42fd@ruckus.brouhaha.com, Paul Rubin wrote:
Dennis Lee Bieber wlfr...@ix.netcom.com writes:
Heap marking, OTOH, tends to run at indeterminate times, which could
have an impact if one needs predictable response timings
Reference counting has the same problem. If you
In message 7xmxs4uzjl@ruckus.brouhaha.com, Paul Rubin wrote:
Gregory Ewing greg.ew...@canterbury.ac.nz writes:
I'd be disappointed if CPython ditched refcounting and
then became unsuitable for real-time games as a result.
Refcounting is susceptable to the same pauses for reasons
Lawrence D'Oliveiro l...@geek-central.gen.new_zealand writes:
static void GetBufferInfo
( ...
do /*once*/
{
TheBufferInfo = PyObject_CallMethod(FromArray, buffer_info, );
if (TheBufferInfo == 0)
break;
AddrObj = PyTuple_GetItem(TheBufferInfo,
Steven D'Aprano steve-remove-t...@cybersource.com.au writes:
I'm not saying that ref counting systems can avoid incrementing and
decrementing the ref counts. That would be silly. But I'm saying that it
is an accident of implementation that writing C extensions requires you
to manually
On Mon, 30 Aug 2010 00:22:17 -0700, Paul Rubin wrote:
I don't think a C compiler could really manage automatic decrementing
while still being C. Think especially of the common style of exception
handling in C using longjmp.
You might very well be right. But that's the problem with C -- it's
Paul Rubin wrote:
These days I think the GC pause issue is overrated except for real-time
control applications.
Also for games, which are a fairly common application
these days. Even a few milliseconds can be too long when
you're trying to achieve smooth animation.
I'd be disappointed if
In message 7xr5hg3a7s@ruckus.brouhaha.com, Paul Rubin wrote:
Actually that code looks suspicious. Suppose in
AddrObj = PyTuple_GetItem(TheBufferInfo, 0);
LenObj = PyTuple_GetItem(TheBufferInfo, 1);
the first PyTuple_GetItem succeeds and the second one fails.
In message 7x39twpuxi@ruckus.brouhaha.com, Paul Rubin wrote:
Lawrence D'Oliveiro l...@geek-central.gen.new_zealand writes:
the CPython API means endlessly finding and fixing refcount bugs that
lead to either crashes/security failures, or memory leaks.
I don’t see why that should be so.
Lawrence D'Oliveiro l...@geek-central.gen.new_zealand writes:
AddrObj = PyTuple_GetItem(TheBufferInfo, 0);
LenObj = PyTuple_GetItem(TheBufferInfo, 1);
the first PyTuple_GetItem succeeds and the second one fails.
Admittedly, I did take a shortcut here: array.buffer_info
Lawrence D'Oliveiro l...@geek-central.gen.new_zealand writes:
If it hasn't happened to you yet, you're either burning a bunch of effort
that programmers of more automatic systems can put to more productive
uses ...
What makes you say that? Avoiding bugs is not a “productive use”?
Avoiding
Gregory Ewing greg.ew...@canterbury.ac.nz writes:
These days I think the GC pause issue is overrated except for real-time
control applications.
Also for games, which are a fairly common application
these days. Even a few milliseconds can be too long when
you're trying to achieve smooth
In article 4c7b279d$0$28650$c3e8...@news.astraweb.com,
Steven D'Aprano steve-remove-t...@cybersource.com.au wrote:
On Sun, 29 Aug 2010 17:52:38 -0700, Paul Rubin wrote:
Attribution lost:
That's a problem with the CPython API, not reference counting. The
problem is that the CPython API is
In message 7xr5hguzzi@ruckus.brouhaha.com, Paul Rubin wrote:
JHC (experimental Haskell compiler) recently started using a mixture of
gc and region inference. It will be interesting to see how that works
out.
That’s what people have been saying about garbage collection for about half
a
Lawrence D'Oliveiro l...@geek-central.gen.new_zealand writes:
Meanwhile, real-world programmers get on to writing real-world code that is
productive and efficient on real-world systems.
It's pretty well established by now that GC doesn't have any significant
speed penalty compared with manual
Steven D'Aprano wrote:
On Sat, 28 Aug 2010 00:33:10 -0700, Paul Rubin wrote:
If you drop the last reference
to a complex structure, it could take quite a long time to free all the
components. By contrast there are provably real-time tracing gc
schemes, including some parallelizeable ones.
Hans Mulder han...@xs4all.nl writes:
Parallelizable garbage collectors have performance issues, but they're
not the same issues as marksweep collectors have. Parallelizable GCs
break up their work in a zillion little pieces and allow the VM to do
some real work after each piece. They won't
Steven D'Aprano st...@remove-this-cybersource.com.au writes:
You can add cycle detection to a reference count gc, at the cost of more
complexity.
But then it's not purely a refcount gc. ;)
If you read the Wikipedia article I linked to, tracing algorithms can
also be unsound: [describes
Steven D'Aprano st...@remove-this-cybersource.com.au writes:
I could be wrong, but how can they not be subject to the same performance
issue? If you have twenty thousand components that all have to be freed,
they all have to be freed whether you do it when the last reference is
cleared, or
On Sun, 29 Aug 2010 17:52:38 -0700, Paul Rubin wrote:
That's a problem with the CPython API, not reference counting. The
problem is that the CPython API is written at too low a level, beneath
that at which the garbage collector exists, so naturally you have to
manually manage memory.
Care
In message 7x4oeftuk4@ruckus.brouhaha.com, Paul Rubin wrote:
I'd say [reference-counting is] not real gc because 1) it's unsound
(misses reference cycles), and 2) it requires constant attention from the
mutator to incr and decr the reference counts. So developing modules for
the CPython
Lawrence D'Oliveiro l...@geek-central.gen.new_zealand writes:
the CPython API means endlessly finding and fixing refcount bugs that lead
to either crashes/security failures, or memory leaks.
I don’t see why that should be so. It seems a very simple discipline to
follow: initialize, allocate,
Dennis Lee Bieber wlfr...@ix.netcom.com writes:
The nice thing about it [reference counting] is that it is sort
of deterministic -- one can examine code and determine that an object
is collected at some point in the execution...
Heap marking, OTOH, tends to run at indeterminate
On Sat, 28 Aug 2010 00:33:10 -0700, Paul Rubin wrote:
Dennis Lee Bieber wlfr...@ix.netcom.com writes:
The nice thing about it [reference counting] is that it is sort
of deterministic -- one can examine code and determine that an object
is collected at some point in the execution...
On Fri, 27 Aug 2010 18:06:19 -0700, Paul Rubin wrote:
Steven D'Aprano st...@remove-this-cybersource.com.au writes:
I've repeatedly asked, both here and elsewhere, why reference counting
isn't real garbage collection. Nobody has been able to give me a
satisfactory answer. As far as I can tell,
In article 4c78572c$0$28655$c3e8...@news.astraweb.com,
Steven D'Aprano st...@remove-this-cybersource.com.au wrote:
On Fri, 27 Aug 2010 09:16:52 -0700, Aahz wrote:
In article mailman.1967.1281549328.1673.python-l...@python.org, MRAB
pyt...@mrabarnett.plus.com wrote:
An object will be available
In article 4c78e7a5$0$28655$c3e8...@news.astraweb.com,
Steven D'Aprano st...@remove-this-cybersource.com.au wrote:
On the other hand, the reason that CPython still has reference counting
is that the alternatives tried so far are unacceptably for non-threaded
code.
No, it's *a* reason, the
In article mailman.1967.1281549328.1673.python-l...@python.org,
MRAB pyt...@mrabarnett.plus.com wrote:
An object will be available for garbage collection when nothing refers
to it either directly or indirectly. If it's unreferenced then it will
go away.
This isn't actually garbage collection as
On 8/11/2010 1:26 PM, EW wrote:
On Aug 11, 2:52 pm, Paul Rubinno.em...@nospam.invalid wrote:
EWericwoodwo...@gmail.com writes:
Well I cared because I thought garbage collection would only happen
when the script ended - the entire script. Since I plan on running
this as a service it'll run
On Fri, 27 Aug 2010 09:16:52 -0700, Aahz wrote:
In article mailman.1967.1281549328.1673.python-l...@python.org, MRAB
pyt...@mrabarnett.plus.com wrote:
An object will be available for garbage collection when nothing refers
to it either directly or indirectly. If it's unreferenced then it will
Steven D'Aprano st...@remove-this-cybersource.com.au writes:
I've repeatedly asked, both here and elsewhere, why reference counting
isn't real garbage collection. Nobody has been able to give me a
satisfactory answer. As far as I can tell, it's a bit of pretentiousness
with no basis in
Hi
I'm writing a multithreaded app that relies on Queues to move data
between the threads. I'm trying to write my objects in a general way
so that I can reuse them in the future so I need to write them in such
a way that I don't know how many producer and how many consumer
threads I might need.
On Aug 11, 12:55 pm, EW ericwoodwo...@gmail.com wrote:
Hi
I'm writing a multithreaded app that relies on Queues to move data
between the threads. I'm trying to write my objects in a general way
so that I can reuse them in the future so I need to write them in such
a way that I don't know
EW ericwoodwo...@gmail.com writes:
I also might have different consumer threads do
different tasks (for example one might write to a log and one might
write to SQL) so that again means I can't plan for a set ratio of
consumers to producers So it's unknown.
So this means that instead of
On Aug 11, 1:18 pm, Paul Rubin no.em...@nospam.invalid wrote:
EW ericwoodwo...@gmail.com writes:
I also might have different consumer threads do
different tasks (for example one might write to a log and one might
write to SQL) so that again means I can't plan for a set ratio of
consumers
EW wrote:
[snip]
So here the P2 thread has ended and gone away but I still have his
Queue lingering.
So on a thread I can use is_alive() to check status and use join() to
clean up but I don't see any analogous functionality for Queues. How
do I kill them? I thought about putting a suicide
On Aug 11, 1:55 pm, MRAB pyt...@mrabarnett.plus.com wrote:
EW wrote:
[snip]
So here the P2 thread has ended and gone away but I still have his
Queue lingering.
So on a thread I can use is_alive() to check status and use join() to
clean up but I don't see any analogous functionality
EW ericwoodwo...@gmail.com writes:
I thought about doing it that way and I could do it that way but it
still seems like there should be a way to clean up Queues on my own.
If I did it this way then I guess I'd be relying on garbage collection
when the script ended to clean up the Queues for
On Aug 11, 2:16 pm, Paul Rubin no.em...@nospam.invalid wrote:
EW ericwoodwo...@gmail.com writes:
I thought about doing it that way and I could do it that way but it
still seems like there should be a way to clean up Queues on my own.
If I did it this way then I guess I'd be relying on
EW ericwoodwo...@gmail.com writes:
Well I cared because I thought garbage collection would only happen
when the script ended - the entire script. Since I plan on running
this as a service it'll run for months at a time without ending. So I
thought I was going to have heaps of Queues hanging
Paul Rubin wrote:
EW ericwoodwo...@gmail.com writes:
Well I cared because I thought garbage collection would only happen
when the script ended - the entire script. Since I plan on running
this as a service it'll run for months at a time without ending. So I
thought I was going to have heaps
On Aug 11, 2:52 pm, Paul Rubin no.em...@nospam.invalid wrote:
EW ericwoodwo...@gmail.com writes:
Well I cared because I thought garbage collection would only happen
when the script ended - the entire script. Since I plan on running
this as a service it'll run for months at a time without
EW ericwoodwo...@gmail.com writes:
Well I can't really explain it but 1 Queue per task for what I'm
designing just doesn't feel right to me. It feels like it will lack
future flexibility.
That makes no sense at all. Multiple readers and writers per queue are
the way Python queues are
75 matches
Mail list logo
