Re: Parallelism and Concurrency was Re: Ideas for a (nntp: message (nntp: message 18 of 20) 14 of 20) Object-Belongs-to-Thread threading model
Em Dom, 2010-05-16 às 19:34 +0100, nigelsande...@btconnect.com escreveu:
3) The tough-y: Closed-over variables.
These are tough because it exposes lexicals to sharing, but they are so
natural to use, it is hard to suggest banning their use in concurrent
routines.
This is the point I was trying to address, actually. Having *only*
explicitly shared variables makes it very cumbersome to write threaded
code, specially because explicitly shared variables have a lot of
restrictions on what they can be (this is from my experience in Perl 5
and SDL, which was what brought me to the message-passing idea).
However, interpreters already have to detect closed over variables in
order to 'lift' them and extend their lifetimes beyond their natural
scope.
Actually, the interpreter might choose to to implement the closed-up
variables by keeping that entire associated scope when it is still
referenced by another value, i.e.:
{ my $a;
{ my $b = 1;
{ $a = sub { $b++ } } }
this would happen by the having every lexical scope holding a reference
to its outer scope, so when a scope in the middle exits, but some
coderef was returned keeping it as its lexical outer, the entire scope
would be kept.
This means two things:
1) the interpreter doesn't need to detect the closed over variables, so
even string eval'ed access to such variables would work (which is, imho,
a good thing)
2) all the values in that lexical scope are also preserved with the
closure, even if they won't be used (which is a bad thing).
It doesn't seem it would be any harder to lift them to shared
variable status, moving them out of the thread-local lexical pads and into
the same data-space as process globals and explicitly shared data.
It is still possible to do the detection on the moment of the runtime
lookup, tho...
My currently favoured mechanism for handling shared data, is via
message-passing, but passing references to the shared data, rather than
the data itself. This seems to give the reason-ability, compose-ability
and controlled access of message passing whilst retaining the efficiency
of direct, shared-state mutability.
That was part of my idea too, I wasn't trying to address remote
processes or anything like that, I was considering doing the queues in
shared memory for its efficiency.
Only the code that declares the shared
data, plus any other thread it choses to send a handle to, has any
knowledge of, and therefore access to the shared state.
If we can overcome the limitations we have in Perl 5 shared values, I'm
entirely in agreement with the above statement (assuming closed-over
values become shared transparently)
Effectively, allocating a shared entity returns a handle to the underlying
state, and only the holder of that handle can access it. Such handles
would be indirect references and only usable from the thread that creates
them. When a handle is passed as a message to another thread, it is
transformed into a handle usable by the recipient thread during the
transfer and the old handle becomes invalid. Attempt to use an old handle
after it has been sent result in a runtime exception.
This is exactly what I meant by RemoteValue, RemoteInvocation and
InvocationQueue in my original idea.
daniel
Re: Parallelism and Concurrency was Re: Ideas for a (nntp: message (nntp: message 18 of 20) 14 of 20) Object-Belongs-to-Thread threading model
On Tue, 18 May 2010 11:39:04 +0100, Daniel Ruoso dan...@ruoso.com wrote:
This is the point I was trying to address, actually. Having *only*
explicitly shared variables makes it very cumbersome to write threaded
code, specially because explicitly shared variables have a lot of
restrictions on what they can be (this is from my experience in Perl 5
and SDL, which was what brought me to the message-passing idea).
Well, do not base anything upon the restrictions and limitations of the
Perl 5 threads/shared modules. They are broken-by-design in so many ways
that they are not a good reference point. That particular
restriction--what a :shared var can and cannot hold--is in some cases just
an arbitrary restriction for no good reason that I can see.
For example: file handles cannot be assigned to :shared vars is totally
arbitrary. This can be demonstrated in two ways:
1) If you pass the fileno of the filehandle to a thread and have it dup(2)
a copy, then it can use it concurrently with the originating thread
without problems--subject to the obvious locking requirements.
2) I've previously hacked the sources to bypass this restrict by adding
SVt_PVGV to the switch in the following function:
SV *
Perl_sharedsv_find(pTHX_ SV *sv)
{
MAGIC *mg;
if (SvTYPE(sv) = SVt_PVMG) {
switch(SvTYPE(sv)) {
case SVt_PVAV:
case SVt_PVHV:
case SVt_PVGV: // !!!
if ((mg = mg_find(sv, PERL_MAGIC_tied))
mg-mg_virtual == sharedsv_array_vtbl) {
return ((SV *)mg-mg_ptr);
}
break;
default:
/* This should work for elements as well as they
* have scalar magic as well as their element magic
*/
if ((mg = mg_find(sv, PERL_MAGIC_shared_scalar))
mg-mg_virtual == sharedsv_scalar_vtbl) {
return ((SV *)mg-mg_ptr);
}
break;
}
}
/* Just for tidyness of API also handle tie objects */
if (SvROK(sv) sv_derived_from(sv, threads::shared::tie)) {
return (S_sharedsv_from_obj(aTHX_ sv));
}
return (NULL);
}
And with that one change, sharing file/directory handles in Perl 5 became
possible and worked.
The problem is, GVs can hold far more than just those handles. And many of
the glob-modules utilise the other slots in a GV (array/hahs scalaer etc.)
for storing state and bless them as objects. At that point--when I tried
the change--the was a conflict between the blessing that Shared.XS uses to
make sharing working and any other type of blessing. The net result was
that whilst the change lifted the restriction upon simple globs, it still
didn't work with many of the most useful glob-based module--IO::Socket::*;
HTTP::Deamon; etc. I guess that now the sharing of blessed objects has
been mage possible, I shoudl try the hack again a see if it would allow
those blessed globs to work.
Anyway, the point is that the limitations and restrictions of the Perl5
implementation of the iThreads model, should not be considered as
fundamental problems with with the iThreads model itself. They aren't.
However, interpreters already have to detect closed over variables
in
order to 'lift' them and extend their lifetimes beyond their natural
scope.
Actually, the interpreter might choose to to implement the closed-up
variables by keeping that entire associated scope when it is still
referenced by another value, i.e.:
{ my $a;
{ my $b = 1;
{ $a = sub { $b++ } } }
this would happen by the having every lexical scope holding a reference
to its outer scope, so when a scope in the middle exits, but some
coderef was returned keeping it as its lexical outer, the entire scope
would be kept.
This means two things:
1) the interpreter doesn't need to detect the closed over variables, so
even string eval'ed access to such variables would work (which is, imho,
a good thing)
You'd have to explain further for me to understand why it is necessary to
keep whole scopes around:
- in order to make closures accessible from string-eval;
- and why that is desirable?
2) all the values in that lexical scope are also preserved with the
closure, even if they won't be used (which is a bad thing).
Please no! :)
This is essentially the biggest problem with the Perl 5 iThreads
implementation. It is the *need* (though I have serious doubts that it is
actually a need even for Perl 5), to CLONE entire scope stacks every time
you spawn a thread that makes them costly to use. Both because of the time
it takes to perform the clone at spawn time; and the memory used to keep
copies of all that stuff that simply isn't wanted; and in many cases isn't
even accessible. AFAIK going by what I can find about the history of
iThreads development, this was only done in Perl 5 in order to provide the
Windows fork emulation.
But as a
Re: Parallelism and Concurrency was Re: Ideas for a (nntp: message (nntp: message 18 of 20) 14 of 20) Object-Belongs-to-Thread threading model
On Tue, 18 May 2010 11:41:08 +0100, Daniel Ruoso dan...@ruoso.com wrote: Em Dom, 2010-05-16 às 19:34 +0100, nigelsande...@btconnect.com escreveu: Interoperability with Perl 5 and is reference counting should not be a high priority in the decision making process for defining the Perl 6 concurrency model. If we drop that requirement then we can simply go to the we-can-spawn-as-many-os-threads-as-we-want model.. I do not see that as a requirement. But, I am painfully aware that I am playing catchup with all the various versions, flavours and colors of Perl6 interpreter. And more importantly, the significance of each of tehm. When I recently started following #perl6 I was blown away (and totally confused) by all the various flavours that the eval bot responded to. The funny thing is that I have a serious soft spot for the timelyness of reference couting GC. And I recently came across a paper on a new RCGC that claimed to address the circular reference problem without resorting to weak references or other labour intensive mechanisms; nor a stop-the-world GC cycle. I scanned the paper and it was essentially a multi-pass coloring scheme, but achived better performaince than most by a) running locally (to scopes I think) so that it had far fewer arenas to scan. b) Using a invative coloring scheme that meant it was O(N) rather than the usual O(N * M) Most of it went over my head, (as is often the case with aademic papers), but it seems real. But I think that is a boat that has long sailed for Perl 6? daniel
