Re: [perl #26057] [PATCH] Unified PMC/PObj accessors phase 2
On Sep 22, 2005, at 03:46, Joshua Hoblitt via RT wrote: [ghenriksen - Thu Feb 05 20:15:50 2004]: Leo, The patch is at the URL below, and I've split it into 4 for you. The classes-include-lib patch must be applied before any of the other 3. I've resolved the 3-4 conflicts that occurred since the patch was first abortively submitted on Monday, so the old patch (named 20040202-pmc-accessors.patch) should be discarded if it resurfaces. http://www.ma.iclub.com/pub/parrot/ I realized that some of the accessor macros should've been named PObj_* instead of PMC_* (since they apply to STRINGs and Buffers, too). So they are as of this patch. I've also macro-ized access to UnionVal in general, since it was sometimes used outside of the context of a pobj. [*] The old syntax continues to work, and so nobody's patches will break excl. those w conflicts. But the pobj-cache.foo_val and PMC_ptr1p/PMC_ptr2v macros ought to be treated as deprecated. — Gordon Henriksen [EMAIL PROTECTED] [* - Somewhat inadvisedly, I think. UnionVal is 8 bytes on a 32-bit architecture, but bloats to 16 bytes on a 64-bit architecture. The generic containers which use UnionVal don't appear to use both ptrs simultaneously or make use of the void*/int pair, so could use an 8-byte structure as their bucket type.] The URL for the patches seems to be dead. Do you still want your patches to be considered? -J There's absolutely no way that these patches would still apply cleanly. It would be less work to recreate them than to attempt to resolve the conflicts. — Gordon Henriksen [EMAIL PROTECTED]
RE: [CVS ci] class refactoring 1 - Integer
Precedence. print(day\n xor night\n); -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED] -Original Message- From: Sam Ruby [mailto:[EMAIL PROTECTED] Sent: Friday December 10, 2004 13:28 To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Subject: Re: [CVS ci] class refactoring 1 - Integer Mike Guy wrote: Perl5 Cxor always returns a standard boolean value, i.e. dualvar(0, '') or dualvar(1, '1').Perl6/Parrot should do the same thing. Try: perl -le print 'day' xor 'night' On the version of Perl I have installed, I get day as the result. - Sam Ruby
RE: First draft, IO event design
Dan Sugalski wrote: Gordon Henriksen wrote: So, for GUI events, could calling into parrot and doing the following from the OS event handler work to synchronously dispatch an event? ... parrot-ify a mouse-moved event into $P5 ... post $P5 checkevent Hm. No. For that I think you'd want: post $P5 wait $P5 I suppose that would work, since the wait would block the main OS thread. It does cause unnecessary context switches, though. What happens in this case if the parrot eventloop is stopped? Does the GUI thread block indefinitely in the wait op? That would keep the program open even though parrot seemingly wanted to shut down. Just decoupling the event queue and dispatch mechanisms seems cleaner to my eye. -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
Re: First draft, IO event design
On May 24, 2004, at 15.20, Dan Sugalski wrote: Event Ops = The opcodes in this section are a combination of event requests and event handling ops. It doesn't include the IO ops--those are separate. Most of the event request ops have two forms, one of which takes a callback PMC and user data PMC. checkevent Explicitly check to see if there are any events pending in the event queue and, if so process one. So, for GUI events, could calling into parrot and doing the following from the OS event handler work to synchronously dispatch an event? ... parrot-ify a mouse-moved event into $P5 ... post $P5 checkevent Hm. No. If there's already an event in the queue, then that won't work... :( Maybe this change, instead: polleventq(out Pevent) Shifts one event off of the queue into Pevent, or nulls Pevent. dispatch(in Pevent) Invokes the registered event handler(s) for Pevent. Which makes checkevent really this: polleventq Py isnull Py, .NO_EVENT dispatch Py .NO_EVENT: Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
RE: Non-flow-control logical tests
Dan Sugalski wrote: Right now the only good way to find out if a value is true or not is to do something like: $I0 = 1 if $P0, done $I0 = 0 done: and look in $I0 for the result. This is OK, but if you're dealing with a language with relatively primitive views of logical operations (i.e they return 0 or 1) you end up with a *lot* of these little code snippets when evaluating something like: if ((foo or bar) and ((x1 = 12) or (x2 15) or (x5 and x6 and x7)) goto somewhere (And yes, that's close to a real code snippet. Feel my pain here :) Not a huge deal to translate, but once you hit 30K basic blocks in a sub the register coloring algorithm really gets unhappy and dies an unpleasant death after an hour or so. So: branches are bad for modern (CPUs|VMs|...). I thought we'd established that? :) Anyway, because of it I'm pondering non-flowcontrol logical ops. That is, something like: istrue I0, P5# I0 = 1 if P5 is true isgt I0, P5, P6 # I0 = i if P5 P6 There's definitely precedence for primitive bools, which is one step beyond where you're writing about. PowerPCs have a condition register, which is really better thought of as a set of 32 one-bit boolean registers. The bits can be filled by gt and lt operators and the like. Next, they can be 'd, ||'d, !'d, etc. Finally, the branch conditional instruction executes, deciding whether to branch just off of that one bit in the CR. Using CR operators can save pipeline bubbles over 's with branches. Of course, short-circuiting expression evaluation requires actually branching mid-expression. Given the semi-self-serving nature of these I think some discussion's in order first. Doesn't seem self-serving at all if it helps make common code constructs more amenable to the optimizer. -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
RE: Events (I think we need a new name)
Matt Fowles wrote: I think Page already has a different meaning in computers, namely a page of memory. Not to mention a web page. For what it is worth, I support event as the name. Being as I think I'm largely responsible for the sense that the name needs to be changed, I should point out that I do actually support calling these events--so long as they're modified to play nice with OS event loops. Upon reflection, that just requires a means to synchronously dispatch an event to a handler chain from a C callback. -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
Re: Event design sketch
On May 12, 2004, at 09.12, Dan Sugalski wrote: At 2:59 PM -0400 5/11/04, Gordon Henriksen wrote: As I pointed out in another post, this doesn't work for integrating with at least two significant event sources: Windows and the Mac OS. :) UI events need to be handled synchronously on the thread to which they were delivered, since the GUI APIs are not threadsafe. Oh, it's worse than thatGUI commands need to be issued from the main thread, at least with OS X. (There's no actual requirement as to which thread handles the actual events as long as you treat the OS event queue as the thread-unsafe thing it seems to be) Or so the docs seem to indicate, though they may be a bit conservative. That's exactly true, and exactly my point. GUI APIs are definitely not thread-safe, and for very good reasons. Not only do the system APIs essentially mandate it, but any depth of thought will make it obvious that UI events must be handled synchronously. Keeping also in mind that the main event loop stays on the execution stack, waking up to call back into event handlers; WaitNextEvent is dead. So funneling all event delivery through a Big Parrot Queue is unfeasible if your definition of event includes UI events. UI events, with their threading and context requirements, are really much more an issue of NCI callbacks, re-entrancy, and wrappers around system types. That said, the technology you're proposing is still, exactly as it stands, incredibly useful: Asynchronous completion. Signal handling. Runtime state change notifications, for undoing speculative optimizations. It's just not at all suitable for handling GUI events. Which is perfectly fine: Just either 1. Don't call it events so that people aren't disappointed and frustrated, or 2. Figure out some way to fold in GUI events. Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
Haps
That's really and truly evil. I love it. -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
RE: Event design sketch
Brent 'Dax' Royal-Gordon wrote: Gordon Henriksen wrote: Oh, it's worse than thatGUI commands need to be issued from the main thread, at least with OS X. (There's no actual requirement as to which thread handles the actual events as long as you treat the OS event queue as the thread-unsafe thing it seems to be) Or so the docs seem to indicate, though they may be a bit conservative. ... 1. Don't call it events so that people aren't disappointed and frustrated, or 2. Figure out some way to fold in GUI events. One general-purpose Parrot event queue, plus another for the GUI thread. When you call a method on the GUI bindings, it really enqueues an event for the GUI thread, which receives the message and calls the actual method. Or for that matter, enqueue an event in the main queue that only the GUI thread is interested in. Brent, That's tantamount to fine-grained locking on a collection; instead of a mutex, an OS thread does the serialization. Fine-grained locking doesn't work. It's also going to be ridiculously slow. Can you imagine what havoc this would wreak with multiple writers? Not a solution. The Carbon Event Manager is (synchronously) interested in your callback's return value; if a Carbon event handler callback returns kEventNotHandled, the Event Manager may supply a default implementation. Not a solution. A GUI event simply MUST be handled from the context that the operating system or GUI toolkit called the callback; anything else is addled. This might be too large a problem, despite seemingly easy to squash into one API at the first pass. Might it help to consider breaking down and decoupling the problem for better flexibility? Here's a pass... - class: EventData Parameter block storing information about an event that has occurred. Passed as the only argument to event handlers by convention. No intrinsic fields or operations. Subclasses: IOEventData, SignalEventData, MacMouseEventData, Win32MouseEventData, ... - class: Event A registery of event handlers for a particular event. Defines ordering and fallback behavior and soforth, should multiple handlers be registered. Instantiate one Event for each distinguishable kind of event which can occur; handlers can then. Anonymous handlers? Not so much. How magic collections which make Events on-demand? Methods: RegisterHandler(handler) UnregisterHandler(handler) This API is awkward without method pointers. So add a userdata parameter; whatever, details. CallHandlers([EMAIL PROTECTED]) Synchronously calls registered handlers. Parrot's event loop would call this, as would anything which wanted to control the thread on which events were dispatched. Subclasses? Maybe to override the behavior of CallHandlers, for specialized fallback behaviors. Or perhaps to register interest in the event with the OS event. - class: EventSource Something that can wake parrot's main event loop. Methods: Enable() Turn on the spigot: Register the event source with parrot's main event loop. Disable() Turn the spigot back off. Non-recurring sources (e.g., one-off timers) ought to disable themselves automatically. Subclasses: Timer, IOEventSource, SignalEventource This is all user-mode API, very close to the operations I'd want to see from an HLL. Adapt for C/parrot core use as appropriate. I've avoided cluttering the API by adding state variables; method pointers generally take care of that just fine. If EventData is kept naïve of Event, and Event is kept naïve of EventSource, then I think this comprises a pretty agile design. All a GUI event loop needs to do to fit in is to invoke CallHandlers(...) on an Event from a C shim. The event will be handled synchronously on that thread, and any return values necessary can be passed back in a field of the EventData. (Parameter blocks are good for both directions, after all.) Since the event loop would know the thread/interpreter, it can safely create the EventData object. For high-performance subsystems, it might be best to be able to skip the EventData if no parameters to the operation are necessary. Also consider user-mode events. A user program might want to use events to, say, signal changes in data to UI views which are displaying that data. That's probably a synchronous, rather than asynchronous, operation. All of the scary asynchronous stuff is hidden away in the EventSource and parrot's event loop. Nothing above is so intrinsically asynchronous. Which is fine. The internal event queue and runloop undoubtedly require considerable
RE: Event design sketch
Dan Sugalski wrote: At 10:33 AM -0700 5/11/04, chromatic wrote: On Tue, 2004-05-11 at 10:24, Dan Sugalski wrote: I'm also curious how to write an interface to an existing event system. Being able to write it all in PASM is a bonus. I don't think it can be all-PASM, except maybe (and maybe not...) with a separate thread for the existing event source. To do it in all pasm means calling back into parrot from interrupt level, which isn't really doable, or have a thread just waiting on events from the alternate event system to post into the parrot event queue. Another approach may be to expose the PollEvent and WaitEvent functions to the event system as alternate sources of events. If I can do that from PASM, I think I'm okay. That'll still need some C. The event system as it stands is all active--all event sources put events into the system, rather than having the event system go looking at event sources for events. You'd either need to queue up regular timer events to go check for new stuff or have a thread actively polling the source for events and throwing them into parrot's event queue. What's the plan for integrating with system events, then? Mac OS X and Windows both have robust, irreplacable, system-managed event loops. parrot's loop can (and should) run in a parallel thread to those, but certainly can't presume to take over entirely. It simply can't work. If you're calling this an event system, it ought to mesh in with the notion of events that every programmer who uses them will have: - Mouse events. - Keyboard events. - Redraw events. - Drag-'n'-drop events. - Menu command events. - etc., etc., etc.. Problems in this domain also include: Focus management, propagation along the responder chain, enabling/disabling commands, default event handlers. You won't get any of these events from an I/O wait on Win32 or on a Mac; they're not even delivered via Unix I/O. Note that the thread on which these events must be handled is the thread to which they are delivered (and not by parrot's event loop): UI APIs are not thread-safe. So the UI thread needs to be able to enter a parrot callback on the same thread. Asynchronous I/O completion is surely considered less of an event-handling problem and more of a thread-synchronization problem. Also, parrot async I/O completion hopefully won't need to be serialized through an I/O retirement thread (event loop, whatever you want to call it) except when the platform winds up requiring that through sloppy async APIs. This is really an asynchronous notification API, where asynchronous completion is a very important fires-once form of asynchronous notification. It's far, far below what programmers will expect when hearing the term event. .NET's thread pool is a very close match to what you're discussing. A .NET WaitHandle is almost identical to your event source. (Right down to the waitone/waitany ops you just suggested.) The intuitive concept of events isn't even in the same class of problem that your document addresses. Probably best to use another term for this (very cool, very necessary) technology. -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED] P.S. - Now that they're in, have you considered using objects and methods instead of opcodes to define parrot APIs? If parrot technologies are exposed through objects, you'll save on opcount, and HLLs won't need to build yet another a shim for every new parrot feature. Dogfood.
RE: Event design sketch
Dan Sugalski wrote: chromatic wrote: So for SDL, I'd start a separate thread that blocks on SDL_WaitEvent, creating and posting events when they happen. My main program would handle the events as normal Parrot events. Standard producer consumer stuff. Since it's blocking, it won't eat up too many resources -- that's nice. It'd be nice to have the SDL event thread ignore events I don't care about though, instead of creating event PMCs I'll just throw away later. You can always Get Horribly Clever in the event handling thread and look at what the SDL library's handed you. If it's uninteresting you can just throw it away rather than creating an event to be discarded. Is this what you have in mind? Yep. As I pointed out in another post, this doesn't work for integrating with at least two significant event sources: Windows and the Mac OS. :) UI events need to be handled synchronously on the thread to which they were delivered, since the GUI APIs are not threadsafe. Trying to handle these events from another thread is, quite simply, a doomed endeavour. -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
Reference/value semantics, destructive operators, etc.
(Just a quick post from work. Lost the thread, sorry, so it's not a
f'up.)
Why not use +=, -=, *=, /=, etc.? Arithmetic operators would take just 2
arguments and mutate the first. I always used this technique for
implementing value-ish classes in C++:
thing operator +(thing lhs, thing rhs) {
return new thing(lhs) += rhs;
}
This is really pretty much the same problem.
Current add-in-place op becomes:
# P3 = P4 + P5
P3 = P4 # (copy)
P3 += P5
Add-in-place to a global becomes:
; $::whatever += P4
fetch_global P3, whatever
P3 += P4
Chained expressions become:
# P3 = P4 + P5 + P6 + P7
P3 = P4 # (copy)
P3 += P5
P3 += P6
P3 += P7
I can see generating this syntax from an AST without completely
inordinate pain. It's not as convenient as a temp-making addition (Px =
Py + Pz) operator would be, but it also solves the modify-in-place
problem (since PMCs have reference semantics). So long as the x=
operators are also available for native types, so that the the same AST
traversal algorithm can work for all register types.
--
Gordon Henriksen
[EMAIL PROTECTED]
Re: Basic Library Paths (was Re: ICU data file location issues)
On Saturday, April 17, 2004, at 10:35 , Gordon Henriksen wrote:
Which suggests to me a linked list of resource resolvers. First one in
the chain to return a file handle to the data or PBC wins. The head of
parrot's own system chain would be available to be appended to any
other chains that wanted it.
And the more I mull this over, the more I really come up with maybe 4
slots in the search chain which are logically important. The order is up
for debate, but they all need to be in there (whenever they apply, that
is).
1. Paths relative to the PBC binary which is searching for a library.
2. Paths relative to the embedding application.
3. Paths relative to parrot itself (be that libparrot.shlib or parrot).
4. Paths to system libraries as specified by the administrator.
When searching for resources, only #1 should be used. Here are some
examples:
PBC File: (whatever)
Host app: /usr/local/bin/parrot
Parrot: /usr/local/lib/libparrot.shlib
Consider searches for:
icu.dat
Search path:
1. /usr/local/shared # Relative to executable
PBC File: D:\inetpub\wwwroot\example.pbchtml
Host app: C:\Apache\libexec\httpd.exe
Parrot: C:\Parrot\lib
Consider searches for:
icu.dat
mod_parrot.pbc
My::WWWUtil.pbc
Time::HiRes.pbc
Search path:
D:\inetpub\wwwroot{,\lib,\..\lib} # Relative to PBC
C:\Apache\libexec{,\lib,\..\lib} # Relative to host app
C:\CPAN\lib # System libraries
C:\Parrot\lib # Relative to parrot
PBC File: ./bin/fib
Host app: /home/me/bin/parrot
Parrot: /home/me/bin/parrot
Consider searches for:
icu.dat
Time::HiRes.pbc
fib.parrot_resource_file
One possible search path:
./bin/{,/lib,/../lib} # Relative to PBC
/usr/local/lib # System libraries
/home/me/lib # Relative to parrot
The scenario which gives me a little bit of heartburn is one like this,
though:
Consider, say, an e-commercesite package. Call it OneStep::ECS. Runs
under mod_parrot in Apache. Has hooks to load plugins:
Third-party plugins to provide connectivity to payment processing
engines (call it OneStep::VeriSignPayflow.pbc).
First-party plugins allowing the customer to integrate his
storefront with his database (call it
MySite::OneStepECSCustomizations.pbc).
Now consider searches for VeriSign::PayflowPro.pbc, PayFlowPro.dll,
MySite::CRM.pbc, MySite::Reporting.pbc, mysite_logo.png,
Time::HiRes.pbc, libparrot.pbc, CGI.pbc
So maybe some libraries are hosts and need to be included in the
search paths of libraries which are linked to them. One could even look
at libparrot that way, in which case the search path model becomes:
Paths relative to this PBC file.
Paths relative to its hosts.
Paths relative to its hosts' hosts.
Paths relative to its hosts' hosts' hosts.
...
Paths configured by the system administrator.
Gordon Henriksen
[EMAIL PROTECTED]
Re: ICU data file location issues
On Saturday, April 17, 2004, at 02:17 , Gordon Henriksen wrote:
On Thursday, April 15, 2004, at 02:25 , Jeff Clites wrote:
For Unix platforms at least, you should be able to do this:
executablePath = isAbsolute($0) ? dirname($0) : cwd().dirname($0)
That absolutely does not work, as already pointed out. Ths looks like a
reasonable reference implementation (LGPL), though:
http://www.opensource.apple.com/darwinsource/10.3/libiconv-9/libiconv/srclib/
progreloc.c
On Windows and Linux, it uses Win32 and /proc to provide a robust
implementation. Otherwise, it guesses by looking at $0 and $ENV{PATH}.
My guess is that there's a more reliable (and non-portable) way to do
this on Mac OS X, since Carbon applications need to reliably open the
resource fork of the executable.
Ah! Indeed there is.
http://developer.apple.com/documentation/Carbon/Reference/Process_Manager/
index.html
And, indeed, it is witheringly non-portable.
CFDictionaryRef dict = ProcessInformationCopyDictionary(
kCurrentProcess, kProcessDictionaryIncludeAllInformationMask);
CFString cfPath = (CFString *) CFDictionaryGetValue(dict,
kIOBundleExecutableKey);
CFIndex length = CFStringGetMaximumSizeForEncoding(cfPath,
kCFEncodingUTF8);
char *path = (char *) malloc(length + 1);
CFStringGetCString(cfPath, path, length + 1, kCFEncodingUTF8);
CFRelease(dict);
Ahem.
I'm sure the ProcessInformationCopyDictionary API is implemented in
terms of something sane at the Darwin level, but God only knows what it
is.
Gordon Henriksen
[EMAIL PROTECTED]
Re: Basic Library Paths (was Re: ICU data file location issues)
Dan Sugalski wrote:
Brent 'Dax' Royal-Gordon wrote:
Dan Sugalski wrote:
3) Parrot itself (the main executable) has a static, global 1K buffer
in it that starts and ends with some recognizable string (like, say,
***+++***START| and |END***+++***) so we can find it and
overwrite the contents if the library gets moved, for use on
platforms where the only way to put a path in is to stick it
statically in the executable.
That's pretty disgusting, but I don't know that I have a better idea.
There isn't one, alas, at least for some people.
Everyone running tripwire, et al. (or simply md5sum'ing files to verify
integrity) will just love this strategy to death.
Finding resource and library files relative to the binary really is a
very good strategy. Windows is adopting the placed-near-the-binary
strategy for locating resources and libraries. It has completely
eliminated DLL hell for .NET programs. Mac OS 7 through X have all
used the same strategy. They have never had major problems with library
or resource location. Looks like a strong precedent and a proven
technique.
Of course, one can find pathological casesespecially on Unix, which
seems designed to thwart this sort of easy-to-administer technology:
parrot binary unlink'd between exec and main(). (Can't happen on
Windows.)
Launched through a symlink to the binary.
Launched through a hard link to the binary.
bin/ is a symlink, so ../share won't work.
Platform can't find the binary. (Can't happen on Windows, Linux,
or Mac OS X.)
chroot (which, in general, near-the-binary solves rather than
complicates).
But I'd say these are all are heavily outweighed by the advantages. And,
in any case, it's a trivial matter at this point in design to offer
support for replacing a call to Parrot_get_bin_path() (or whatever) with
/usr/local/bin at configure time. That resolves all of the above. With
a loss of functionality, true, but: Users on platforms which can't
support this feature won't after all expect /opt/parrot to work after it
was mv'd.
As for the security concerns of trusting anything but one's current
binary*, parrot could adopt a cryptographic solution for verifying
integrity of resource files, if anybody's really all that worried about
an errant Unicode character database.
Gordon Henriksen
[EMAIL PROTECTED]
* - Is the binary itself is really all that trustworthy in the first
place? If a user is executing a program through an untrusted or
compromised path, they're already putting their life in their hands, and
accessing ${bin}/../share won't make the configuration any more
trustworthy.
Re: Basic Library Paths (was Re: ICU data file location issues)
On Thursday, April 15, 2004, at 01:48 , Dan Sugalski wrote: At this point I can say I don't honestly care all that much, and most of my worries are based on vague feelings that there are platforms out there where finding the actual executable name is somewhere between hard and impossible. I will, then, do the sensible thing and just punt on this--we can work out a best practices thing and enshrine it as the default on systems which can support it and be done with it. The other question, then, is do we see the need for multiple categories of library which would want separately settable library paths? Wouldn't it be sensible to build something robust enough to also solve the problems of finding parrot user libraries and user resources? In which case, a static search path is decidedly retro. It would hardly make sense to not include, at the front of the search path, directories relative to the PBC file trying to find its libraries or resources.* For finding resources, one doesn't generally want to fall back to system paths. Finding libraries is another matter. Then there's the mention of using URLs to load resources (e.g., over HTTP). Which seems sensible and forward-thinking to me. Which suggests to me a linked list of resource resolvers. First one in the chain to return a file handle to the data or PBC wins. The head of parrot's own system chain would be available to be appended to any other chains that wanted it. Gordon Henriksen [EMAIL PROTECTED] (* - The directory containing every loaded PBC file is not at all important; consider an application like Apache+mod_parrot which is loading multiple independent PBC files. It would be useful to allow the administrator to install both the production PBC in addition to a development release of the same application on the same web server [just at different paths], with confidence that mod_parrot won't get the two confused. [IIS can do this. It's very cool.])
Re: ICU data file location issues
On Thursday, April 15, 2004, at 02:25 , Jeff Clites wrote:
For Unix platforms at least, you should be able to do this:
executablePath = isAbsolute($0) ? dirname($0) : cwd().dirname($0)
That absolutely does not work, as already pointed out. Ths looks like a
reasonable reference implementation (LGPL), though:
http://www.opensource.apple.com/darwinsource/10.3/libiconv-9/libiconv/srclib/
progreloc.c
On Windows and Linux, it uses Win32 and /proc to provide a robust
implementation. Otherwise, it guesses by looking at $0 and $ENV{PATH}.
My guess is that there's a more reliable (and non-portable) way to do
this on Mac OS X, since Carbon applications need to reliably open the
resource fork of the executable.
Gordon Henriksen
[EMAIL PROTECTED]
RE: Dates. Or, rather, months
(YAY!) -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED] -Original Message- From: Dan Sugalski [mailto:[EMAIL PROTECTED] Sent: Thursday March 11, 2004 10:34 To: [EMAIL PROTECTED] Subject: Dates. Or, rather, months Okay, unless there are objections I'm going to rejig the date decoding logic to return months from 1-12, rather than 0-11. We already fix years, so it seems to make sense. -- Dan --it's like this--- Dan Sugalski even samurai [EMAIL PROTECTED] have teddy bears and even teddy bears get drunk
RE: Dates and Times
Jared Rhine wrote: Gordon Henriksen wrote: gmclock(out Nx) UTC clock in seconds since hrs Jan 1, 2000, ignoring leap seconds. tolocal out Nx, out Iy, in Nz x is set to z converted to the local time zone. y - 1 if Daylight Savings Time was in effect at z; y - 0 otherwise. splittime Px, Nx Splits date up like Perl 5 gmtime. (But without annoying y -= 1900 and m -= 1?) add_months(out Nx, in Ny, in Nz) Sets x to y + z months. This proposal was perhaps taken offline by the core team, but to avoid a Warnock, I'll pipe up and say I like this proposal as a core instruction set. Thanks for the anti-Warnock. It'd be nice if the first item, gmclock wasn't defined in terms of UTC. Regardless of the future fate of UTC leap seconds, any UTC-based clock would need to account for leap seconds going back, right? So it seems that GMT should be preferred in the core, with any UTC calculations being performed at the language or module level. So, gmclock seems the right opcode (not utcclock), making the definition GMT clock in seconds since hrs Jan 1, 2000. Fair enough. I wasn't trying to load terms for once. In point of fact, this set of opcodes is very much agnostic to whether or not leap seconds are used. The reason that add_months belongs as an opcode is because there's been no decision made re leap seconds, and in fact some anti-decisions have been proposed. Without some way to manipulate dates in accordance with the actual rules that get used (which might be platform dependent at this point), a portable program can't be written. Were this decision made solidly, then all of the ops except gmclock and possibly tolocal would be better written as library code. Were leap seconds to fall in the middle of a month, then we would need to have an add_days op. Since that's not the case, the assumption that a day is 86400 seconds long is safe so long as you avoid adding enough days to cross a year boundary. Meanwhile, add_months encapsulates the leap-second worries AND lets IMC code conveniently manipulate dates without days-per-month tables and is-it-a-leap-year algorithms (which are required to convert add_months to add_days). Daylight savings time calculation seems appropriate to stay in the core, but perhaps additional opcodes are need to set the clock's understanding of the time zone used for DST calculations? Indeed; the local time zone is pretty narrow-minded and backwards- thinking. The problem is balancing weight with efficiency, and time zone databases are pretty heavy critters, which make me think get thee OUT of the core!... # years Nz = Py[5] Nz = Nz - 2000# epoch based at year 2000 Nz = Nz * 12 # = months per year inappropriate sarcasmI look forward to the exciting advancement of changing from subtracting 1900 from all my dates to subtracting 2000 or dealing with negative numbers; that'll change everything and really advance the state of the art./sarcasm What you're complaining about is the elements of Perl's gmtime/localtime arrays. Actually, I suggested removing that quirk from splittime: splittime Px, Nx Splits date up like Perl 5 gmtime. (But without annoying y -= 1900 and m -= 1?) ^^^ ^^ The Nz - 2000 is conversion *to* the epoch, not reinterpreting the value in the array. Unless you think storing dates and times as the number of seconds since Jan 1 is a good idea, perhaps. :) I'd used Jan 1 2000 as an epoch since Larry suggested that's what he was planning for Perl 6 to use. -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
RE: Dates and Times
Edward S. Peschko wrote: Gordon Henriksen wrote: Leave parsing and formatting entirely to libraries. Absolutely no need for that in the instruction set. well, I have a bit of a problem with that... As it was pointed out before, people have gone hogwild with the parsing and formatting routines, and its a bloodbath of modules on CPAN with different methods for time parsing. Not an opcode doesn't mean balkanized. There is a parrot/stdlib directory. Perhaps there could be an extra argument for locales - and maybe the op could be split in two (ie: for (r_)gm_fmtime, (r_)local_fmtime), but for me, in processing time values most of the time you know the format for dates - as long as they are machine generated. If they are not, you can always fall back to the very slow Date::Parse and cousins, or maybe string these ops along in a chain of '||' to query for multiple formats. But how many times are you going to need to parse formats like '3 weeks from next Wednesday?'. Uh. Yeah. This sort of creeping featuritis is why date formatting and especially parsing do NOT belong as opcodes. It's too big a problem to solve in the core, and regardless of how rich the interface is, it'll never be quite rich enough to satisfy everyone. Someone depends on formats like 3 weeks from next Wednesday on a regular basis. You might not need to format dates from the Chinese lunar calendar, but you can bet that it's vital to someone. Eras? Well, sure. Fuzzy date parsing? (Guessing.) Oh, baby. Global time zone database? Bring it on. A kl-hw (Klingon Homeworld) locale? But of course! There'll always be pressure to further enhance the feature, increasing parrot's *core* memory footprint. You can't skip loading the implem- entation of core parrot ops like you can avoid DateTime.pm. So keep it to (un)loadable user code, and provide a class that solves 90% of the problem in the stdlib. Also, trying to nail down a rich interface before seeing what Larry has in mind for Perl 6 date handling is roughly a waste of time -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
Re: Dates and Times
Edward, Want to call strptime? Use NCI. No need for anything new in the core. That's WHY it's the CORE. Gordon Henriksen [EMAIL PROTECTED]
RE: Dates and Times
Oracle actually has the most elegant Gregorian time manipulation implementation I've seen. The only operations it really supports are add_months and add_days. Year +/- can be implemented with add_months*12. All of add_week|hour|minute|second|ms|us|ns can be synthesized from add_days. This scarcity of operations yields the elegance, and is perfect for an instruction set. Based on that model, I'd suggest: gmclock(out Nx) UTC clock in seconds since hrs Jan 1, 2000, ignoring leap seconds. gmtime(out Px, in Nx) Splits date up like Perl 5 gmtime. (But without annoying y -= 1900 and m -= 1?) localtime(out Px, in Nx) Splits date up like Perl 5 localtime. (But without annoying y -= 1900 and m -= 1?) add_months(out Nx, in Ny, in Nz) Sets x to y + z months. That's the minimal core set of operations. But the redundancy of gmtime and localtime has always bothered me, so I could see this instead: gmclock(out Nx) UTC clock in seconds since hrs Jan 1, 2000, ignoring leap seconds. tolocal out Nx, out Iy, in Nz x is set to z converted to the local time zone. y - 1 if Daylight Savings Time was in effect at z; y - 0 otherwise. splittime Px, Nx Splits date up like Perl 5 gmtime. (But without annoying y -= 1900 and m -= 1?) add_months(out Nx, in Ny, in Nz) Sets x to y + z months. By contrast, date manipulation in Perl 5 is truly horrid. I've seen modules which turned a gmtime array back into an epoch-base value by using localtime to converge on the correct value using progressive approximation, as if finding the root of an arbitrary mathematical function. Doing the same using the above instructions can easily be implemented in 17 instructions flat with no branches: # out Nx: clock-style seconds-since-epoch # in Py: splittime-style array # Nz: temp Nx = 0 # years Nz = Py[5] Nz = Nz - 2000# epoch based at year 2000 Nz = Nz * 12 # = months per year add_months Nx, Nx, Nz # months Nz = Py[4] add_months Nx, Nx, Nz # days Nz = Py[3] Nz = Nz * 86400 # = 24 * 60 * 60 seconds per day Nx = Nx + Nz # hours Nz = Py[2] Nz = Nz * 3600 # = 60 * 60 seconds per hour Nx = Nx + Nz # minutes Nz = Py[1] Nz = Nz * 60 # = 60 seconds per minute Nx = Nx + Nz # seconds Nz = Py[0] Nx = Nx + Nz Leave parsing and formatting entirely to libraries. Absolutely no need for that in the instruction set. -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
Re: [PATCH] Configure test for inet_aton
On Friday, February 27, 2004, at 12:26 , Andrew Dougherty wrote: On Fri, 27 Feb 2004, Gordon Henriksen wrote: inet_pton is preferrable, as it supports IPv6. That is, inet_pton should trump inet_aton if both are available. Probably, but that would require testing to see if inet_pton is available. That information is not available from perl5's Configure, and there currently is no infrastructure in parrot for making such a test. Obviously that needs to change eventually. On Mac OS X, the preferred technique would be to weak link with inet_pton and test for its availability at run-time. (inet_pton is not available on 10.1, but is available on 10.3.) This would be another case entirely. Yes, indeed, that's the sort of thing that makes testing for function availability so much fun :-). FWIW, this can't work across the 10.1/10.2 schizm. http://developer.apple.com/technotes/tn2002/tn2064.html So parrot would need to: - either use a separate 10.1 binary (yuck), - or forgo use of inet_pton and IPv6 on Mac OS X 10.2 even though it is available (yuck), - or link in at runtime a different driver DLL for 10.1 vs. 10.2 (yuck). Roll-your-own doesn't really sound so bad. But until the implementation is not hardcoded to IPv6, preferring inet_aton (as present) gets the right behavior. But IPv6 can't be ignored forever. Whatevsies. Gordon Henriksen [EMAIL PROTECTED]
RE: [PATCH] Configure test for inet_aton
inet_pton is preferrable, as it supports IPv6. I think the code should
read either:
#if PARROT_HAS_INET_PTON
... use inet_pton ...
#else
... use inet_aton ...
#endif
or
#if PARROT_HAS_INET_PTON
... use inet_pton ...
#elsif PARROT_HAS_INET_ATON
... use inet_aton ...
#else
#error
#endif
That is, inet_pton should trump inet_aton if both are available.
On Mac OS X, the preferred technique would be to weak link with
inet_pton and test for its availability at run-time. (inet_pton is not
available on 10.1, but is available on 10.3.) This would be another case
entirely.
--
Gordon Henriksen
IT Manager
ICLUBcentral Inc.
[EMAIL PROTECTED]
-Original Message-
From: Andrew Dougherty [mailto:[EMAIL PROTECTED]
Sent: Friday February 27, 2004 11:52
To: Perl6 Internals
Subject: [PATCH] Configure test for inet_aton
On Fri, 27 Feb 2004, Leopold Toetsch wrote:
Peter Sinnott wrote:
I have downloaded the snapshot dated 27-Feb-2004 00:01 from
http://cvs.perl.org/snapshots/parrot/ and tested it on hpux.
I order to get it to compile I have to revert to using inet_aton
at line 623 of io_unix.c. inet_pton was causing
unsatisfied symbols
at link time.
We need a config test here.
Something like this ought to do the trick for now. Longer
term, we need
to build up some infrastructure for testing for functions on
our own. We
also need to centralize things more to avoid some of the
duplication of
code and information we're ending up with here.
diff -r -u -N parrot-current/MANIFEST parrot-andy/MANIFEST
--- parrot-current/MANIFEST 2004-02-27 03:00:34.0 -0500
+++ parrot-andy/MANIFEST 2004-02-27 09:36:23.0 -0500
@@ -78,6 +78,7 @@
config/auto/format.pl []
config/auto/funcptr.pl[]
config/auto/funcptr/test_c.in []
+config/auto/functions.pl []
config/auto/gc.pl []
config/auto/gc/test_c.in []
config/auto/gcc.pl[]
diff -r -u -N parrot-current/config/auto/functions.pl
parrot-andy/config/auto/functions.pl
--- parrot-current/config/auto/functions.pl 1969-12-31
19:00:00.0 -0500
+++ parrot-andy/config/auto/functions.pl 2004-02-27
09:36:56.0 -0500
@@ -0,0 +1,33 @@
+#! perl -w
+# Copyright: 20004 The Perl Foundation. All Rights Reserved.
+# $Id:$
+
+=head1 NAME
+
+config/auto/functions.pl - Probe for Various Functions
+
+=head1 DESCRIPTION
+
+This command probes for the existence of various functions.
+For the moment, it just pulls information from perl5's Configure;
+in the future, it ought to go looking on its own.
+
+=cut
+
+package Configure::Step;
+
+use strict;
+use vars qw($description @args);
+
+$description=Looking for various functions...;
+
[EMAIL PROTECTED]();
+
+sub runstep {
+# Do we have inet_aton() ?
+Configure::Data-set(
+ d_inet_aton = $Config{d_inetaton},
+);
+}
+
+1;
diff -r -u -N
parrot-current/config/gen/feature_h/feature_h.in
parrot-andy/config/gen/feature_h/feature_h.in
--- parrot-current/config/gen/feature_h/feature_h.in
2004-01-08 19:01:00.0 -0500
+++ parrot-andy/config/gen/feature_h/feature_h.in
2004-02-27 09:36:23.0 -0500
@@ -96,6 +96,21 @@
print OUT EOP;
+/* from config/auto/functions.pl */
+EOP
+if (${d_inet_aton}) {
+ print OUT EOP;
+#define PARROT_HAS_INET_ATON 1
+EOP
+}
+else {
+ print OUT EOP;
+/* #undef PARROT_HAS_INET_ATON */
+EOP
+}
+
+print OUT EOP;
+
/* from config/auto/inline */
EOP
if (${inline} ne '') {
diff -r -u -N parrot-current/io/io_unix.c parrot-andy/io/io_unix.c
--- parrot-current/io/io_unix.c 2004-02-19
19:00:06.0 -0500
+++ parrot-andy/io/io_unix.c 2004-02-27 11:09:48.0 -0500
@@ -617,11 +617,16 @@
{
struct sockaddr_in sa;
/* Hard coded to IPv4 for now */
-int family = AF_INET;
+#if !PARROT_HAS_INET_ATON
+int family = AF_INET; /* for inet_pton() */
+#endif
char * s = string_to_cstring(interpreter, addr);
-/*if(inet_aton(s, sa.sin_addr) != 0) {*/
+#if PARROT_HAS_INET_ATON
+if(inet_aton(s, sa.sin_addr) != 0) {
+#else
if(inet_pton(family, s, sa.sin_addr) != 0) {
+#endif
/* Success converting numeric IP */
}
else {
diff -r -u -N parrot-current/lib/Parrot/Configure/RunSteps.pm
parrot-andy/lib/Parrot/Configure/RunSteps.pm
--- parrot-current/lib/Parrot/Configure/RunSteps.pm
2003-11-27 19:00:50.0 -0500
+++ parrot-andy/lib/Parrot/Configure/RunSteps.pm
2004-02-27 09:36:23.0 -0500
@@ -4,6 +4,8 @@
use vars qw(@steps);
# EDIT HERE TO ADD NEW TESTS
+# Also update the slightly different version of this list
+# in Parrot::Configure::Docs:Section:Config.pm
@steps=qw(
init
RE: [PATCH] Configure test for inet_aton
Andrew Dougherty wrote: On Fri, 27 Feb 2004, Gordon Henriksen wrote: On Mac OS X, the preferred technique would be to weak link with inet_pton and test for its availability at run-time. (inet_pton is not available on 10.1, but is available on 10.3.) This would be another case entirely. Yes, indeed, that's the sort of thing that makes testing for function availability so much fun :-). Further, this trickery requires that such compatible builds of parrot be built with the latest environment (where the function prototypes at least are available)... Ah, fun, fun, fun. -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
Re: pdd15_objects.pod, attributes, properties, and life
On Friday, February 27, 2004, at 12:22 , Dan Sugalski wrote:
At 4:41 PM +0100 2/27/04, Paolo Molaro wrote:
On 02/27/04 Dan Sugalski wrote:
What .NET calls an attribute parrot calls a property
What .NET calls a property parrot calls an attribute
[...]
Oh, yeah. No matter which way we go we'll be confusing someone, since
there are languages that call what we call an attribute an attribute.
The .NET reversal's just the big nasty one.
[Paolo's property and attribute for .net explanation snipped]
Okay, now I'm *really* confused. This seems to contradict the
explanations in the C# in a Nutshell book, assuming that C# and .NET
use the same underlying terminology. OTOH I could be misreading the
book, and OTTH I'd be foolish to not take Paolo's explanation as most
correct. I think I may well just chop out part of the glossary.
I swear, I'm going to rename these things to Fred and Barney and not
tell anyone which is which...
Ordered by ascending esotericism:
Fields: Class data members.
Properties: Syntactic sugar for getter and setter methods.
Attributes: Extensible metadata for compile-time objects.
In context:
using System;
using ICLUBcentral.Testing;
namespace ICLUBcentral.Example {
class Eg {
// This is a field:
private int _num;
// This is a property:
public int Num {
get { return _num; }
set { _num = value; }
}
[TestCase(2)] // -- This is an attribute.
private void _TestNum() {
Test.OK(Num == 0, default value);
Num = 43;
Test.OK(Num == 43, changed value);
}
}
}
I have a program which will load the resultant DLL and run all methods
which have a TestCase attribute applied to them.
Am very, very sure of this terminology.
Gordon Henriksen
[EMAIL PROTECTED]
Re: Build broken due to missing inet_aton on Solaris 8
On Thursday, February 12, 2004, at 10:46 , Melvin Smith wrote:
At 10:26 AM 2/12/2004 -0500, Andrew Dougherty wrote:
A fresh checkout of parrot won't build for me due to the missing
inet_aton symbol on Solaris 8. My perl5 configuration correctly
records $Config{d_inetaton}=undef, but io_unix.o unconditionally
expects inet_aton.
cc -o parrot -L/usr/local/lib -R/usr/local/lib imcc/main.o \
blib/lib/libparrot.a -lsocket -lnsl -ldl -lm -lpthread -lrt
Undefined first referenced
symbol in file
inet_aton blib/lib/libparrot.a(io_unix.o)
ld: fatal: Symbol referencing errors. No output written to parr
I know I only develop on Linux and Solaris. Linux can use either
but Solaris needed inet_pton.
I'm not sure why Leo changed it, but I'll put it back.
Leo do you have an OS that does not have inet_pton?
(Sorry if this has already been discussed; catching up after being out
of down.)
Early versions of Mac OS X do not have inet_pton.
Gordon Henriksen
[EMAIL PROTECTED]
As the world stops: of GC and threads
So, I haven't heard any convincing evidence that execution in other threads can continue while garbage collection is executing, copying collector or not. (Point of fact, the copying collector has nothing to do with it.) So what are the options to stop the world? I've heard the first 2 of these from Dan and Leo. The third is mine. Any others? * Taking the read half of a reader-writer lock for all pointer mutations The point here is to block all mutators from mutating the object graph while the collector is traversing it. I can't imagine this being good for performance. This is in addition to the mutex on the PMC. Other threads might be scheduled before GC completes, but don't have to be for GC to proceed. * OS pause thread support Lots of operating systems (including at least several versions of Mac OS X) don't support this. Maybe parrot can use it when it is available. It's always a dangerous proposition, though; the paused thread might be holding the system's malloc() mutex or something similarly evil. (This evil is why platforms don't always support the construct.) I don't think this is feasible for portability reasons. In this case, other threads would not be scheduled before GC completes. * Use events This is my proposition: Have the garbage collector broadcast a STOP! event to every other parrot thread, and then wait for them all to rendezvous that they've stopped before GC proceeds. Hold a lock on the thread mutex while doing this, so threads won't start or finish. This has the worst latency characteristics of the three: All other threads would need to be scheduled before GC can begin. Corner cases exist: NCI calls could be long-running and shouldn't block parrot until completion; another thread might try to allocate memory before checking events. Neither is insurmountable. Gordon Henriksen [EMAIL PROTECTED]
RE: [PATCH] Unified PMC/PObj accessors phase 2
[* - Somewhat inadvisedly, I think. UnionVal is 8 bytes on a 32-bit architecture, but bloats to 16 bytes on a 64-bit architecture. That's likely so because of alignment. But real numbers would be better of course. Err? No, I'd think it's because the union contains two 16-byte structs (64-bit ptr + 64-bit ptr = 128-bit struct = 16 bytes). Shouldn't be any padding in UnionVal unless there's a 32-bit architecture out there that wants to align 32-bit values to 64- bit boundaries... -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
Re: [PATCH] Unified PMC/PObj accessors phase 2
Leopold Toetsch wrote: Gordon Henriksen wrote: The patch is at the URL below, and I've split it into 4 for you. The classes-include-lib patch must be applied before any of the other 3. I've resolved the 3-4 conflicts that occurred since the patch was first I've applied now pmc-accessors2-classes-include-lib. *But* 2) The *misc.patch doesn't compile in jit/i386 3) *src-a*.patch reverts Mike's docu patch Ack! Bad cvs update! No cookie! Not sure why those didn't merge. http://www.ma.iclub.com/pub/parrot/ now lists a .tgz with separate patches for each file. You can apply the patches in any order, or not at all; there are no interdependencies. Except!: include_parrot_pobj.h will remove the compatibility interfaces, so you may wish to sit on that for a month or so. -- Gordon Henriksen [EMAIL PROTECTED]
Patch vaporized?
I've submitted a patch to bugs-parrot, and it didn't seem to get posted to RT or otherwise handled. Anyone know where it might've gone? http://www.parrotcode.org/openpatches http://www.parrotcode.org/openpatches isn't working (ERROR RETRIEVING DATA). -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
RE: Patch vaporized?
Larry Wall wrote: Gordon Henriksen wrote: I've submitted a patch to bugs-parrot, and it didn't seem to get posted to RT or otherwise handled. Anyone know where it might've gone? Did it have an executable attachment? :-) Thanks, Larry, but no. :) It was a very lage patch, however; it converts pmc-cache.int_val etc. to PObj_int_val(pmc) etc. across the entire source tree. (Sorry, Leo, by far the easiest way to find them all was to change the field names and fix them as make found them; rather makes it difficult to create incremental patches.) So maybe sheer size was the problem. I'll post the file to my web server this evening and just include a link to it in YA resubmission, I suppose. How annoying... -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
[PATCH] Unified PMC/PObj accessors phase 2
Leo, The patch is at the URL below, and I've split it into 4 for you. The classes-include-lib patch must be applied before any of the other 3. I've resolved the 3-4 conflicts that occurred since the patch was first abortively submitted on Monday, so the old patch (named 20040202-pmc-accessors.patch) should be discarded if it resurfaces. http://www.ma.iclub.com/pub/parrot/ I realized that some of the accessor macros should've been named PObj_* instead of PMC_* (since they apply to STRINGs and Buffers, too). So they are as of this patch. I've also macro-ized access to UnionVal in general, since it was sometimes used outside of the context of a pobj. [*] The old syntax continues to work, and so nobody's patches will break excl. those w conflicts. But the pobj-cache.foo_val and PMC_ptr1p/PMC_ptr2v macros ought to be treated as deprecated. Gordon Henriksen [EMAIL PROTECTED] [* - Somewhat inadvisedly, I think. UnionVal is 8 bytes on a 32-bit architecture, but bloats to 16 bytes on a 64-bit architecture. The generic containers which use UnionVal don't appear to use both ptrs simultaneously or make use of the void*/int pair, so could use an 8-byte structure as their bucket type.]
Re: Alignment Issues with *ManagedStruct?
On Thursday, February 5, 2004, at 05:23 , Leopold Toetsch wrote:
Nested structs are ugly. The alignment of the first item seems to depend
on the biggest item in the struct.
[...]
Yeah, the struct is aligned to align the largest element therein. e.g.:
struct { char; struct { char; char; char; } }
is at +0, +1, +2, +3
But: struct { char; struct { char; char; short; } }
is at +0, +2, +3, +4
And: struct { char; struct { char; short; char; } }
is at +0, +2, +4, +6
So we need some notion of nested structs or a hint for correct
alignment. I've checked in basic alignment handling for normal
cases, but not the second one above.
Maybe you ought to capitulate to the hierarchical nature of it all and
simply push on another struct layout specifier to handle the nesting.
struct xt {
char x;
struct yt {
char i;
int j; // use different items here
} _y;
char z;
} _x;
int main(void) {
printf(x : %d\n, offsetof(struct xt, x));
printf(i : %d\n, offsetof(struct xt, _y.i));
printf(j : %d\n, offsetof(struct xt, _y.j));
printf(z : %d\n, offsetof(struct xt, z));
return 0;
}
Mac OS X output is:
x : 0
i : 4
j : 8
z : 12
Gordon Henriksen
[EMAIL PROTECTED]
Re: Patch vaporized?
On Thursday, February 5, 2004, at 11:35 , Robert Spier wrote: I think I've tracked this down, mostly. The patch was rejected from the mailing list because it was too big. (Several hundred k.) That's what I figured. You can find it in RT as 26056: [PATCH] Unify PMC accessor macros 2 26057: [PATCH] Unified PMC/PObj accessors phase 2 (26057 is one that points to a URL. Gordon, should I merge the two tickets?) Either merge them or close 26056 as wontfix or equivalent (sorry, I'm a Bugzilla feak :). The first patch has conflicts with current source, which is why I freaked out when it didn't show up pronto. Gordon Henriksen [EMAIL PROTECTED]
RE: More on threads
Dan Sugalski wrote:
Gordon Henriksen wrote:
Leopold Toetsch wrote:
Gordon Henriksen wrote:
... Best example: morph. morph must die.
Morph is necessary. But please note: morph changes the vtable of
the PMC to point to the new data types table. It has nothing to do
with a typed union.
The vtable IS the discriminator. I'm referring to this:
typedef union UnionVal {
struct {/* Buffers structure */
void * bufstart;
size_t buflen;
} b;
struct {/* PMC unionval members */
DPOINTER* _struct_val; /* two ptrs, both are
defines */
PMC* _pmc_val;
} ptrs;
INTVAL int_val;
FLOATVAL num_val;
struct parrot_string_t * string_val;
} UnionVal;
So long as the discriminator does not change, the union is
type stable.
The vtable's not the discriminator there, the flags in the pmc are
the discriminator, as they're what indicates that the union's a
GCable thing or not. I will admit, though, that looks *very*
different than it did when I put that stuff in originally. (It used
to be just a union of FLOATVAL, INTVAL, and string pointer...)
Hm. Well, both are a discriminator, then; dispatch to code which
presumes the contents of the union is quite frequently done without
examining the flags. Maybe use a VTABLE func instead to get certain
flags? i.e.,
INTVAL parrot_string_get_flags(..., PMC *pmc) {
return PMC_FLAG_IS_POBJ + ...;
}
Then, updating the vtable would atomically update the flags as well.
Or, hell, put the flags directly in the VTABLE if it's not necessary
for them to vary across instances.
I have the entire source tree (save src/ tests) scoured of that rat's
nest of macros for accessing PMC/PObj fields, but I broke something and
haven't had the motivation to track down what in the multi-thousand-
line-diff it was, yet. :( Else you'd have the patch already and plenty
of mobility in the layout of that struct. Near time to upgrade my poor
old G3, methinks; the build cycle kills me when I touch parrot/pobj.h.
Do any PMC classes use *both* struct_val *and* pmc_val concurrently? I
was looking for that, but am afraid I didn't actually notice.
--
Gordon Henriksen
IT Manager
ICLUBcentral Inc.
[EMAIL PROTECTED]
RE: More on threads
Leopold Toetsch wrote: Gordon Henriksen wrote: ... in the multi-thousand- line-diff it was, yet. :( Else you'd have the patch already 1) *no* multi-thousands line diffs 2) what is the problem, you like to solve? Er? Extending to the rest of the source tree the huge patch to classes which you already applied. No logic changes; just cleaning those PObj accessor macros up. -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
RE: More on threads
Leopold Toetsch wrote:
Gordon Henriksen wrote:
Or, hell, put the flags directly in the VTABLE if it's not
necessary for them to vary across instances.
No, flags are mutable and per PMC *not* per class.
Of course there are flags which must remain per-PMC. I wasn't
referring to them. Sorry if that wasn't clear.
If a flag is only saying my VTABLE methods use the UnionVal as {a
void*/a PObj*/a PMC*/data}, so GC should trace accordingly, it may be
a waste of a per-object flag bit to store those flags with the PMC
instance rather than with the PMC class. And if it's with the VTABLE,
then it doesn't need to be traced. (But, then, all PObjs don't have
VTABLES...)
Sidebar:
If we're looking at lock-free concurrency, flag updates probably have
to be performed with atomic 's and |'s. BUT: Doesn't apply during GC,
since other threads will have to be stalled then.
Do any PMC classes use *both* struct_val *and* pmc_val concurrently?
E.g. iterator.pmc. UnmanagedStruct uses int_val pmc_val. This is no
problem. These PMCs don't morph.
Er, int_val and pmc_val at the same time? That's not quite what the
layout
provides for:
typedef union UnionVal {
struct {/* Buffers structure */
void * bufstart;
size_t buflen;
} b;
struct {/* PMC unionval members */
DPOINTER* _struct_val; /* two ptrs, both are defines */
PMC* _pmc_val;
} ptrs;
INTVAL int_val;
FLOATVAL num_val;
struct parrot_string_t * string_val;
} UnionVal;
Says to me:
struct_val and pmc_val concurrently
-- or --
bufstart and buflen concurrently
-- or --
int_val
-- or --
num_val
-- or --
string_val
I don't know if C provides a guarantee that int_val and ptrs._pmc_val
won't
overlap just because INTVAL and DPOINTER* fields happen to be the same
size.
At least one optimizing compiler I know of, MrC/MrC++, would do some
struct
rearrangement when it felt like it.
--
Gordon Henriksen
IT Manager
ICLUBcentral Inc.
[EMAIL PROTECTED]
RE: Re[2]: Embedding vs. extending interface types
Dan Sugalski wrote: And pointless. Let's just rename it to Parrot_Interp everywhere. I've submitted a patch for this already. -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
Re: DOD, mutation, and generational collectors
On Thursday, January 29, 2004, at 08:41 , Dan Sugalski wrote: So, while I'm heaping much grumpiness on threads (though I suppose, as I've been out of touch for a bit maybe you've all solved the problem. That'd be nice) I've also been thinking about DOD, as there's a fair amount of overlap in the things that cause problems for threads and the ones that cause problems for generational garbage collectors. Can you elaborate a bit on your concept of generational collection as they apply to parrot? To my ear, generational collection implies all of this (which is a lot, and a lot of it grossly inapplicable to parrot as the runtime now stands): GENERATIONAL GARBAGE COLLECTORS Rationale: Most objects are short-lived, and those which are not are less likely to become garbage. Marking or copying the older is statistically wasted effort. A garbage collector which usually traces only new objects will thus tend to perform better than one which traces all of them. Implementation: Most generational collectors are copying collectors. They are generally mutations upon a hemispace collector. The variation is that collection always proceeds from a small, fixed-sized space to a larger, growable space, rather than swapping back and forth between two symmetric spaces. These asymmetric spaces are the so-called generations, and the small one containing the younger objects is dubbed the nursery. Moving an object from the nursery is called tenuring. Thus, the generation of an object is not so much a property of an object as it is a property of the pool from which the object is presently allocated. Some generational collectors have intermediate generations, but most do not. (Parrot might benefit from a younger-than-the-nursery generation for impatient destructors, but then that property would need to be ascertainable at construction time.) The nursery is typically explicitly un-fragmented. All live objects in it are tenured from it when it fills. Thus, allocation can be as simple as an atomic pointer increment and filling in an object header. The tenured generation might be a heap. To avoid tracing from the roots, the generational collector requires a cooperative mutator to maintain a table of references from tenured objects to objects within the nursery. This is often forced upon the mutator through the use of VM write barriers. The write barrier function marks the VM page as dirty and then removes write protection for that page. When invoked, the garbage collector will examine all dirtied pages, updating the table of references. A truly cooperative mutator will update a table of dirty pages every time it changes a pointer variable. Usually, the pages are smaller than VM pages, and thus called cards instead. This operation can be as simple as card_table[((int) ptr) n] = 1; ptr = obj;.[1] An expensive proposition, at least doubling the memory bandwidth overhead of pointer manipulation. But, since cards are smaller than pages, it can reduce the amount of memory which has to be scanned at collection time. Despite being incremental by nature, a generational collector is not a concurrent collector. That is, it still stops the world, it just doesn't do it for quite a long as a traditional hemispace or mark-sweep collector might. Being a copying collector in the traditional mold has its drawbacks. It requires accurate reference tracing, in particular. A traditional copying collector cannot work with a conservative collector. For instance, it cannot operate on the OS stack (unless the program is written very cooperatively), since the collector cannot ascertain for certain whether a value is a pointer (which should be traced and possibly updated) or data (which should be left alone). But parrot has anchored objects and other sorts of things which are incompatible with this. In particular, the fast-path nursery allocator (the very fast allocation being a major benefit of generational collectors) can't work: If an object were anchored in the nursery, it must stay there after GC rather than being tenured; this leaves the nursery fragmented and complicates the very hot allocation code path. Then there's the whole PMCs don't move guarantee. And there's also the conservative tracing of system areas. Gordon Henriksen [EMAIL PROTECTED] [1] To be that simple, all allocated memory must be contiguous. So it's not usually that simple. More likely, this: (((alloc_header*) obj) - 1)-pool.card_table[((int) obj) n]; obj-ptr = newvalue;
Re: Threads... last call
set of collections (incl. Vector). It was a major design flaw, crippling the performance of early Java programs. Fixing it for Java 2's class library was quite non-trivial, but because the VM was powerful enough to allow it, more efficient unsynchronized collections could be made available as a class library even for the earlier Java VMs. Gordon Henriksen [EMAIL PROTECTED]
Re: Messages delayed, out of order
On Thursday, January 29, 2004, at 09:12 , Matt Fowles wrote: I have been getting out of order messages from this list for months... I just assumed that the internet was a mysterious thing... Methinks the list is also manually moderated Gordon Henriksen [EMAIL PROTECTED]
Re: Threads... last call
On Thursday, January 29, 2004, at 11:55 , Melvin Smith wrote: At 11:45 PM 1/28/2004 -0500, Gordon Henriksen wrote: On Wednesday, January 28, 2004, at 12:53 , Melvin Smith wrote: At 12:27 PM 1/23/2004 -0800, Damien Neil wrote: Java Collections are a standard Java library of common data structures such as arrays and hashes. Collections are not synchronized; access involves no locks at all. Multiple threads accessing the same collection at the same time cannot, however, result in the virtual machine crashing. (They can result in data structure corruption, but this corruption is limited to surprising results rather than VM crash.) But this accomplishes nothing useful and still means the data structure is not re-entrant, nor is it corruption resistant, regardless of how we judge it. It does accomplish something very useful indeed: It avoids the overhead of automatic locking when it isn't necessary. When *is* that locking necessary? To a second order approximation, ***NEVER.*** Pardon me but I've apparently lost track of context here. I thought we were discussing correct behavior of a shared data structure, not general cases. Or maybe this is the general case and I should go read more backlog? :) A shared data structure, as per Dan's document? It's a somewhat novel approach, trying to avoid locking overhead with dynamic dispatch and vtable swizzling. I'm discussing somewhat more traditional technologies, which simply allow an object to perform equally correctly and with no differentiation between shared and unshared cases. In essence, I'm arguing that a shared case isn't necessary for some data structures in the first place. Gordon Henriksen [EMAIL PROTECTED]
Re: [perl #25253] [PATCH] Remove Parrot_INTERP
Seiler Thomas wrote: Gordon Henriksen wrote: The Parrot_INTERP type from embed.c and embed.h serves no purpose. [linking failures...] mem_alloc_executable mem_free_executable mem_realloc_executable [...] Re-ran Configure.pl and these went away, in case anyone else has this. inet_pton Is a IPv6 avare inet_aton. Attached is a Configure.pl test to determine if inet_pton is available and a small patch to switch to inet_aton if not. Works for cygwin and linux but no windows support yet. I would like to eventually include a fall-back implementation of inet_pton in order to prevent the ifdefs, but had not enough time-knowledge yet... Def. the right solution. (I'd just uncommented the usage of inet_aton out of laziness. :) Gordon Henriksen [EMAIL PROTECTED]
Re: [RESEND] Q: Array vs SArray
On Sunday, January 25, 2004, at 06:10 , Michael Scott wrote: On 25 Jan 2004, at 00:50, Gordon Henriksen wrote: [...] Is there something so terribly wrong with English? How about a general scheme of adjective* noun? So, respectively, MixedArray Array FixedArray StringArray FixedStringArray Array is what Perl familiars will usually want. Did I miss something? What is Array here? General scheme: Need something more specific? Type more. Not sure on Fixed, but I like it more than FLen. Other option is to flip it around and tag Resizable, but that violates my previous pricipal of most useful = most convenient. I don't think MixedArray will see much use, so FixedMixedArray doesn't worry me too much. :) I would definitely avoid the words mutable/immutable, as that will certainly be read by many (me :) to pertain to the values contained within the array. Yes, I'm already anti-me on that. It should have been Resizable. And though that's more precise than Fixed, I agree with your principle. So, to go back to Dan's original list, does that give us: (FixedMixedArray - fixed-size, mixed-type array) MixedArray - variable-sized, mixed-type array FixedPMCArray - fixed-size, PMC array PMCArray - variable-sized, PMC array I'd abbreviate PMC as , since it's certain to be the most commonly-used. OTOH, PMC isn't a very long identifier to type. FixedStringArray - fixed-size, string array StringArray - variable-sized, string array FixedNumberArray - fixed-size, number array StringArray - variable-sized, number array FixedIntegerArray - fixed-size, integer array IntegerArray - variable-sized, integer array with fixedarray - abstract fixed-size array array - abstract variable-sized array Perhaps base*. Or perhaps not. Gordon Henriksen [EMAIL PROTECTED]
Re: More on threads
Leopold Toetsch wrote: Gordon Henriksen wrote: I overstated when I said that morph must die. morph could live IF: [ long proposal ] Increasing the union size, so that each pointer is distinct is not an option. This imposes considerable overhead on a non-threaded program too, due its bigger PMC size. That was the brute-force approach, separating out all pointers. If the scalar hierarchy doesn't use all 4 of the pointers, then the bloat can be reduced. So long as a morph can't make a memory location which was formerly a pointer variable point to a new type, or reuse the memory location for data--then pointers can be part of a union without violating the principal of operation. And what of the per-PMC mutex? Is that not also considerable overhead? More than an unused field, even. To keep internal state consistent we have to LOCK shared PMCs, that's it. This locking is sometimes necessary for reading too. Sometimes? Unless parrot can prove a PMC is not shared, PMC locking is ALWAYS necessary for ALL accesses to ANY PMC. (This is easy to show; for any thread, hypothesize a preemption and morph at an inconvenient PC. Some morph will always put an int where a pointer was expected, or change the type of a pointer.) We seem to be painting ourselves into a corner here with impossible constraints. Clearly, adding size to PMCs is undesirable, and I recognize that you put a lot of work into shrinking the PMC struct over a Perl 5 scalar. But, fresh from CVS: BENCHMARK USER SYS %CPUTOTAL -- --- --- - addit.imc 23.53s 0.05s 96% 24.517 addit2.imc 15.97s 0.07s 98% 16.258 arriter.imc 0.03s 0.03s 37%0.159 arriter_o1.imc 0.02s 0.04s 37%0.160 fib.imc 4.41s 3.07s 97%7.691 addit.pasm 17.84s 0.05s 95% 18.827 bench_newp.pasm 4.53s 0.16s 97%4.824 freeze.pasm 1.47s 0.18s 82%1.991 gc_alloc_new.pasm0.20s 0.41s 72%0.836 gc_alloc_reuse.pasm 19.09s 13.15s 98% 32.731 gc_generations.pasm 14.02s 5.42s 95% 20.424 gc_header_new.pasm 7.91s 1.37s 97%9.558 gc_header_reuse.pasm11.45s 0.09s 98% 11.733 gc_waves_headers.pasm3.05s 0.40s 95%3.597 gc_waves_sizeable_data.pasm 2.22s 3.81s 97%6.200 gc_waves_sizeable_headers.pasm 7.48s 1.78s 97%9.480 hash-utf8.pasm 7.02s 0.07s 95%7.404 primes.pasm 61.63s 0.10s 97% 1:03.300 primes2.pasm39.26s 0.09s 97% 40.189 primes2_p.pasm 69.00s 0.17s 96% 1:11.840 stress.pasm 2.19s 0.28s 91%2.705 stress1.pasm46.95s 1.33s 95% 50.377 stress2.pasm 8.63s 0.24s 98%9.026 stress3.pasm26.19s 0.78s 96% 28.063 TOTAL7:21.890 And with a PMC struct that's bloated by 16 bytes as proposed: BENCHMARK USER SYS %CPUTOTAL -- --- --- - addit.imc 23.67s 0.10s 96% 24.745 addit2.imc 16.05s 0.07s 98% 16.414 arriter.imc 0.03s 0.03s 44%0.135 arriter_o1.imc 0.03s 0.04s 48%0.144 fib.imc 4.47s 2.99s 86%8.651 addit.pasm 18.08s 0.08s 98% 18.442 bench_newp.pasm 4.73s 0.18s 97%5.019 freeze.pasm 1.56s 0.29s 89%2.075 gc_alloc_new.pasm0.25s 0.35s 89%0.673 gc_alloc_reuse.pasm 18.90s 13.58s 96% 33.642 gc_generations.pasm 13.84s 5.80s 98% 19.942 gc_header_new.pasm 7.98s 1.29s 97%9.492 gc_header_reuse.pasm11.37s 0.05s 98% 11.552 gc_waves_headers.pasm3.09s 0.33s 96%3.538 gc_waves_sizeable_data.pasm 1.98s 4.01s 96%6.207 gc_waves_sizeable_headers.pasm 7.67s 1.60s 91% 10.112 hash-utf8.pasm 7.03s 0.05s 97%7.287 primes.pasm 61.68s 0.15s 98% 1:02.800 primes2.pasm39.17s 0.12s 99% 39.553 primes2_p.pasm 69.01s 0.18s 96% 1:11.640 stress.pasm 2.32s 0.43s 96%2.840 stress1.pasm50.85s 1.79s 93% 56.189 stress2.pasm 9.18s 0.32s 98%9.689 stress3.pasm29.06s 1.23s 98% 30.748 TOTAL7:31.527 That's only 2.1%. What do you think the overall performance effect of fine-grained locking
Re: Embedding vs. extending interface types
On Saturday, January 24, 2004, at 02:28 , Leopold Toetsch wrote:
Mattia Barbon [EMAIL PROTECTED] wrote:
I feel I'm becoming annoying, but: the embedding and extending
interfaces are still using different names for
Parrot_Interp/Parrot_INTERP. Which one is correct?
AFAIK both. Embedding and extending are to different APIs. The former
has access to internals the latter doesn't. So they get different types
for e.g. the interpreter.
All embedders see is this:
/* include/parrot/interpreter.h */
struct Parrot_Interp;
typedef struct Parrot_Interp *Parrot_Interp;
i.e., struct Parrot_Interp is declared, but never defined. So the extra
Parrot_INTERP type is just cruft. Also, it's not used anywhere in the
source tree except in extend.h and extend.c.
And does anyone ever use the Interp type from below?
/* include/parrot/interpreter.h */
typedef struct Parrot_Interp {
...
} Interp;
Gordon Henriksen
[EMAIL PROTECTED]
Re: Embedding vs. extending interface types
On Sunday, January 25, 2004, at 03:44 , Leopold Toetsch wrote: Gordon Henriksen [EMAIL PROTECTED] wrote: All embedders see is this: typedef struct Parrot_Interp *Parrot_Interp; I don't do decisions on embedding or extending interfaces. But it seems to be the time to decide (and clean/unite) [...] Speaking of cleaning and uniting, what is with this? #define bufstart obj.u.b.bufstart #define buflen obj.u.b.buflen #if ! DISABLE_GC_DEBUG # define pobj_version obj.pobj_version #endif #define struct_val ptrs._struct_val #define pmc_val ptrs._pmc_val #define cache obj.u #define metadata pmc_ext-metadata #define next_for_GC pmc_ext-next_for_GC #define synchronize pmc_ext-synchronize # define PMC_data(pmc) (pmc)-pmc_ext-data #else # define PMC_data(pmc) (pmc)-data #endif Gordon Henriksen [EMAIL PROTECTED]
Re: Benchmark Suite
On Sunday, January 25, 2004, at 06:01 , Matt Fowles wrote: Of late it seems that everybody has been throwing around their own little homegrown benchmarks to support their points. But many people frequently point out that these benchmarks are flawed on one way or another. I suggest that we add a benchmark/ subdirectory and create a canonical suite of benchmarks that exercise things well (and hopefully fully). Then we can all post relative times for runs on this benchmark suite, and we will know exactly what is being tested and how valid it is. Well, there's already examples/benchmarks. If those programs are not at all realistic, then more realistic benchmarks should be added. Would be nice if there were a convenient way to run the lot of them and collect the timing information, though. Gordon Henriksen [EMAIL PROTECTED]
Re: t/src/io failure
On Sunday, January 25, 2004, at 06:42 , Michael Scott wrote: I see that t/src/io is now failing on OS X 10.3.2. Is anyone else seeing this on another system? Also on 10.1.5. Gordon Henriksen [EMAIL PROTECTED]
Re: Embedding vs. extending interface types
On Sunday, January 25, 2004, at 07:08 , Leopold Toetsch wrote: Gordon Henriksen [EMAIL PROTECTED] wrote: Speaking of cleaning and uniting, what is with this? #define bufstart obj.u.b.bufstart #define buflen obj.u.b.buflen These are *currently* necessary macros, until the PMC/PObj layout is really carved in stones. You had in your proposal a different PMC structure layout. Testing such things is vastly simplified by having accessor macros. Yes, I know. ;) I was going to test a real layout change, but the macros were so tangled up that I wound up just appending 4 ints to struct PMC to simulate. I'm not a friend of such macros. But they are needed as long as the actual structure might change. I did introduce all pobj.h related macros. They can go away, when *all* issues are resolved. I know what they are, and have no problem with them. I don't think they should be removed. But this particular batch are a mess. I just submitted a couple of patches to clean them up a little, though I didn't get through the entire source tree yet. Gordon Henriksen [EMAIL PROTECTED]
Re: More on threads
On Saturday, January 24, 2004, at 09:23 , Leopold Toetsch wrote:
Gordon Henriksen [EMAIL PROTECTED] wrote:
... Best example: morph. morph must die.
Morph is necessary. But please note: morph changes the vtable of the
PMC to point to the new data types table. It has nothing to do with a
typed union.
The vtable IS the discriminator. I'm referring to this:
typedef union UnionVal {
struct {/* Buffers structure */
void * bufstart;
size_t buflen;
} b;
struct {/* PMC unionval members */
DPOINTER* _struct_val; /* two ptrs, both are defines */
PMC* _pmc_val;
} ptrs;
INTVAL int_val;
FLOATVAL num_val;
struct parrot_string_t * string_val;
} UnionVal;
So long as the discriminator does not change, the union is type stable.
When the discriminator does change, as per here:
void
Parrot_PerlInt_set_string_native(Parrot_Interp interpreter, PMC*
pmc, STRING* value)
{
VTABLE_morph(interpreter, pmc, enum_class_PerlString);
VTABLE_set_string_native(interpreter, pmc, value);
}
void
Parrot_perlscalar_morph(Parrot_Interp interpreter, PMC* pmc, INTVAL
type)
{
if (pmc-vtable-base_type == enum_class_PerlString) {
if (type == enum_class_PerlString)
return;
PObj_custom_mark_CLEAR(pmc);
pmc-vtable = Parrot_base_vtables[type];
return;
}
if (type == enum_class_PerlString) {
pmc-vtable = Parrot_base_vtables[type];
VTABLE_init(interpreter, pmc);
return;
}
PObj_custom_mark_CLEAR(pmc);
pmc-vtable = Parrot_base_vtables[type];
}
... then these can both run:
Parrot_scalar_get_string(Parrot_Interp interpreter, PMC* pmc)
{
return (STRING*)pmc-cache.string_val;
}
FLOATVAL
Parrot_scalar_get_number(Parrot_Interp interpreter, PMC* pmc)
{
return pmc-cache.num_val;
}
That clearly allows a struct parrot_string_t * to freely share the same
memory as a double. Were it an int and a double, the surprising
results from this unprotected access wouldn't violate the no crashes
guarantee. But it's a pointer! Dereferencing it could cause a segfault,
or a read or write of an arbitrary memory location. Both clearly violate
the crucial guarantee.
Gordon Henriksen
[EMAIL PROTECTED]
Re: More on threads
Leopold Toetsch wrote:
Gordon Henriksen [EMAIL PROTECTED] wrote:
... Best example: morph. morph must die.
Morph is necessary. But please note: morph changes the vtable of the
PMC to point to the new data types table. It has nothing to do with a
typed union.
I overstated when I said that morph must die. morph could live IF:
the UnionVal struct were rearranged
bounds ere placed upon how far a morph could... well, morph
It doesn't matter if an int field could read half of a double or v.v.;
it won't crash the program. Only pointers matter.
To allow PMC classes to guarantee segfault-free operation, morph and
cooperating PMC classes must conform to the following rule. Other
classes would require locking.
With this vocabulary:
variable: A memory location which is reachable (i.e., not garbage). [*]
pointer: The address of a variable.
pointer variable: A variable which contains a pointer.
access: For a pointer p, any dereference of p*p, p-field, or
p[i]whether for the purposes of reading or writing to that variable.
And considering:
any specific pointer variable (ptr), and
all accesses which parrot might perform[**] on any pointer ever
stored in ptr (A) [***], and
any proposed assignment to ptr
Then:
If any A which once accessed a pointer variable would now access a
non-pointer variable,
Then the proposed assignment MUST NOT be performed.
This is a relaxed type-stabilitydefinition. (Relaxed: It provides type
stability only for pointer variables, not for data variables. It does
not discriminate the types of pointers, only that the data structures
they directly reference have the same layout of pointers. Also, a
loophole allows non-pointer variables to become pointer variables, but
not the reverse.)
These rules ensure that dereferencing a pointer will not segfault. They
also ensure that it is safe to deference a pointer obtained from a union
according to the union's discriminatorregardless of when or in which
order or how often parrot read the pointer or the discriminator.[***] I
think they're actually the loosest possible set of rules to do this.
[*] Two union members are the same variable.
[**] This is in the variable ptr specifically, not merely in the same
field of a similar struct. That is, having an immutable discriminator
which selects s.u.v or s.u.i from struct { union { void * v; int i; }
u } s is valid. A mutable discriminator is also validso long as the
interpretation of pointer fields does not change.
[***] But only if the architecture prevents shearing in pointer reads
and writes.
From another perspective this is to say:
Every pointer variable must forever remain a pointer.
Union discriminators must not change such that a pointer will no longer
be treated as a pointer, or will be treated as a pointer to a structure
with a different layout.
The first step in conforming to these rules is guaranteeing that a
perlscalar couldn't morph into an intlist or some other complete
nonsense. So the default for PMCs should be to prohibit morphing. Also,
morphable classes will have a hard time using struct_val without
violating the above rules. But for this price, parrot could get
lock-free, guaranteed crash-proof readers for common data types. But
note that pmc-cache.pmc_val can be used freely! So if exotic scalars
wrap their data structures in a PMC
*cough*perlobject*cough*managedstruct*ahem*, then those PMCs can be part
of a cluster of morphable PMC classes without violating these rules.
Next, the scalar hierarchy (where morphing strikes me as most important)
could be adjusted to provide the requisite guarantees, such as:
perlstring's vtable methods would never look for its struct
parrot_string_t * in the same memory location that a perlnum vtable
method might be storing half of a floatval. Right now, that sort of
guarantee is not made, and so ALL shared PMCs REALLY DO require locking.
That's bad, and it's solvable.
Specifically, UnionVal with its present set of fields, would have to
become something more like this:
struct UnionVal {
struct parrot_string_t * string_val;
DPOINTER* struct_val;
PMC* pmc_val;
void *b_bufstart;
union {
INTVAL _int_val;
size_t _buflen;
FLOATVAL _num_val;
} _data_vals;
};
If no scalar types use struct_val or pmc_val or b_bufstart, then those
fields can go inside the union.
Unconstrained morphing is the only technology that *in all cases*
*completely* prevents the crash-proof guarantee for lock-free access to
shared PMCs. Without changes to this, we're stuck with implicit PMC
locking and what looks like an unusable threading implementation.
This is only the beginning! For example, if parrot can povide type
stability, mutable strings can be crash-proofed from multithreaded
access. Wha?!
1. Add to the buffer structure an immutable
Re: More on threads
I wrote: With this vocabulary: variable: A memory location which is reachable (i.e., not garbage). [*] pointer: The address of a variable. pointer variable: A variable which contains a pointer. access: For a pointer p, any dereference of p*p, p-field, or p[i]whether for the purposes of reading or writing to that variable. And considering: any specific pointer variable (ptr), and all accesses which parrot might perform[**] on any pointer ever stored in ptr (A) [***], and any proposed assignment to ptr Then: If any A which once accessed a pointer variable would now access a non-pointer variable, Then the proposed assignment MUST NOT be performed. D'oh. This actually has to be recursive. Considering: any specific pointer variable (ptr'), and all accesses which parrot might perform on any pointer ever stored in ptr, and all accesses which parrot might perform on any pointer ever stored in those variables, ..., ... A, and any proposed assignment to ptr else it allows char * a = ...; char ** b = a; Doesn't change the conclusions I drew at all. (Nor does it require some massively recursive algorithm to run at pointer assignment time, just as the first one didn't require anything more than pointer assignment at pointer assignment time.) Could probably be simplified with the addition of pointer type to the definitions section. Anyhoo. Gordon Henriksen [EMAIL PROTECTED]
Re: [RESEND] Q: Array vs SArray
On Friday, January 23, 2004, at 11:05 , Tim Bunce wrote: Here's my preference: *) ArrayFLenMixed - fixed-size, mixed-type array *) ArrayVLenPMC- variable-sized PMC array *) ArrayFLenPMC- fixed-size PMC array *) ArrayVLenString - variable-sized string array *) ArrayFLenString - fixed-size string array (Of course VLen/FLen could be VSize/FSize if preferred, and Mixed seemed better than Any as I recall it's not truly any type.) The general scheme is container typequalifierscontained type Is there something so terribly wrong with English? How about a general scheme of adjective* noun? So, respectively, MixedArray Array FixedArray StringArray FixedStringArray Array is what Perl familiars will usually want. General scheme: Need something more specific? Type more. Not sure on Fixed, but I like it more than FLen. Other option is to flip it around and tag Resizable, but that violates my previous pricipal of most useful = most convenient. I don't think MixedArray will see much use, so FixedMixedArray doesn't worry me too much. :) I would definitely avoid the words mutable/immutable, as that will certainly be read by many (me :) to pertain to the values contained within the array. Gordon Henriksen [EMAIL PROTECTED]
Re: Embedding vs. extending interface types
On Saturday, January 24, 2004, at 11:28 , Mattia Barbon wrote: I feel I'm becoming annoying, but: the embedding and extending interfaces are still using different names for Parrot_Interp/Parrot_INTERP. Which one is correct? Mattia, Both are correct. Sort of. :) Parrot_INTERP is an opaque type, which is a technique for improving binary compatibility. In the core, which is always compiled as a unit, the fields of the interpreter structure can be accessed directly. But to preserve binary compatibility if the struct layout is modified, embedders and extensions (which are built separately from the core) must use accessor functions, and in fact do not generally even have the struct definition. Thus, in parrot/extend.h, For non-core users: typedef void * Parrot_INTERP in non-core users For core users: typedef struct Parrot_Interp * Parrot_INTERP for core users Looks like Parrot_INTERP is the best name to use in general, though. This is rather on the unclear side. I would have clarified this by using private or opaque on the internal name for the struct, or prefixing with an underscore to indicate a private identifierrather than differentiating just by capitalization. ... okay, I'm beginning to agree with you. There's another name, Interp, for the same structure, defined in parrot/interpreter.h. That, and struct Parrot_Interp is a struct while Parrot_Interp is a pointer. Gordon Henriksen [EMAIL PROTECTED]
Re: Embedding vs. extending interface types
I wrote: Mattia Barbon wrote: I feel I'm becoming annoying, but: the embedding and extending interfaces are still using different names for Parrot_Interp/Parrot_INTERP. Which one is correct? [blahblahblah] Spoke too soon. Parrot_INTERP looks unnecessary. Parrot_Interp already has the needed opacity guards in place. Gordon Henriksen [EMAIL PROTECTED]
Re: More on threads
Pete Lomax wrote:
Gordon Henriksen wrote:
snip
It doesn't matter if an int field could read half of a double or v.v.;
it won't crash the program. Only pointers matter.
snip
These rules ensure that dereferencing a pointer will not segfault.
In this model, wouldn't catching the segfault and retrying (once or
twice) work?
Determining how to retry in the general case would be... much more
interesting than this proposal. :)
Furthermore, worse than segfaults could potentially result from using
half of a double as a pointer. There's no assurance that *((caddr_t *)
double) won't in fact be a valid memory address. In this case, there
would be no segfault, but memory would be subtly corrupted. There's no
way to detect that, so there's no way to retry.
The point of all the tweaks and care is to prevent ever dropping
something else in a particular variable where parrot would, at another
point in the program, expect a pointer of a particular type.
I think you probably got the following, but I'd just like to elaborate
more specifically. I think the greatest subtlety of the rules was in the
interpretation of
accesses which parrot might perform
and the word specific in
any specific pointer variable ptr
Without understanding precisely what I meant there, one might think that
even a simple polymorphic system like the following is prohibited:
struct eg;
typedef int (func_ptr*)(struct eg*);
struct {
func_ptr fp;
union { int *pointer; int integer; } u;
} eg;
int pointer_meth(struct eg *thiz) { return ++*(thiz-u.pointer); }
int integer_meth(struct eg *thiz) { return ++(thiz-u.integer); }
void main() {
struct eg eg1 = { pointer_meth, NULL };
struct eg eg2 = { integer_meth, 1 };
eg1.u.pointer = malloc(sizeof(int));
*eg1.u.pointer = 0;
print_it(eg1, eg1);
print_it(eg2, eg2);
}
void print_it(char *name, struct eg *some_eg) {
printf(%s says %d\n, some_egfp(eg2));
}
But the program IS allowed. While print_it might behave in any number of
ways depending on some_eg-fp, it will always access a particular eg.u
in a consistent fashion, since it always respects the discriminator
(eg.fp), which the program never changes. By extension of this, C++
instances do not violate these rules, either.[*] Were the following line
added to main(), though, then the program would be in violation:
eg1.fp = integer_meth;
Because now some other thread could have obvserved eg1.fp ==
pointer_meth and begun invoking pointer_meth. pointer_meth might now
access u.pointer and, instead of a pointer, see n + (int) u.pointer.
That probably won't segfault for small values of n, but will certainly
not do the right thing either.
This is trite tangent, but also note that the type stability rule
prohibits this:
eg1.u.pointer = NULL;
But would not if the definition of pointer_meth became:
int pointer_meth(struct eg *thiz) {
int* pointer = this-u.pointer;
return pointer == NULL? -1 : ++*pointer;
}
Because now the program will now not dereference u.pointer if its value
is NULL. How cute. (But if u.pointer were not copied to a local, then
bets are off again, because C might perform an extra load and get a
value inconsistent with the one it received when testing for NULL.)
... But even that extra local copy isn't required if u.pointer begins
NULL, and can become non-NULL, but will not become NULL again. Why?
all accesses which parrot MIGHT perform on any pointer ever
storED in ptr (A)
Note the past tense there. That's why.
If I'm reading you correctly, which is unlikely,
That has little to do with you, but much to do with my burying important
parts of my message in pages of dense text. :)
this has little to do with program correctness, but about the
interpreter not crashing because of an unfortunate context switch..
which the programmer should have guarded against in the first place...
Yes. Precisely.
no, I think I just lost the plot again ;-)
I think you're pretty close, just missing a few of the subtleties that
got buried in that long missive. :)
Gordon Henriksen
[EMAIL PROTECTED]
[*] Even though C++ changes the vtable of an instance during
instantiation, it does so in a broadening fashion, making
formerly-inaccessible variables accessible. A C++ instance of class
derived_class : public base_class is not a derived_class until
base_class's constructor finishes and derived_class's constructor
begins. (Side-effect: A subclass cannot influence the instantiation
behavior of a base class.) (Objective C's class methods are wildly
useful. Static languages tend to ignore them. It's sad.)
Re: [DOCS] C code documentation
Dan Sugalski wrote: Michael Scott wrote: Perhaps the most controversial feature of all this is that I'm using rows of 80 '#'s as visual delimiters to distinguish documentation sections from code. Please don't. If you really, really must, chop it down to 60 or so characters. 80 may wrap in some editors under some situations, and anything bigger than 72 may trigger wrapping behaviour in mail programs. (I think I'd as soon you didn't put in delimiters like this at all, but I can live with it if I must) And more than 71 and messages containing patch snippets will wrap in some mail programs. And more than 69 and replies to messages containing patch snippets will wrap in some mail programs. ad infinitum... Gordon Henriksen [EMAIL PROTECTED]
RE: Threads... last call
Deven T. Corzine wrote: The most novel approach I've seen is the one taken by Project UDI (Uniform Driver Interface). This is very much the ithreads model which has been discussed. The problem is that, from a functional perspective, it's not so much threading as it is forking. -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
Re: Signals and Events
On Friday, January 23, 2004, at 09:39 , Dan Sugalski wrote: At 12:05 PM +0100 1/23/04, Leopold Toetsch wrote: So my local src/events.c does: 1) block all signals before any thread creation 2) install an event handler for SIGINT 3) start the event handler thread 4) start an IO handler thread, which does: - unblock SIGINT - select in a while loop - if select returns -1, check for EINTR and the sig_atomic_t flag set by the signal handler All this stuff needs to get out of events.c at some point, and we probably better do it now rather than later. Not because they're not events, but because this behaviour is all very platform-dependent, and so ought to go in the platform-local source instead. (It's not even Unix-dependent, as all the different unix flavors handle signals and threads differently) What we need to do is get an abstraction in place for this and write platform-specific code to get concrete implementations of that abstraction in place. Which also means we need to have more complex processing of the platform source files. It may even be important enough to probe this stuff at runtime and select the appropriate implementation then, so that a single parrot binary can be compatible across different versions of the same operating system. For instance: Linux flavors with per-thread PIDs, or different versions of Mac OS X with evolving pthreads implementations (or with and without poll()). Or Windows 98 vs. Server 2003. Finally, if the most performant strategy might differ for uniprocessors and multiprocessors. Gordon Henriksen [EMAIL PROTECTED]
Re: Start of thread proposal
On Friday, January 23, 2004, at 11:09 , Dan Sugalski wrote: Ah, OK, I see. The problem comes in where we've got an object in the transient root set (basically the processor stack and registers) that gets anchored into the base root set (stash, pads, or whatever) after the DOD has traced where it's going into and falls out of the transient root set before the DOD traces over to it. (Worse than that. It could come from any untraced locationor possibly even be brand new, depending upon memory allocation details.) Gordon Henriksen [EMAIL PROTECTED]
More on threads
PMCs (deadlock avoidance!) - invoke* - an upwards branch - reassigning a PMC register (changing reg[n], not reg[n]-something) - any blocking operation - anything that checks for events Bytecode validation can ensure that PMC locks are held before operating on a PMC that is: - not a temporary - not known to be a threadsafe PMC (I repeat: morph is evil) Pain in the ass? Maybe not. Here, I think IMCC can pull most of the weight: lock* shouldn't even be available through IMCC. Since it's just a matter of chugging through the rules, IMCC can apply locking automatically. (Hell, to avoid changing PBC, parrot could implement them automatically at runtime! Wouldn't want to be there, but the above set of rules is darned easy to apply.) However, I would again repeat that the value provided to user programs by atomic PMC operations is negligible (since a user's data structures will typically be higher-level, and thus require user-level locking)and is certainly not offset by a 400% performance penalty. Locking PMCs is the easy way out. If type-stable memory can be provided (morph must die), it may be possible to look away from automatic PMC locks in more cases. For instance, an Int PMC would require no locks whatsoeverso long as its type cannot change. Nor would an immutable string, or a fixed-size array. On platforms with 8-word atomic writes, a Number PMC could get by without locks, too. morph must die. Dan Sugalski wrote: Okay, we're going to mandate that we have read/write locks, each interpreter pool has one, and mutating vtable entries must get a read lock on the pool read/write lock. Pricey (ick) but isolated to mutators. The DOD gets a write lock on it, which'll block all read/write access so no mutators can be in process while the pool DOD runs. Adding a reader-writer lock on every pointer mutation will be completely and utterly unfeasible from a performance standpoint. There will be massive reader contention even on uniprocessors, and the bus pollution on multiprocessors will be disastrous. Reader-writer locks are more costly than mutexes. Can I suggest an algorithm to completely eliminate that reader-writer lock, replacing it with a form of coordination that's cheaper in aggregate, and free in the common case? It's not simple, but I'd rather one elephant than a billion mice. Might help with infant mortality, too. Gordon Henriksen [EMAIL PROTECTED] P.S. - morph must die.
Re: Start of thread proposal
On Monday, January 19, 2004, at 07:58 , [EMAIL PROTECTED] wrote: Is there any likelyhood that memory allocation will be hidden behind macros at two levels: - ALLOCPOOL() for allocating large chunks of memory (ppols) that are later sub-allocated (and managed) by: - SUBALLOC() for sub allocating within a pool Are you wanting something akin to Apache 2 pools, which are hierarchical and designed to reduce path length when freeing blocks of objects? For instance, all objects and sub-pools allocated during an HTTP request cycle can be deallocated just by free()'ing the top-level request pool. I don't think parrot could make use of that model, since it can't very well guarantee that the user cannot retain references past the lifetime of the pool. Apache trusts modules not to make such errors; parrot can't trust the byte-code it's executing any further than it can throw it. A generational collector is a more likely means by which parrot might reduce memory-related overhead. Gordon Henriksen [EMAIL PROTECTED]
RE: Shared vs non shared PMC timings
Leopold Toetsch wrote: (Overall timings aren't really comparable, the SharedRef also does a LOCK for mark, which slows that down as well) ?? Why'd you do that? Competetive threads MUST be suspended (most likely with their cooperation, not with an OS suspend call) during the mark phase. -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
Re: Start of thread proposal
On Tuesday, January 20, 2004, at 07:56 , [EMAIL PROTECTED]
wrote:
I believe that parrot already has the concept of memory pools in it's
memory management. The idea is that by allocating similarly sized
objects from separate (large) allocations, you can reduce the
fragmentation of chunks and reduce the incidences where the memory need
to be GC'd and compacted.
Don't presume that. General purpose memory allocators tend to handle
this scenario... generically. Use of custom allocators is usually a
premature optimization; the best generic memory allocators are very,
very good.
But the ultimate effect of the overuse of custom allocators or pools is
to move memory fragmentation to a higher level, preventing the generic
allocator from addressing it even when the memory allocation patterns
would otherwise allow it.
If the allocation of pools, and the allocation of bit-of-a-pool, are
macroised, it makes it possible for OS's where there are multiple APIs
for memory allocation to bypass the CRT memory allocation routines and
use which ever native APis are best suited for the type of allocation.
Would much rather see them abstracted, as they are now, beneath an API.
This reduces the number of ways in which compiled parrot extensions can
be incompatible with the parrot core.
Personally, I would like to see memory allocation for each class type
be managed by the class constructor itself. This would theoretically
allow each class that has a fixed instance size to manage it's own pool
on OS's where that makes sense. The class would allocate a pool for
itself when loaded and then allocate instances from that pool on new()
and deallocate upon DESTROY. If it's memory pool was exhausted when new
was called, it would invoke the GC on *it's pool only*.
A PMC class is free to do precisely that. Only the PMC header cannot be
thus managed, and that's already pooled.
This separation would mean that each run of the GC would have a much
smaller pool of memory to compact and garbage collect when it was
invoked.
This is false. The mark phase will still need to run over the entire
process, else it cannot detect all references into the pool. (Unless you
can provide a type-safety guarantee that your type can only be
referenced by other instances of itself. In which case, your type is
necessarily garbage and the best optimization strategy would be
dead-code elimination. :)
I predict the best overall throughput would be with the sweep or copy
phase run immediately after the mark phase, across the entire process:
It would be wasteful to do all that marking and yet leave known garbage
uncollected.
It would also be less likely to be called, as each allocation from a
pool of fixed sized sub allocations will only ever need to call the GC
when it's pool is entirely exhausted.
Statistically, multiple pools will individually exhaust themselves
*MORE* frequently than a global pool. The ideal case for collection
frequency is that there is only one pool, or that all pools rise to
capacity concurrently. In these ideal cases, the segregated pools will
exhaust themselves precisely *AS* frequently as a global pool. In any
case, there is no possibility for a decrease in collection events
through the use of pooled memory. As per above, collection events will
also not become less expensive. Thus, expect garbage collection to have
a negative net impact on performance if pooled memory is used.
But that is a radical departure :), so if would just like to see
separate calls for pool allocation/reallocation and element
allocation/reallocation, rather than having calls to malloc() scattered
through out the codebase.
Uh. See memory.c and smallobject.c...
Digression:
Most copying collectors avoid free space fragmentation by design.
(That's one of their benefits.) An allocator as might be employed by a
standard compacting, generational system (Java, MS CLR), might resemble
the following:
#define SYS_ALIGN sizeof(double) /* for instance */
struct generation {
volatile void *next;
void *top;
void *base;
}
struct generation *generation_init(size_t capacity) {
struct generation *g = (struct generation *) malloc(sizeof(struct
generation));
g-next = g-base = malloc(capacity);
g-top = g-base + capacity;
}
void* generation_alloc(struct generation* p, size_t len) {
len = (len + SYS_ALIGN - 1) ~(SYS_ALIGN - 1); /* round up */
do {
void *ptr = ATOMIC_ADD(p-next, len);
if (ptr = p-top) {
return ptr - len; /* break the loop */
}
gc();
} while (true); /* try again */
}
+ No free space fragmentation at all.
+ Super-duper fast.
+ Threadsafe.
+ Lock-free unless ATOMIC_ADD cannot be implemented without a mutex.
- There is copying overhead when the generation is exhausted.
- One could say that the generation is fragmented by garbage.
+ It is no more fragmented by garbage than a GC system which uses a
freelist allocator.
Gordon Henriksen
RE: Start of thread proposal
Dan Sugalski wrote: [A] copying collector is generally untenable in a threaded environment. Can you elaborate upon the reasoning behind this statement? The first thing that any vtable function of a shared PMC must do is to aquire the mutex of the PMCs in its parameter list, in ascending address order. When the mutexes are released they are not required to be released in any order. Note that this locking strategy cannot avoid deadlock if the user is allowed to acquire these locks--HLL locks must be altogether different beasts from automatic PMC locks. That's okay. Just a design consequence worth noting for everyone. -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
RE: Start of thread proposal
Dan Sugalski wrote: For a copying collector to work, all the mutators must be blocked, and arguably all readers should be blocked as well. True of non-moving collectors, too. Or, let me put it this way: non- copying *GC* (the sweep or copy phase) can be threadsafe, but the mark phase is never threadsafe. The method in which marking is not threadsafe is a bit more pathological (i.e., it's not the common case as it is with the copying collector), but a standard tracing DOD cannot be correct when competeting with mutators. It WILL collect non- garbage (those are MUTATORS, remember), and the result WILL be Heizenbugs and crashes. Some of what I've written up addresses why. It's pretty simple to demonstrate a single case to prove the point, but I don't feel like re-creating the ASCII art right now. :) I'll send that section when I get out of the office. parrot will have to be able to suspend all threads in the environment. Unfortunate, but really quite unavoidable. Oh, arguably it can't avoid deadlock at all, what with vtable methods having access to the full environment. I can live with deadlocks, only because there's no real alternative. But PMC implementations have to fall inside of the trusted environment, so that's not really a failure. Of course uncooperative code can break a cooperative algorithm. :) -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
Re: Start of thread proposal
On Monday, January 19, 2004, at 06:37 , Gordon Henriksen wrote:
Dan Sugalski wrote:
For a copying collector to work, all the mutators must be blocked,
and arguably all readers should be blocked as well.
True of non-moving collectors, too. [...]
Some of what I've written up addresses why. [...] I'll send that
section when I get out of the office.
Consider this simple object graph:
Root set = { A, B }
[ A=NULL]
[ B=C ] --- [ C=]
Collection begins in thread 1 and marks A as reachable before being
preempted by thread 2:
[xA=NULL]
[ B=C ] --- [ C=]
Thread 2 sets A to C, and nullifies B before being preempted again by
thread 1:
[xA=C ] --- [ C=]
[ B=NULL]
Thread 1 marks B as reachable:
[xA=C ] --- [ C=]
[xB=NULL]
The mark phase complete, thread 1 sweeps:
[ A=???] --- ???
[ B=NULL]
C was not marked reachable (although it was) and was thus erroneously
collected, leaving a dangling pointer. This problem applies equally to
copying and mark-sweep collectors.
Gordon Henriksen
[EMAIL PROTECTED]
Re: Events and JIT
On Saturday, January 17, 2004, at 05:53 , Leopold Toetsch wrote: Gordon Henriksen [EMAIL PROTECTED] wrote: Other threads than the target could be executing the same chunk of JITted code at the same time. No. JITed (and prederefed) code is thread-specific, because register addresses are converted to absolute memory locations. Ack! Might want that to be configurable for the JIT, to reduce the overhead of starting a thread... Gordon Henriksen [EMAIL PROTECTED]
Re: Events and JIT
On Saturday, January 17, 2004, at 05:51 , Leopold Toetsch wrote: Jeff Clites [EMAIL PROTECTED] wrote: ... Also, once the handler is entered we'd have to fix all of patched locations, which again could be time-consuming for large bytecode. Please remember: only the current code segment has to be patched, which theoretically could be very large, but that's unlikely. What if control leaves the current code segment before you finish patching it? If an event is being delivered from another thread, this is very likely to occur if the event delivery thread is preempted by the event receiver. Gordon Henriksen [EMAIL PROTECTED]
Re: cygwin link failure
On Friday, January 16, 2004, at 03:34 , Seiler Thomas wrote: inet_pton has not yet been implemented in cygwin, but it is being worked on... http://win6.jp/Cygwin/ Indeed, but I think there might be other unix-like environments that (do not yet|will never) provide the inet_pton function. Mac OS X 10.1 does not. Gordon Henriksen [EMAIL PROTECTED]
Re: Events and JIT
On Saturday, January 17, 2004, at 12:47 , Leopold Toetsch wrote: Gordon Henriksen [EMAIL PROTECTED] wrote: What if control leaves the current code segment before you finish patching it? Ops that can leave the code segment have to explicitely check for events. Then no need to patch invoke*. If the target thread handles the event and unpatches the bytecode before the source thread finishes patching? Gordon Henriksen [EMAIL PROTECTED]
RE: Events and JIT
Leopold Toetsch wrote: Event handling currently works for all run cores[1] except JIT. The JIT core can't use the schemes described below, but we could: 2) Patch the native opcodes at these places with e.g. int3 (SIG_TRAP, debugger hook) cpu instruction and catch the trap. Running the event handler (sub) from there should be safe, as we are in a consistent state in the run loop. I don't think that bytecode-modifying versions should fly; they're not threadsafe, and it would be nice to write-protect the instruction stream to avert that attack vector. 1) explicitely insert checks, if events are to be handled 1a) everywhere or 1b) in places like described below under [1] c) I like this (1b). With the JIT, an event check could be inlined to 1 load and 1 conditional branch to the event dispatcher, yes? (So long as interp is already in a register.) If that's done before blocking and at upward branches, the hit probably won't be killer for most of code. For REALLY tight loops (i.e., w/o branches or jumps, and w/ op count less than a particular threshold), maybe unroll the loop a few times and then still check on the upward branch. Those branches will almost always fall straight through, so while there will be load in the platform's branch prediction cache and a bit of bloat, there shouldn't be much overhead in terms of pipeline bubbles. The event ready word (in the interpreter, presumably) will stay in the L1 or L2 cache, avoiding stalls. No, it's not zero-overhead, but it's simple and easy enough to do portably. Crazy platform-specific zero-overhead schemes can come later as optimizations. -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
RE: Events and JIT
Michal, But rather than always monitoring for that, I'd want to register a listener and then have parrot handle the polling for me. This is precisely what's being discussed. This is probably a dumb question but: what if signals threw exceptions instead? I'd hope that the event handler for a signal event could elect to throw an exception; it could even be the default. But the exception has to get into the thread somehow-- exceptions don't autonomously happen, and they require considerable cooperation from the thread on which the exception occurs. High-priority events are that mechanism through which the code which will throw the exception can interrupt normal program execution. -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
RE: Events and JIT
Leopold Toetsch wrote: Why? The bytecode is patched by a different thread *if* an event is due (which in CPU cycles is rare). And I don't see a thread safety problem. The (possibly different) CPU reads an opcode and runs it. Somewhere in the meantime, the opcode at that memory position changes to the byte sequence 0xCC (on intel: int3 ) one byte changes, the CPU executes the trap or not (or course changing that memory position is assumed to be atomic, which AFAIK works on i386) - but next time in the loop the trap is honored. Other threads than the target could be executing the same chunk of JITted code at the same time. -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
Re: JVM as a threading example (threads proposal)
point with respect to VM-internal state, morph scares me quite a bit. Type-stable memory is vital to lock-free (efficient) data access. Consider: a_pmc is of type A --- thread 1 is executing --- Thread 1: load a_pmcvtable-whatever Thread 1: call to a_vtable_whatever begins... --- thread 2 preempts thread 1 --- Thread 2: morph PMC to type B Thread 2: Allocate new B-type pmc_ext structure Thread 2: update a_pmc-vtable Thread 2: adjust a_pmc-pmc_ext --- thread 1 resumes --- Thread 1: ... finish setting up the call Thread 1: a_vtable_whatever reads ((A_ext) a_pmc-pmc_ext)-foo; B_ext happens to keep some float at the same offset in the struct Thread 1: chaos Keeping in mind the reality of delayed writes on SMP systems, there's just no way to code around this except to acquire a lock on the entire PMC for every read or write. Bye-bye performance: Acquire 2-3 locks to add 2 PerlInts together? ... Or bad performance and poor concurrency: Acquire 1 global lock whenever performing any PMC operation. I'm tired Gordon Henriksen [EMAIL PROTECTED]
Re: JVM as a threading example (threads proposal)
follow when generating a sequence of opcodes from Java source code. What I think it's really saying, again translated into Parrot terms, is this: What it's really saying is that JVM must be wary of aliasing, a bane of optimizing compilers. store_global foo, P0 # internally, may cache value and not push to main memory Well, might not immediately push to main memory. But, yes, a dirty write cache would qualify as an in-flight memory transaction (an uncommitted store operation). find_global P0, foo # internally, can't pull value from main memory if above value was not yet pushed there True, but it's enforced by a different rule: Let action A be a load or store by thread T on variable V, and let action P be the corresponding read or write by the main memory on variable V. Similarly, let action B be some other load or store by thread T on that same variable V, and let action Q be the corresponding read or write by the main memory on variable V. If A precedes B, then P must precede Q. (Less formally: operations on the master copy of any given variable on behalf of a thread are performed by the main memory in exactly the order that the thread requested.) and I think the point is this: find_global P0, foo # internally, also caches value in thread-local storage find_global P0, foo # internally, can use cached thread-local value Don't think of working storage as a cache. Think of it as a necessity: The processor can't operate on data unless it has been brought into the working storage. The true complexities of caching are encompassed by the read and write operations and the orderings imposed on themparticularly with respect to the fact that only explicit locking imposes an ordering on reads and writes between threads. Caches, the stack, and registers all fall under the rubric of working storage in this spec. And, as mentioned in section 8.6, any time a lock is taken, cached values need to be pushed back into main memory, and the local cache emptied. This doesn't make any sense if the thread's working memory is interpreted as the stack. It does make sense. The rules says that all thread-local changes must be flushed completely to main memory before the thread blocks. It ensures that other threads will see changes made by the thread, and visa versa, so that data protected by the lock is always viewed in a consistent state. On a PowerPC, the JVM just needs to store all assigned-but-unstored variables and then execute the PPC sync instruction. Gordon Henriksen [EMAIL PROTECTED] [Vocab note: A write-back cache only stores changes to main memory when the variable is evicted from the cache (it optimizes both loads and stores). Its corollary is a write-through cache, which stores changes to main memory on every update (it only optimizes loads). A write-back cache prevents implicit SMP cache coherency, because it gives the other processor caches no sign that the memory word has been changed until the word is evicted naturally or a cache-coherency instruction (like sync) forces the issue.]
RE: JVM as a threading example (threads proposal)
-Original Message- From: Jeff Clites [mailto:[EMAIL PROTECTED] Sent: Thursday January 15, 2004 01:28 To: Gordon Henriksen Cc: [EMAIL PROTECTED] Subject: Re: JVM as a threading example (threads proposal) On Jan 12, 2004, at 10:03 AM, Gordon Henriksen wrote: On Monday, January 12, 2004, at 04:29 , Jeff Clites wrote: 5) Java seems to use a check-in/check-out model for access to global data, in which global data lives in a central store, but is copied back-and-forth to thread-local storage for modification. I don't fully understand the performance benefit which I assume this provides, but this is something else we need to understand. It's discussed in detail in the threads section of the JVM spec, http://java.sun.com/docs/books/vmspec/2nd-edition/html/ Threads.doc.html. The passage in question being: Every thread has a working memory in which it keeps its own working copy of variables that it must use or assign. As the thread executes a program, it operates on these working copies. The main memory contains the master copy of every variable. There are rules about when a thread is permitted or required to transfer the contents of its working copy of a variable into the master copy or vice versa. There's that passage, but actually the following sections go into depth about the semantic constraints on how this is supposed to work. This is very obliquely phrased, but it refers to the register file and stack frame. Or, since the Java bytecode is stack-based, it refers to the stack. You might be right, but that's not exactly how I read it, because later it says, A use action (by a thread) transfers the contents of the thread's working copy of a variable to the thread's execution engine. This action is performed whenever a thread executes a virtual machine instruction that uses the value of a variable. I assume that the stack would correspond to the execution engine here, rather than to the thread's working copy, but I could read it either way. In any case, I thought this sounded like an interesting model, but based on the rules in that section about how accesses to data in the thread-local store have to correspond to accesses of the main store, it wasn't apparent to me how there was an advantage over just accessing things directly out of the main store. But I assume I'm overlooking some subtlety, and that there may be an idea here that we could use to our advantage. The two-level memory model in the spec seems to address, as a group, the register file, stack (JVM stack and/or system call stack; same difference), and the cache hierarchy in an SMP system. All in all, it's a very oblique way of saying that changes might not be immediately visible to all threads. For example, on the PPC, a sync instruction is required to ensure that all the processor caches have a consistent view of main memory. sync and its equivalents are far too expensive to execute after every change to any variable, so the JVM's spec is written to allow implementations to avoid it most of the time. -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
Re: JVM as a threading example (threads proposal)
On Monday, January 12, 2004, at 04:29 , Jeff Clites wrote: 5) Java seems to use a check-in/check-out model for access to global data, in which global data lives in a central store, but is copied back-and-forth to thread-local storage for modification. I don't fully understand the performance benefit which I assume this provides, but this is something else we need to understand. It's discussed in detail in the threads section of the JVM spec, http://java.sun.com/docs/books/vmspec/2nd-edition/html/ Threads.doc.html. The passage in question being: Every thread has a working memory in which it keeps its own working copy of variables that it must use or assign. As the thread executes a program, it operates on these working copies. The main memory contains the master copy of every variable. There are rules about when a thread is permitted or required to transfer the contents of its working copy of a variable into the master copy or vice versa. This is very obliquely phrased, but it refers to the register file and stack frame. Or, since the Java bytecode is stack-based, it refers to the stack. Gordon Henriksen [EMAIL PROTECTED]
Re: More object stuff
On Monday, January 12, 2004, at 11:37 , Dan Sugalski wrote:
At 4:07 PM + 1/12/04, Harry Jackson wrote:
Dan Sugalski wrote:
Well...
What I'd like, I think, is something simple and straightforward.
Right now we've got to fetch each column for each row one by one,
which is a pain in the neck if you want to get a full row back.
Having a fetchrow that returned an Array with the value for column
one in the 0th slot, column 2 in the 1st slot and so on would be
about 80% of the solution.
I have done this part.
Having a fetchrow_hash that returned a Hash where the keys are the
column names and the values are the column values would be most of
the rest.
I read somewhere that accessing a hash was slightly slower than
accessing and array which is one reason why I never used it. The other
reason is that if I name the fileds in the hash then the user needs to
know the names to access them or perform some magic to get them. With
an array they come out in the order they are aksed for.
Definitely. However... Having seen code in production, generally the
fields aren't changeable and are known at compile-time, or folks are
writing generic code (for better or worse). In the first case people
use hash access because they know there's a name and city field in
the results, and in the second case they're iterating across the keys
and pulling out values.
Since folks are going to wrap the results in a hash regardless of
whether it's a good idea or not (not going there... :) we might as well
have it in at the very lowest level where we can get the information
most efficiently.
fetchrow_hashref is definitely a very useful, but my favorite (and also
the most efficient) DBI methodology is bind_columns. DBI maintains a
list of references corresponding to columns in the result set, and when
the result set is advanced, stores the values into the variables
referenced. e.g., Perl 5:
my $sth = $dbh-prepare(select a, b, c from tab);
$sth-execute;
$sth-bind_columns(\my($a, $b, $c));
while ($sth-fetch) {
print a: $a, b: $b, c: $c\n;
}
Equivalent to:
my $sth = $dbh-prepare(select a, b, c from tab);
$sth-execute;
$sth-bind_col(0, \my $a);
$sth-bind_col(1, \my $b);
$sth-bind_col(2, \my $c);
while ($sth-fetch) {
print a: $a, b: $b, c: $c\n;
}
So if you're going to basically go all out in emulating DBI's fetch_*
permutations, don't forget this one. :)
Gordon Henriksen
[EMAIL PROTECTED]
Re: Threads Design. A Win32 perspective.
On Sunday, January 4, 2004, at 03:17 , Jeff Clites wrote:
On Jan 3, 2004, at 8:59 PM, Gordon Henriksen wrote:
On Saturday, January 3, 2004, at 04:32 , Nigel Sandever wrote:
Transparent interlocking of VHLL fat structures performed
automatically by the VM itself. No need for :shared or lock().
Completely specious, and repeatedly proven unwise. Shouldn't even be
pursued.
Atomic guarantees on collections (or other data structures) are rarely
meaningful; providing them is simply a waste of time. Witness the
well-deserved death of Java's synchronized Vector class in favor of
ArrayList. The interpreter absolutely shouldn't crash due to threading
errorsit should protect itself using standard techniquesbut it would
be a mistake for parrot to mandate that all ops and PMCs be
thread-safe.
What are these standard techniques? The JVM spec does seem to guarantee
that even in the absence of proper locking by user code, things won't
go completely haywire, but I can't figure out how this is possible
without actual locking. (That is, I'm wondering if Java is doing
something clever.) For instance, inserting something into a collection
will often require updating more than one memory location (especially
if the collection is out of space and needs to be grown), and I can't
figure out how this could be guaranteed not to completely corrupt
internal state in the absence of locking. (And if it _does_ require
locking, then it seems that the insertion method would in fact then be
synchronized.)
So my question is, how do JVMs manage to protect internal state in the
absence of locking? Or do they?
The JVM won't crash, but individual objects can easily become corrupt.
Java makes no guarantees of the level you're proposing. All
synchronization in Java is explicit, using either synchronized methods,
or synchronized (...) { ... } blocks; the runtime doesn't implicitly
burn cycles synchronizing objects for the user. It does, however, give
the user the threading primitives which allow them to write thread-safe
code. Thread-safety is, indeed, left entirely to the user. This is
crucial to allowing code to execute quickly and for the optimizer to
function. The JVM protects itself, ensures that its core is thread-safe,
and provides a secure operating environment for user code; it is the
responsibility of the authors of classes to protect instances of their
classes. Remember, Java's libraries are implemented atop these
primitives *in Java.* It would be analogous to propose that parrot-based
languages be primarily implemented atop its primitives in PBC, and that
the sections which are not so-implemented (the core) must be
scrutinized carefully for threadsafety.
What are these threadsafe primitives which Java provides? For example:
1. Fixed-size arrays are safe to access concurrently from multiple
threads.
2. ints, chars, pointers, etc. can be updated atomically.
3. Java's memory allocator is thread-safe.
4. Strings are immutable (and thus sharable without threadsafety
violations).
5. The JVM does not allow memory to be resized, making bounds-checking
threadsafe.
6. No objects can be moved (while retaining their identity) except by
the garbage collector.
7. All threads are suspended during GC.
8. All pointers are traceable by GC.
That parrot is largely implemented in C, and is designed to operate in
terms of fat data structures, makes it much more difficult to guarantee
in parrot that segfaults will not occur. I have some ideas which I'll
share after I sync up with this fast-moving thread (of conversation).
But one of my premises is that implicitly guaranteeing atomic PMC
accesses will render threaded parrot dead-on-arrival. Just like the last
2 abortive attempts at creating a threading Perl implementation. (And
threaded parrot would be a separate executable, because the
performance hit will be so staggering.) The sheer overhead of locking is
an obvious deal-killer, but I think that deadlocks and prohibition of
optimizations would quickly put a few extra nails in the coffin. I don't
want to see the Perl community left adrift with another toy threading
implementation unsuitable for any serious work.
Gordon Henriksen
[EMAIL PROTECTED]
Re: Threads Design. A Win32 perspective.
On Sunday, January 4, 2004, at 01:43 , Dan Sugalski wrote: At 11:59 PM -0500 1/3/04, Gordon Henriksen wrote: On Saturday, January 3, 2004, at 04:32 , Nigel Sandever wrote: Transparent interlocking of VHLL fat structures performed automatically by the VM itself. No need for :shared or lock(). Completely specious, and repeatedly proven unwise. Shouldn't even be pursued. Erm... that turns out not to be the case. A lot. (Yeah, I know, I said I wasn't paying attention) An interpreter *must* lock any shared data structure, including PMCs, when accessing them. Otherwise they may be in an inconsistent state when being accessed, which will lead to data corruption or process crashing, which is unacceptable. These locks do not have to correspond to HLL-level locks, though it'd be a reasonable argument that they share the same mutex. Process crashing unacceptable? Absolutely. Complete agreement. Data corruption unacceptable? I disagree. It depends on the contract put forward by the language in question. Notably, Perl makes no such guarantees thus far, being as how it doesn't (any longer) run in a traditional threaded model. Successfully threaded languages and virtual machines explicitly make NO such guarantees. I'm completely okay with the potential for inconsistent data where the user doesn't take precautions. If the alternative is my code bogging down to half of its ideal speed because parrot is wasting time: acquiring three mutexes with deadlock detection so that it can execute add $P10, $P11, $P12; acquiring a mutex every time it indexes a string register; acquiring a mutex so it can examine cache-int_valif those are the alternatives, I'm for the potential for inconsistent data. (And I say wasting because it's completely unnecessary in a good 99% of cases; the vast majority of data is not shared between threads at all.) I'm not okay with parrot crashing for any reason, ever, though, so where locking is not provided, PMC classes *must* be coded *very* carefully so as to avoid crashes. Gordon Henriksen [EMAIL PROTECTED]
Re: Threads Design. A Win32 perspective.
On Saturday, January 3, 2004, at 04:32 , Nigel Sandever wrote: Transparent interlocking of VHLL fat structures performed automatically by the VM itself. No need for :shared or lock(). Completely specious, and repeatedly proven unwise. Shouldn't even be pursued. Atomic guarantees on collections (or other data structures) are rarely meaningful; providing them is simply a waste of time. Witness the well-deserved death of Java's synchronized Vector class in favor of ArrayList. The interpreter absolutely shouldn't crash due to threading errorsit should protect itself using standard techniquesbut it would be a mistake for parrot to mandate that all ops and PMCs be thread-safe. The details of threaded programming cannot be hidden from the programmer. It's tempting to come up with clever ways to try, but the engine really has to take a back seat here. Smart programmers will narrow the scope of potential conflicts by reducing sharing of data structures in their threaded programs. Having done so, any atomicity guarantees on individual objects proves to be wasted effort: It will be resented by parrot's users as needless overhead, not praised. Consider the potential usage cases. 1. All objects in a non-threaded program. 2. Unshared objects in a threaded program. 3. Shared objects in a threaded program. The first two cases will easily comprise 99% of all usage. In only the third case are synchronized objects even conceivably useful, and even then the truth of the matter is that they are of extremely limited utility: Their guarantees are more often than not too fine-grained to provide the high-level guarantees that the programmer truly needs. In light of this, the acquisition of a mutex (even a mutex that's relatively cheap to acquire) to push an element onto an array, or to access a string's datawell, it stops looking so good. That said, the interpreter can't be allowed to crash due to threading errors. It must protect itself. But should a PerlArray written to concurrently from 2 threads guarantee its state make sense at the end of the program? I say no based upon precedent; the cost is too high. Gordon Henriksen [EMAIL PROTECTED]
Factoring threads friends
I just thought I would put forward a summary of how .NET factors its
thread-related data structures. It's a successful, performant, lock-free
design, and I think a quite interesting factorization of what parrot
presently presents as the interpreter. It's the single most successful
threading design I've encountered to date.
The entities involved are the process, the thread, the app domain.
These entities take separate responsibility for some of the features of
the parrot interpreter.
The process is the global-most .NET state, owning threads and app
domains, and coordinating events such as GC. This is truly global state,
and there's not a whole lot of it.
.NET threads contain little more than a stack and an instruction pointer.
App domains are lightweight protected execution environments. Code is
loaded into app domains, and types registered in data structures owned
by the app domain. Objects are allocated within an app domain. The
majority of the .NET runtime is a property of the app domain. Individual
classes and libraries cannot be unloaded in .NET, but entire app domains
can be.
Stack frames are obviously within a thread's stack, and thus owned by
that thread. But each stack frame also is attached to a particular app
domain, and thus knows how to allocate objects, etc. without the
overhead of an extra pointer in every object; the app domain is .NET's
implicit argument, akin to parrot's interpreter parameter. If this were
a relational database, one might say that a stack frame was a
many-to-many relationship between threads an app domains. So a thread
does not belong to a single app domain: This is not a hierarchical
design.
Objects live in one and only one app domain. Since every stack frame
also knows its app domain, it follows that all visible objects belong to
the app domain of the executing thread. Thus memory pools can be easily
identified for memory allocation, and the type registry found, etc. etc.
etc.in general, the runtime can be utilizedwithout the waste of an
extra pointer in the object header.
App domains cannot directly reference each others' objects.
Communication between app domains most resembles RPC. Only through proxy
objects can one app domain interact with another. The default behavior
is that objects passed as arguments to proxy methods will be serialized
and deserialized into the target app domain (although they can specify
that remote proxies will instead be created in the target app domain).
These proxies obviously do need to store a pointer to the peer app
domain, but other objects do not, keeping overhead low.
Objects do not have fine-grained locks. If their methods require
synchronization, their implementations must request it with a block:
lock (this) { /* code goes here */ }. Most of the objects in the
libraries are not safe for multithreaded access (although their static
methods generally are).
GC affects more than one app domain: It halts all threads. (Certain
other events do as well, including unloading an app domain, or mucking
with loaded types, or backing out JIT optimizations.) The garbage
collector is aware of proxy objects that bridge app domains. Since it
operates at the process level, it can collect garbage cycles even across
app domains (where normal RPC proxies would cause reference cycles).
The .NET design doesn't meet all of requirements Dan set forth, but it's
at least an interesting case study in a successful threading environment
for a high-performance virtual machine.
Gordon Henriksen
[EMAIL PROTECTED]
Threading design
I wish the threading design for parrot would look more toward successful, performant multithreaded systems, rather than setting up new design experiments based upon the results of failed experiments (in particular, all forms of Perl 5 threading). I think that environment-cloning and fine-grained locks have both been adequately proven antithetical to what is expected from threads: Lightweight, low-overhead concurrency. Environment cloning is high-performance, but high overhead. Fine-grained locks are low overhead, but low-performance. Gordon Henriksen [EMAIL PROTECTED]
Re: Threads and Events (was Re: threads and shared interpreter data structures)
On Tuesday, December 23, 2003, at 08:40 , Rod Adams wrote: - Most treaded code can be converted to an event loop (+async I/O) without issue. Esp if we have co-routines. - For the other stuff, we'd still have Type 2 threads, which gets the job done. (Just got back from vacation and was reviewing this aging thread...) Not to throw a damper on the events party or anything, but an event-based system with asynchronous I/O isn't capable of saturating an SMP box's processors. This is one of the major reasons for using threads in web servers. It's also a significant reason for using threads in desktop applications. Yes, N interpreters for N processors will get you there, but at the cost of times-N static memory consumption (for JITted code, type registries, vtables, etc.: interpreter overhead), and at the cost of fine-grained, lightweight inter-thread communication between the segregated threads. Further, threading implemented as context switches at block time amounts to a cooperative multithreading environment. Yes, it may provide near-optimal throughput. Despite that, it also has some very bad indeed worst-case latency characteristics. If a worker thread fails to block, the thread which started it will never (or rarely) run and the program will become unresponsive. This makes such a threading model unsuitable for use as in a web application host. One misbehaving HTTP request handler mustn't block other requests. A worker thread mustn't block the UI thread. Sidenote: Shades of System 7: CPU benchmarks on the old Mac OS do run several percentage points faster than on preemptive systems. The preemptive model is clearly superior in the general case, though; its perceived performance under load is by far superior. Also worth noting: parrot will already be paying the preemptive performance penalty on any modern OS. I can only hail core events and asynchronous I/O as great advances in parrot. But they are not a general replacement for preemptive multithreading. Of course, TMTOWTDI and YMMV, but parrot should support both models well, and the above line of thought isn't doing threading justice in my opinion. Gordon Henriksen [EMAIL PROTECTED]
Re: Threads
Leopold Toetsch wrote:
Dan Sugalski wrote:
Leopold Toetsch wrote:
Again, not only strings but all kind of containers using managed
resources suffer from that problem.
All that seems to imply, that we need two locks for shared access: one
for protecting DOD/GC and a second for the PMC. Any container
(including
PerlStrings) would have to aquire the first (interpreter-global) lock,
while a PerlNum only needs the second per PMC mutex. Ugly.
Yeah, it all potentially gets nasty.
[...]
[3] They can't be suspended inmidst of an arbitrary operation (and
pthread doesn't even provide suspending) so they would be sent a
suspend event so that they are in a safe state in the run-loop, or
better in the event-handler opcode, called from there.
(Just got back from vacation and was reviewing this aging thread...)
Yes, yes! Stopping other threads when a rare condition occurs is a great
way to avoid penalizing a threaded program's common case with excessive
synchronization overhead. There are just too many things that can go
wrong otherwise; fine-grained locking is the alternative, and programs
will very, very rapidly find themselves spending 40% of their cycles
acquiring mutexes.
This is definitely the way to go to achieve blistering performance while
retaining dynamic capabilities. The work is already done to ensure that
while true { ; } will receive events in a timely fashion. Broadcasting
parrot events should be an excellent means of implementing inter-thread
synchronization.
Gordon Henriksen
[EMAIL PROTECTED]
RE: Namespaces
Leopold Toetsch [EMAIL PROTECTED] wrote:
Dan Sugalski [EMAIL PROTECTED] wrote:
Okay, okay, I give -- hierarchic namespaces are the way to go. Makes
local overrides somewhat interesting, but we'll burn that bridge
when
we get to it.
find_global P1, ['global', 'namespace', 'hierarchy'], thingname
So (excuse the Perl 5), $::thingie would be this:
find_global P1, [], thingie
(Or [], $thingie.)
Makes it odd to actually name a symbol, since it's broken in one + N.
What's the good of separating the symbol name from the namespace name?
That is, split the namespace path from the name of the thing, and
make the namespace path a multidimensional key.
Or I suppose we could just punt and toss the special global access
entirely and make the global namespace a hash 'o hashes hanging off
the interpreter and access it like any other variable,
What about:
getinterp P2
set P1, P2['global';'namespace';'hierarchy';'thingname']
That is get_pmc_keyed() on a ParrotInterpreter PMC with a multi-key,
straight-forward. The set_pmc_keyed() stores a global.
What if global.namespace happens to be autoloaded or otherwise magic?
Will the get_keyed break down and do something equivalent to this?
getinterp P2
set P1, P2['global';'namespace']
set P1, P1['hierarchy';'thingname']
Meaning the construction of an extra multikey? Yech, maybe.
Constructing a multy-key by hand isn't that simple (or would need
hacking imcc) (and Key componenent separator is a semicolon).
YECH.
I can see y'all are already leaning away from this direction, but I
thought
I'd still bring them up as ideas:
- The nameless root namespace is an easy-to-get-to parrot or
interpreter global, so there's no need to have a separate interface
to access the global namespace vs. accessing a namespace in a PMC
register.
(Though interp-is-a-namespace is cute. Maybe too cute, if namespaces
are subclassable.)
- Namespaces have methods/ops to look up a contained symbol (just
one level), where any string is a valid symbol. (And there's the
option of not recursing into parent namespaces; i.e., looking only
at declared members.)
- Namespaces also have methods to traverse a path of symbols, where
all but the last symbol in the path must itself be a namespace.
Symbol paths are mangled using a DEAD SIMPLE, absolutely canonical
encoding, with debug-friendly printable ASCII escapes. e.g.:
# Example only:
# '.' is path delimiter and '%' is escape char
# Using '\' would induce LTS in IMCC C.
foreach (@path) {
s/\%/\%\%/g;
s/\./\%\./g;
}
$path = join(., @path);
So unless you use a '.' or '%' in your symbol, it won't be mangled
at all, and if it started out printable ASCII, it'll stay that way.
And if you do use one of those reserved characters, the mangled form
won't be too terribly surprising.
This does presume that symbols must be in a charset with '.' and
'%'. My heart bleeds for those that aren't.
Bad Thought: If the empty symbol were prohibited (or ignored) in paths,
the delimiter . could be its own escape character, e.g.:
[float;1.0] could be float.1..0.
While a strawman symbol path lookup could be implemented in terms of
single-level symbol lookup, smarter namespace implementations might
optimize away the array (multikey) and extra strings. And this doesn't
require any massive restructuring of IMCC or PIR. (Unless you want them
to just do the mangling internally, rather than storing that [, ,
]
as an array constant. But why bother? .. is shorter than [, , ],
and doesn't suggest to a compiler author's mind keep a struct { char**;
char*; } around for each symbol reference.)
This would make Dan's example into:
find_global P1, global.namespace.hierarchy.thingname
shorthand for:
get_globals Py
find_sym P1, Py, global.namespace.hierarchy.thingname
and functionally equivalent to the wildly pedantic:
get_globals Py
find_onesym Py, Py, global
find_onesym Py, Py, namespace
find_onesym Py, Py, hierarchy
find_onesym P1, Py, thingname
Spelling aside, anyhow. (e.g., where find_onesym yada might be spelled
set yada[].)
--
Gordon Henriksen
IT Manager
ICLUBcentral Inc.
[EMAIL PROTECTED]
RE: [CVS ci] object stuff
Melvin Smith [EMAIL PROTECTED] wrote: my $foo = Oracle::Instance::DEV1::db_block_buffers; The namespace lookup in Oracle::Init checks the Oracle config parameters which is external code. All sorts of neat possibilities. :) It is truly remarkable the lengths that Perl programmers seem to be willing go to in order to hide a function call or obscure the existence of an object. :) -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
Re: [RfC] Testing for null
On Monday, December 8, 2003, at 10:03 , Dan Sugalski wrote: At 1:21 PM -0500 12/3/03, Melvin Smith wrote: We should have 1 recommended way for testing NULL registers. If we support get_bool() then lets make sure it works for REAL NULL pmc registers as well as PMCNULL. If not, code will appear to work correctly on a safe core but will seg fault on some other. Also, I see no reason not to use PMCNULL in all cores now. Okay, lets do this: Add an isnull branch op: isnull Px, destination How about this to test if Px is really null? null Py eq_addr Px, Py Gordon Henriksen [EMAIL PROTECTED]
Re: [RFC] IMCC pending changes request for comments
On Tuesday, December 2, 2003, at 11:49 , Melvin Smith wrote: At 07:59 PM 12/2/2003 -0800, Steve Fink wrote: Do you really want to generate the extra unused code at the end of all the subroutines? I don't think you want to autodetect whether the code is guaranteed to return. Good point. Its easy to detect returns if the code uses high level IMC directives, but in cases where it is all low-level PASM, it could get a little troublesome. It would also add a few dead instructions here and there. Nix that idea. Maybe just pad the end of the compilation unit with the opcode equivalent of PMCNULL, to fire an exception rather than having undefined behavior? Gordon Henriksen [EMAIL PROTECTED]
RE: Calling conventions. Again
Leo,
Consider this (as I'm sure you already have)...
class ClassA;
method meth($i is int) { ... }
class ClassB;
method meth($f is float) { ... }
module main;
my $obj = (rand(1.0) 0.5? ClassA : ClassB)-new;
$obj-meth(1);
Perl's going not going to be able to bind to a HLL prototype, much less
a low-level one. Even if it could bind to the HLL prototype, it has to
protect against mutation of the method. Either perl6 recompiles the
caller when a new version of the sub is linked in, or perl6 uses a
calling convention that always works: Unprototyped.
Prediction: perl6 winds up using One True Parrot Unprototype for ALL of
its method and sub calls.
The parrot calling conventions thus become rather academic to Perl. The
target languages under consideration just aren't going to get little
mileage out of them.
Leo and Dan,
As for encoding varargs in the parrot calling conventions, parameter
counts are very much useless--they're might as well be a very poor
one-way hash of the prototype. Have you considered using 4 bitfields to
indicate what register parameters are what? E.g.
prototype: intin In+0,
float in Nn+1,
string in Sn+2,
string in Sn+3,
float in Nn+4,
intin In+5,
PMCin Pn+6
Iregs = (1 n+0) + (1 n+5) = 0b...011...
Fregs = (1 n+1) + (1 n+4) = 0b...0010010...
Sregs = (1 n+2) + (1 n+3) = 0b...0001100...
Pregs = (1 n+6) = 0b...100...
This burns no more space in the register file than the 4 parameter
counts, and it doesn't throw away ordering data. Downside: Counting
total number of arguments is harder.
Four 32-bit bitfields match the 4 sets of 32 registers, so even if the
set of registers used for passing parameters grow, this convention is
safe. Overflow parameters... Well, figure something out; there's plenty
of space in the overflow array. (4 binary Sregs with likewise bitfields?
Or just use the PMC type.)
But imposing the overhead of 4 Ireg assignments on the caller and 4 Ireg
compares and branches on the callee at the beginning of every routine,
just to check that the register conventions are right? Don't think
that's a good idea. What's the recovery scenario going to be, anyhow? To
blow up... What a waste of cycles: It's a needless toll for 99.99%
of calls. Check that the prototype is right when the sub is loaded into
the runtime: In effect, throw the exception at link time. Then this
vararg cruft can be cut out of the common case. That is, support 3
parrot calling conventions:
- unprototyped (your parameters are in an array)
- prototyped (what you want in the register file is there)
- vararg (I've documented what I put in the register file)
But, honestly, a vararg prototype string in an Sreg probably makes more
sense than either 4 register counts or 4 register masks. It avoids
burning integer registers--the most useful part of the register file for
the sort of low-level code which will actually USE parrot prototypes. It
can be set up by the caller with a single string constant load. It can
be checked by the callee with a single string compare instruction (if
the vararg list is actually static). It's easy to reflect, too.
--
Gordon Henriksen
IT Manager
ICLUBcentral Inc.
[EMAIL PROTECTED]
-Original Message-
From: Dan Sugalski [mailto:[EMAIL PROTECTED]
Sent: Friday, November 14, 2003 10:34 AM
To: Leopold Toetsch
Cc: [EMAIL PROTECTED]
Subject: Re: Calling conventions. Again
On Fri, 14 Nov 2003, Leopold Toetsch wrote:
Dan Sugalski [EMAIL PROTECTED] wrote:
I've seen it with some depressing regularity over the
years. It generally
takes the form of an upgrade to a library that breaks existing
executables, something we're going to have to deal with
as we're looking
to encourage long-term use of bytecode-compiled programs.
This seems to me the same, as that strcpy(3) should be
guarded against
wrong argument count at runtime. But swapped
destination/source can't be
detected anyway ;)
We can't detect bugs like that, true. But we can detect when
someone calls
us with two arguments and someone has, in the mean time,
helpfully added
an optional length arg to strcpy.
... But there are
several issues here:
1) vararg calls with non-pmc registers involved
I already did propose the syntax:
[Snip]
at least, so that runtime checks can be omitted for certain cases.
No IMCC syntax that's purely compile-time is of any help
here. The code
doing the calling
2) Runtime modification of sub definitions
are evil, forbidden, disallowed. This just can't work.
True, false, false. It happens, in some cases a *lot*. This is perl,
python, and ruby we're talking about, where changing the
definition of a
sub is as trivial as a reference assignment into a global
hash. It's easy,
people do it. Often, in some cases. (Heck, I've
RE: Ordered destruction and object graph traversal
Dan Sugalski wrote: Allow me to haul out this bucket of ice-water I keep around for just such an eventuality. :) There's a low limit to the complexity of any sort of traversal we can provide. We *can't* go recursive in a traversal, if it crosses the C-Parrot or Parrot-C boundary as part of the recursion, or stays entirely in C. We can only count on a 20-40K stack *total*, and it doesn't take too many calls into a recursive procedure to eat that up. That limits the mechanisms we can use to traverse the graph of objects pretty badly. The linked list mechanism the DOD system is using is one of the few that isn't recursive. There are other mechanisms that can be used for traversal that avoid recursion, but they generally trade off pretty heavy memory usage for that lack, which makes them unsuitable for general use. That's what's beautiful about this: -visit isn't providing the traversal algorithm at all. This vtable is agnostic to the traversal algorithm. Breadth-first or depth-first; seen hash or mark bit or what have you; zero-resource or resource-consuming: Whatever is best suited to the requirements at hand. And it cleanly exposes that core functionality to HLLs (which may have algorithmic requirements for traversals that involve queues or stacks). If parrot is embedded multi-threaded in an RDBMS or web server (which strikes me as a more likely scenario than parrot embedded in a resource-constrained embedded environment), then blocking the interpreter for a serialization on one thread will negatively impact response time, concurrency, and scalability. That might be considered more damaging than additional resource consumption. -visit neatly enables BOTH resource-consuming, threadsafe serialization AND zero-resource, blocking serialization. -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
Re: Ordered destruction and object graph traversal
On Monday, October 20, 2003, at 11:40 , Jeff Clites wrote: My solution was to define a new vtable method--I've called it visit(), though the name's not the important part--to which you pass a callback (plus an optional context argument). It's job is to invoke that callback on each of it's referenced children (also passing the context object with each invocation), and that's it. The PMC doesn't know or care what the callback does--in a GC context the callback may set a flag on what's passed in and stash it for later processing; in another context it may be used inside of an iterator which vends out objects one-by-one and guards against loops. Great! This is exactly the fundamental operation I'd been musing would be the best building block to enable serialization and pretty-print operations, and I'm sad to see that your post has been Warnocked. :( This mechanism is excellent in that it enables both breadth-first and depth-first traversals, and is neutral to whether a pointer is traversed or not: The client (callback) can decide that based upon the live bit or the ordered destruction bit or a seen table or the phase of the moon. This also doesn't alone imply any resource consumption. And it doesn't affect concurrency; threading is preserved. Great! Serialization is a very natural client of this API, and at least some of the arguments surrounding serialization should be lessenedbecause the core technology is amenable to use by various possible algorithms, and so deciding upon those algorithms early becomes less vital. Serialization simply becomes an easier problem with this in place. I would venture, though, that DoD may well need a separate and heavily optimized implementation, purely for efficiency's sake. Gordon Henriksen [EMAIL PROTECTED]
RE: cvs commit: parrot/languages/imcc/t/syn pcc.t
* e.g. add_n_i_n = add_n_n_i * div_n_ic_n = div_n_nc_n * div_n_i_n = set_n_i ; div_n_n_n + * ge_n_ic_ic = ge_nc_ic -+-+ | | | | _|_| / \ | ( X X ) | \___/ | | | ---+--- | | | | | | | A | / \| / \ | / \ | / \ | / \| | | | | A C D E G I| | N S T V|
RE: Next Apocalypse
. (A ward against newly-born threads.) 4. Change the world, firing notifications as per above. 5. Release the mutex from step 1. 6. Done. - Stalled threads acquire and release the mutex, exit the event handler, and jump into the frame rewriter if their compilation was invalidated. Notes: - See any possibility of deadlocks? 3 could loop forever... - This has a only very slightly stronger requirements than those of a copying GC. -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
RE: Pondering argument passing
Dan Sugalski wrote: On Sun, 14 Sep 2003, Steve Fink wrote: I suppose that was a question for the language list. But then I'd have to read the language list. A fate worse than razor burn, to be sure. Possibly one worse than really bad Mexican food, but either way I'd not wish it on anyone. :) What's to read? -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
Re: [RFT] File Spec
On Sunday, September 14, 2003, at 12:50 , Michael G Schwern wrote: On Sat, Sep 13, 2003 at 09:55:48PM +0300, Vladimir Lipskiy wrote: To be clearer: concat_dirnames(b, /foo) == error. As long as concat_dirnames() will be taught to divine whether its arguments are absolute paths or relative paths, it could easily rotate its arguments so the above-mentioned call would become concat_dirnames(/foo, b). That would be really silly. I must agree. The only reasonable results of that are either (a) an error [] or (b) /foo. (I'd go with [b].) Gordon Henriksen [EMAIL PROTECTED]
RE: [RfC] vtable-dump
Dan Sugalski [EMAIL PROTECTED] wrote: On Tue, 9 Sep 2003, Gordon Henriksen wrote: Random thought There's some discussion on perl-qa right now about how Test::More should implement is_deeply, which executes a code block and tests that the return value is equivalent to a particular nested data structure. (The question posed on that list is with regard to how to handle tie()'d objects, which is not what I'm addressing here.) Result of that discussion's outcome, should the serialization API being discussed here enable the following behavior? ok(freeze($expected) eq freeze($actual)); I bring this up because using object addresses as IDs in the serialization entirely prevents this usage. Good. Having waded through one of the threads on p5p just a minute ago, I'm not willing to guarantee this. In fact, I'm willing to explicitly not guarantee this. If we want to compare two frozen structures, string equality is *not* the way to go. Each freeze could, theoretically, choose a different freezing method yet still represent the original. What do you mean by a different freezing method? That key-value pairs from two externally identical hashes could be frozen in different orders? I can see that. Sorting large hashes can be expensive, and certainly requires memory allocation. Or do you mean that freeze($objA) and freeze($objB)--freezing (definitionally identical!) object graphs and with no intervening code between the two calls to freeze--could internally and arbitrarily select a significantly divergent object graph encoding? I can't see that at ALL... Over time (and revisions), I certainly can see a desire not to marry parrot-freeze to a specific binary representation. That's not the question I intended to raise--I asked a question only of repeatability, not of permanent format invariance. This is a Good Place for a black-box comparison op, which string equality definitely is not. (At which point do extremely complex routines cease to be operators?) A black-box comparison of the (live) object graphs, or black-box comparison of the serializations themselves? I can see the former--while it's precisely the problem that consistent traversal outputs avoids, a deep == could have utility and avoid the serialization overhead. But how do you implement concurrent traversals without private seen tables? (e.g., if the graphs overlap, one of the traversals will find the other traversal's visited flag and fail to visit the entire graph.) Reentrant traversal again. Pathological case: $aa = [1]; $ab = [1]; %ha = ( a = \$aa, b = \$ab ); %hb = ( a = \$ab, b = \$aa ); # %ha and %hb overlap and are deep-identical, # but not deeply reference-identical. Comparing serialized object graphs strikes me as tremendously esoteric, e.g. a maintenance burden to be used by very few clients. (One of the few significant uses being that of replacing string-equals for the testing of the serializer itself.) It also strikes me as very, very, very complicated should the freeze methods diverge even slightly. I've never seen any such mechanism in any other environment. To compare graphs that I had saved in long-term storage, I as a caller would expect to need to deserialize the graphs and use a deep equals--and I would expect to implement deep equals (another feature I've never before seen as a specific feature of any environment) by serializing the deserialized graphs again and doing a string (or file) comparison. E.g., if I had read $a or $b from a file, I would expect to have to: freeze(thaw($a)) eq freeze(thaw($b)) instead of: $a eq $b That synthesizes the functionality from a minimal set of operations--freeze and thaw--and with a lot less maintenance for parrot. (At the cost of a bit of memory and runtime speed. But anybody who performs deep-compares in truly performance-critical code needs to meet my Clue Bat[tm].) -- Gordon Henriksen IT Manager ICLUBcentral Inc. [EMAIL PROTECTED]
