Re: continuation enhanced arcs
On Nov 28, 2004, at 2:48 AM, Piers Cawley wrote: I just thought of a heuristic that might help with register preservation: A variable/register should be preserved over a function call if either of the following is true: 1. The variable is referred to again (lexically) after the function has returned. 2. The variable is used as the argument of a function call within the current compilation unit. That doesn't solve it, though you'd think it would. Here's the counter-example: x = 1 foo() print x y = 2 return y You'd think that x and y could use the same memory location (register, variable--whatever), since ostensibly their lifetimes don't overlap. But continuation re-invocation can cause foo() to return multiple times, and each time it should print 1, but it won't if x and y use the same slot (it would print 2 each time after the first). In truth, their lifetimes do overlap, due to the hidden (potential) loops created by continuations. The problem isn't preservation across calls per se, it's the implicit loops. Continuations are basically gumming up tons of potential optimizations. JEff
Re: Another issue with pdd03
On Nov 15, 2004, at 12:38 AM, Leopold Toetsch wrote: Bill Coffman [EMAIL PROTECTED] wrote: [ pdd03 ] The way I read it, paragraph one implies that when you print P5 after calling foo(), you are expecting to get the return value. You didn't save and restore register P5, so you wanted foo() to do something to it. The nasty thing of a function call is: i = foo() # fine P5 returned vs. foo() # P5 clobbered by foo (but you can replace P5 with P15 too, or every register R5..R15) This has never bothered me, probably because of the comparison to the register-based calling conventions that the PPC uses: A called function (which returns a value) has to store its result in some register, whether or not the caller wants it. A call like i = foo() is really two steps: call the function, then copy the result from r3 to the appropriate location (maybe a location on the stack, maybe another register). It may be possible to optimize so that i is already using the same register as the return value, but in general that can't be arranged for most cases; consider how this would compile: i = foo() //call foo, copy result from r3 to other register--must since bar() would clobber j = bar() //call bar, copy result from r3 to other register--could avoid copy, if j not //needed past baz baz(i + j) //add those two other registers into r3, and call baz And due to the register-preservation semantics on the PPC, even a call to a void-return function could clobber r3, since it could call another function which returns a result and thus uses r3. Not that parrot has to necessarily work this way, but it at least has precedent, so it's not totally strange behavior. JEff
Re: Another issue with pdd03
On Nov 14, 2004, at 9:32 AM, Leopold Toetsch wrote: Defining now that P5 has to be preserved in main, because it's a possible return result of foo() and therefore may be clobbered by foo() is meaning, that we have effectively just 16 registers per kind available for allocation around a function call. If the latter is true according to current pdd03 then we are wasting half of our registers for a very doubtful advantage: being able to pass return values in R5..R15. In effect this is quite similar to the PPC calling conventions: we have roughly half of the registers preserved across function calls. In terms of the volatile registers, it's fine to use them for local calculations, as long as either you're using them to hold values which don't need to persist across function calls, or you preserve-and-restore them yourself. But that loops back to a previous proposal of mine: If they're not being preserved, and in fact need to be synced between caller and callee, then having these registers physically located in the interpreter structure, rather than in the bp-referenced frame, saves all the copying, and makes it more obvious what's going on. JEff
Re: Continuations, basic blocks, loops and register allocation
On Nov 15, 2004, at 10:29 AM, Leopold Toetsch wrote: Jeff Clites [EMAIL PROTECTED] wrote: Picture this call stack: main -- A -- B -- C -- D -- E The call from D to E uses the RESUMEABLE label, and E stores the resulting continuation in a global, and everything up to main returns. Then, main invokes the continuation, and we find ourselves back in D. Now, C, B, and A will return again, without any compile-time hint. That's fine. The return is an expected operation. I don't think that's the problem. The error in gc_13 is a call path like: choose() - try (with_cc) - fail() - | (choose again) - + --+ | choose() - try (with_cc) - fail() -| || (choose again) - +| | fail() --+ The problem now is not per se the path in main from the two choose() calls down to fail is executed more then once (as it's the case with multiple returns). The problem is the loop in main. By denoting the loop from the call to fail() to the first choose() with some kind of syntax, the register allocator does the right thing. But consider even this simple function: sub B { a = 1 foo() print a b = 2 return b } If something called by foo() captures a continuation, and something invokes it after B() returns, then there's a hidden branch, in effect, from the return to the print, isn't there? The register allocator could decide to use the same register for a and b, but then the second return from foo() would print 2 instead of 1, which is wrong. And the author of B(), of course, may have no idea such a thing would happen, so wouldn't be able to supply any syntax to tell the compiler. I'm just trying to come up with a simpler example, since it seems to me that there's a problem any time a function returns, and the last thing that executed in that frame wasn't a call to that function. (There's a lot going on in the gc_13 example, and I think some of it is distracting from the main point.) But a RESUMABLE label seems like the information that's needed by the compiler. But on the other hand in an example like the above, the function B() may not be written to expect foo() to be resumed. So, what should happen at runtime, if the label is absent? We could declare undefined behavior, but that would mean that for defined behavior, you'd need the RESUMABLE label all the way up the stack, and Ruby and Scheme don't have that syntactic constraint. With Scheme, it's only clear from the syntax what's going on locally--but you can invoke a continuation far from any call/cc, if that continuation was stored away into a variable. JEff
Re: Continuations, basic blocks, loops and register allocation
On Nov 16, 2004, at 10:03 AM, Matt Fowles wrote: Since both you and Leo are arguing against me here, it seems like that I am wrong, but I would like to figure out exactly why I am wrong so that I can correct my train of thought in the future. Here's a real example you can play with, if you have Ruby installed: % cat continuation6.ruby def strange callcc {|continuation| $saved = continuation} end def outer a = 0 strange() a = a + 1 print a = , a, \n end # these two lines are main outer() $saved.call % ruby continuation6.ruby a = 1 a = 2 a = 3 a = 4 a = 5 a = 6 a = 7 a = 8 a = 9 a = 10 ...infinite loop, by design What happens when the program runs is that outer() is called (only once) which creates a continuation (inside of strange()), increments a, prints and returns. The next thing that happens is that the continuation is invoked. Control jumps to the location in strange() right after the callcc line, then that return and we are at the line in outer() where 'a' is incremented. So 'a' increments from the last value it had in that frame (since we are magically back again inside of the same single invocation of outer()), then 'a' is printed and outer() returns again (note: outer only called once, returned twice so far), and then we call the continuation again, and start the loop over. We only ever create one continuation in this example, since we only ever call strange() once. The continuation preserves the frame (the mapping from logical variables to their values), but not the values of those variables at the time the continuation was created. In effect, I think the continuation is arranging to preserve the state of the variables as they were when code in the frame was last executed, rather than at the time the continuation was created. The behavior you were describing is what I had thought would happen, but then I realized I wasn't sure, so I confirmed that it wasn't. The above is the behavior of Ruby, and I believe Scheme works the same way. What you described would be useful for backtracking (jumping back not only to a previous location in a computation, but also its previous state), but it's not what these languages seem to do. JEff
Re: Continuations, basic blocks, loops and register allocation
On Nov 14, 2004, at 3:03 AM, Leopold Toetsch wrote: Matt Fowles [EMAIL PROTECTED] wrote: Yes, but in the case of the continuation resuming after foo, the continuation should restore the frame to the point where it was taken. Thus all of the registers will be exactly as they were when the continuation was taken (i.e. in the correct place). Yes, but Jeff's example wasn't really reflecting the problem. How come? (Not quibbling--just afraid I'm missing something.) It seems that even this function body shows the problem: a = 1 foo() print a b = 10 return b It would seem (w/o continuations) that b should be able to re-use a's register, but if it does then we'll print 10 instead of 1 the second time. So what to do: 1) Extending register frame size ad infinitum and never reuse a Parrot register will definitely blow caches. 2) Generating an edge for every call to every previous calls will blow the CFG and cause huge pressure on the register allocator. A lot of spilling will be the result. 3) Using lexicals all over is slower (but HLL compilers will very likely emit code that does exactly that anyway). So the problem may not be a real problem anyway. We just know that an optimizer can't remove the refetch of lexicals in most of the subroutines. It seems that, in term of cache locality, the same problem is there with more registers v. spilling v. lexicals. That is, if you have 100 local variables whose lifetimes overlap (due to continuations), then you need 100 distinct memory locations to (repeatedly) access. 4) Having an explicit syntax construct (call-with-current-continuation that expresses verbatim, what's going on, like e.g. with a reserved word placed as a label: RESUMEABLE: x = choose(arr1) I don't think that really helps either: If you have such a call, then all the frames higher up the stack also can return multiple times, so they have the behavior, even w/o the label. On the other hand, this Ruby code really bugs me (note: $ variables in Ruby are globals): % cat continuation5.ruby def strange callcc {|continuation| $saved = continuation} end def outer a = 0 strange() a = a + 1 print a = , a, \n a = hello print a = , a, \n end outer() $saved.call % ruby continuation5.ruby a = 1 a = hello continuation5.ruby:8:in `+': failed to convert Fixnum into String (TypeError) from continuation5.ruby:8:in `outer' from continuation5.ruby:14 What bugs me is that outer gets an error when the continuation is invoked, because the second time strange() returns, a is a string and so you can't add 1 to it. But looking at the definition of outer, you'd expect that you could never get such an error. (Without the line setting a to hello, you get an infinite loop, printing increasing integers.) JEff
Re: Continuations, basic blocks, loops and register allocation
On Nov 15, 2004, at 3:27 AM, Leopold Toetsch wrote: Jeff Clites [EMAIL PROTECTED] wrote: On Nov 14, 2004, at 3:03 AM, Leopold Toetsch wrote: Yes, but Jeff's example wasn't really reflecting the problem. How come? Your code didn't reuse Ca after the call. Oops. It seems that, in term of cache locality, the same problem is there with more registers v. spilling v. lexicals. Not quite. We can't extend just one register set, we'd do that for all 4 kinds. You can not easily have 64 PMC registers and just 10 INTVALs. A lexical access is just an array lookup (at least if the compiler uses the indexed addressing of lexicals). Ah. What I don't like, though, is that in the in the lexical case you have things sitting in an array, from which you need to move things back-and-forth to registers to work on them. In the more registers case, they're sitting in a quite similar array, but you can work on them directly there. Adding 2 such INTVALs is one op, instead of 4 (2 loads, 1 add, 1 store). Since the problem exists for all register types (unless HLLs only use PMCs, which is possible), then we either need 4 lexical arrays for maximum locality (so that they can be sized independently), or we need to be able to size the register frames independently for the 4 types. (I realize that currently lexicals must be PMCs, but it seems we have the same issue with other reg. types.) ... That is, if you have 100 local variables whose lifetimes overlap (due to continuations), then you need 100 distinct memory locations to (repeatedly) access. Sure. If the program is complex you need the storage anyway. But I think the real problem is that it's only due to CPS that the lifetimes are overlapping so much--that's what's biting us. (By expanding my example code, you could come up with simple code which uses 100 locals be would only need 1 register w/o continuations, and needs 100 spots with them.) It's just a pretty unfortunate price we're paying, if the feature is not extensively used. Now we're just figuring out how to survive it. 4) Having an explicit syntax construct (call-with-current-continuation that expresses verbatim, what's going on, like e.g. with a reserved word placed as a label: RESUMEABLE: x = choose(arr1) I don't think that really helps either: If you have such a call, then all the frames higher up the stack also can return multiple times, so they have the behavior, even w/o the label. The RESUMABLE label is of course at the invocation that might resume, somehwere up the call chain. Again: the HLL compiler (or the programmer) knows the effect of the continuation. Just the PIR code is too dumb, i.e. is lacking this information. Picture this call stack: main -- A -- B -- C -- D -- E The call from D to E uses the RESUMEABLE label, and E stores the resulting continuation in a global, and everything up to main returns. Then, main invokes the continuation, and we find ourselves back in D. Now, C, B, and A will return again, without any compile-time hint. On the other hand, this Ruby code really bugs me (note: $ variables in Ruby are globals): ... , you get an infinite loop, printing increasing integers.) Sure. With callcc you are calling the function again and again. I know--the infinite loop was the desired behavior (just mentioned it so that people wouldn't have to run it). What bugs me is that you can get that error, though looking at the code it should be impossible. The author of outer() might have no clue that could happen, so it's not really his bug, and the person using a continuation needs really detailed knowledge of everything in the call stack, to know if it will work. I guess that's just how it is, but it seems to mean that continuations have limited usefulness in languages with side-effects, except for very local usage (breaking out of a loop and such). JEff
Re: Continuations, basic blocks, loops and register allocation
On Nov 14, 2004, at 1:53 PM, Dan Sugalski wrote: Since, for example, it's completely reasonable (well, likely at least) for a called sub to rebind lexicals in its parent What does that mean, exactly? It seems like that directly contradicts the meaning of lexical. For instance, see Larry's comments from Re: Why lexical pads at September 25, 2004 10:01:42 PM PDT (the first of the 3 messages from him on that day). JEff
Re: Continuations, basic blocks, loops and register allocation
On Nov 13, 2004, at 8:53 AM, Leopold Toetsch wrote: 2) Continuations (t/op/gc_13.imc [Hi Piers]) Again we have a hidden branch done by a Continuation, or better a loop. The control flow of the main program is basically represented by this conventional code fragment: arr1=[...]; arr2=[...] loop1: x = choose(arr1) loop2: y = choose(arr2) ... failed = fail() goto (loop1, loop2)[failed] except that the gotos are performed by backtracking via the continuations. So we don't have these loop labels and the continuation continues at the next opcode after the invocation of choose() and not at the shown position above. So again, the register allocator doesn't see that there is a branch, the variable's arr2 register is clobbered, in this case by the fail closure. As we never know, if a subroutine captures the return continuation and creates such loops like above, we have in the absence of other syntax actually no chance to hold any variable in a register as far as I can see now. We'd have just to force using lexicals for all vars, except for leaf subroutines (that don't call other subs) or subs that just call one sub (they can't create loops). Another idea is to create edges from all 1..n function calls in a sub to the previos 0..n-1 calls, so that basically all possible loops done via possible continuations are represented in the CFG. That analysis looks correct to me--any time you have a function call, the subsequent op might be a branch target, if there is a subsequent function call. We'd have just to force using lexicals for all vars Having variable-size register sets would solve this, since you could have fixed assignments of variables to registers, and not have to suffer the overhead of moving data between registers and lexical pads over-and-over. Well, it doesn't really solve it--just makes it workable. JEff
Re: Continuations, basic blocks, loops and register allocation
On Nov 13, 2004, at 11:16 AM, Matt Fowles wrote: All~ On Sat, 13 Nov 2004 10:52:38 -0800, Jeff Clites [EMAIL PROTECTED] wrote: On Nov 13, 2004, at 8:53 AM, Leopold Toetsch wrote: We'd have just to force using lexicals for all vars Having variable-size register sets would solve this, since you could have fixed assignments of variables to registers, and not have to suffer the overhead of moving data between registers and lexical pads over-and-over. Well, it doesn't really solve it--just makes it workable. I like the idea of mandating lexicals vars. This would also eliminate the need for spilling (I think), as the register allocator would only need to refetch the lexical rather than save it off somewhere to be restored later. In a way I feel like they're both same thing, just under a different description: spilling means moving data back-and-forth between registers and some other storage, and so does using lexicals. But the only reason we have to do that sort of dance (under either description) is because we are RISC-ish: We have a limited number of registers, and calculations can only target registers (that is, you can't add 2 numbers directly in a lexical pad or other storage--they have to be moved to registers first). You don't have to move data back-and-forth if either you have an unlimited number of (preserved) registers, or you allow calculations to act directly on other memory locations. And I think this is again just two different ways of describing the same thing: you have an unlimited number of stable storage locations, and you do calculations directly from those locations. It's just that the former (unlimited preserved registers) feels cleaner, and doesn't require an explosion of op variants. That's oversimplifying a bit, but I feel like those are the core issues (stemming from the observation of Leo's that continuations in effect give all variables a lifetime that extends to their entire block, in most cases). JEff
Re: Continuations, basic blocks, loops and register allocation
On Nov 13, 2004, at 2:46 PM, Matt Fowles wrote: On Sat, 13 Nov 2004 14:08:12 -0800, Jeff Clites [EMAIL PROTECTED] wrote: That's oversimplifying a bit, but I feel like those are the core issues (stemming from the observation of Leo's that continuations in effect give all variables a lifetime that extends to their entire block, in most cases). This does not give all variables extended lifetimes. It only gives variables that are used in the exception handler such a lifetime. Thus temporaries used in calculations can be safely overwritten. Perhaps we should try adding the control flow arcs to the basic block analysis and see how the allocator handles it then... Not all variables, but due to Leo's case (2), it should be all variables which are referenced after the first function call out of a subroutine, if there's a later function call; for instance, consider: ... foo(); a = 10; b = 12; ... use a and b bar(); ... never use a or b again c = 1; d = 2; ... use c and d baz(); ... never use c or d again zoo(); Normally, the lifetime of a and b would end at the call to bar, and that of c and d would end at the call to baz, but due to continuations, the call to zoo might resume at the op right after the call to foo (since the continuation created when calling foo may have been captured), so a,b,c,d have to persist at least until the last function call is made. We could teach the allocator about these possibilities as you mentioned, and that would give us correct program behavior, but we know a priori that we'll have much higher register pressure that you might expect, so a different overall strategy might work better. JEff
Re: Tail calls and continuations
On Nov 10, 2004, at 11:53 PM, Leopold Toetsch wrote: Jeff Clites [EMAIL PROTECTED] wrote: ...it sounds like we have an easy way to tell if a real continuation has a claim on the register frame, because creating such a real continuation can mark the frame, There is no such mark. If ctx-current_cont isa(RetContinuation), then it's save to do the tail call. Good--implicit mark then. This OTOH is meaning that we can do the check only at runtime. Thus the Ctailcall or Ctailcallmethod opcodes have to do plain calls if they detect such a situation. Yep, although there will be some situations where we can know for sure at compile time--for instance if all function calls within a function are tail calls. (We could have a special sort of Ctailcalldontcheck, but that depends on our philosophy--if it's okay for the VM to trust the compiler. Or possibly the VM could detect this, and cache this information as a sort of dynamic optimization; or a bytecode verifier could ensure that Ctailcalldontcheck is valid in a given context. Multiple options here.) And also, even when Ctailcall has to fall back to a plain call, it doesn't have to fall all the way back--it can still pass along the return continuation from its caller, and get some benefit. ... (In fact that mark should be a reference count, That's really not needed. If you return from the function and you call it next time, you've again a RetContinuation. If the continuation was created somewhere deeper in the call chain, it's gone or not after the GC cycle. But if ctx-current_cont has been promoted to a real continuation (as a result of something that happened deeper in the stack), it will never be turned back to a RetContinuation, even if it could have been (ie, if GC reclaimed the things that caused the promotion), so we might forego a tail call that we could have made. And you know - starting with refcounting one objects ends up with refcounting containers holding that item ... Not really. In this case, the only things which are allowed to point to a register frame (via a ctx) are the interpreter itself, and continuations. There are just a couple of specific points where the ref count would need to be incremented/decremented (including the destroy() of continuations). But that's the clear place to stop--you don't need to ref count the continuations themselves, since that wouldn't be practical (you'd need write barriers to always check if you're referencing or un-referencing a continuation, etc.). We wouldn't have to go off the deep end with it. JEff
Re: #line directives from PMC file should now work
On Nov 11, 2004, at 6:53 AM, Leopold Toetsch wrote: Nicholas Clark [EMAIL PROTECTED] wrote: Builds pass with the --no-lines option in Makefile removed. Should this be removed from the template Makefile so that all builds now use #line directives? Yep. Is there still that %ENV var around to turn line numbers off? - Or was that in ops2c.pl - PARROT_NO_LINE is it, could be handy sometimes. I'd really like a way to turn them off easily (for the ops as well, actually). I find them to be counterproductive (for our stuff), since what gets shown in the debugger isn't stuff you can actually get gdb to evaluate. JEff
Re: Tail calls and continuations
On Nov 11, 2004, at 9:44 AM, Michael Walter wrote: On Thu, 11 Nov 2004 12:30:16 -0500, Dan Sugalski [EMAIL PROTECTED] wrote: Tail calls should be explicit, compile-time things. Otherwise we're going to run afoul of traceback requirements and suchlike things Nah, that's just the difference between running optimized and unoptimized. Actually, with a tailcall op that's effectively a hint, it would be ideal to have 3 run modes: a just obey the hints mode (for those that trust their compiler), a never do a tail call mode (for debugging purposes), and a do tail calls whenever possible mode (independent of whether the tailcall op was used). Not sure if those are run modes or assembler (optimizer) modes, though. Besides, it's a lot easier in general for a language compiler to decide when a tail call's in order than it is for us. I think it's pretty straightforward to tell, at least at the PIR level. Even at the pasm level, the only requirement is that there's effectively no code between a call from foo to bar, and a return from foo. It's a pure optimization--good fodder for an optimizer. Even further, it's necessary for some languages (Scheme)/paradigms (loop by recursion) that a tailcall is not just a hint but mandatory. I think that actually doesn't matter. Even in the case where we think we can't do a full tail call optimization (because of a continuation that's been taken), we can still actually remove the calling frame from the call stack--we just can't immediately re-use the register frame. That satisfies the Scheme requirement, I would think. You can still do unbounded recursion, just that GC may need to run to clean up call frames. Leo said: $ time parrot -j fact.imc 1 # [1] maximum recursion depth exceeded I'd think that long-term our max recursion depth limit should only apply to net frame depth--tail calls shouldn't increase the count. (Probably we'd need 2 counts--net depth and logical depth.) JEff
Re: Tail calls and continuations
On Nov 11, 2004, at 11:24 AM, Dan Sugalski wrote: At 11:16 AM -0800 11/11/04, Jeff Clites wrote: On Nov 11, 2004, at 9:44 AM, Michael Walter wrote: On Thu, 11 Nov 2004 12:30:16 -0500, Dan Sugalski [EMAIL PROTECTED] wrote: Even further, it's necessary for some languages (Scheme)/paradigms (loop by recursion) that a tailcall is not just a hint but mandatory. I think that actually doesn't matter. Even in the case where we think we can't do a full tail call optimization (because of a continuation that's been taken), we can still actually remove the calling frame from the call stack--we just can't immediately re-use the register frame. That satisfies the Scheme requirement, I would think. You can still do unbounded recursion, just that GC may need to run to clean up call frames. I only skimmed the earlier parts of this, but continuations shouldn't affect tail calls at all. You should read the thread then. If this is from some side effect of call speed optimization (I remember seeing some discussion of stack allocation of call frames or something, but I don't recall if it was before or after I said we weren't optimizing this stuff right now) then we need to rip out those optimizations. Tail call optimization *is* and optimization That's what the whole feature is. And there's a difference between a pure optimization (which increases speed at the cost of design), and an architectural feature which improves speed. JEff
Re: Tail calls and continuations
On Nov 11, 2004, at 12:59 PM, Leopold Toetsch wrote: Jeff Clites [EMAIL PROTECTED] wrote: I think that actually doesn't matter. Even in the case where we think we can't do a full tail call optimization (because of a continuation that's been taken), we can still actually remove the calling frame from the call stack--we just can't immediately re-use the register frame. As Dan already has outlined a Continuation doesn't have any impact on tail calls (my argument WRT that was wrong) I'm confused then. What from the previous discussion in this thread was incorrect? JEff
Re: #line directives from PMC file should now work
On Nov 11, 2004, at 11:12 AM, Leopold Toetsch wrote: Jeff Clites [EMAIL PROTECTED] wrote: I'd really like a way to turn them off easily (for the ops as well, actually). I find them to be counterproductive (for our stuff), since what gets shown in the debugger isn't stuff you can actually get gdb to evaluate. It depends. While hacking PMC files its much more useful to have the gcc error line within the pmc files. Debugging ops is sometimes more intuitive, if you see the corresponding .c file in the debugger, but not always. Sure--definitely depends on what you're doing. For me, I'm often debugging a JIT problem, and when I end up in a normal-op call I need to be able to examine the environment (registers, subsequent ops). Anyway, as $ENV{PARROT_NO_LINE} is already in ops2c.pl, it should be present in all build tools. Or maybe also CLI options to the tools, so at least you can hand-modify the Makefile to easily turn off some, and leave on other. Having a Configure.pl parameter is also more consistent with everything else we do, compared to an env. variable. JEff
Tail calls and continuations
I was thinking: Implementing tail calls seems easy; the normal calling sequence of do some setup, then jump just turns into don't bother with (most of) the setup, just jump. That is, don't move a new register base-pointer into place, etc. But there's one wiggle: If you've created a continuation previously (and it still exists), then any call has to preserve the frame--you basically can't do a tail call, with its key implication of the current frame vaporizing, or being re-used (depending on how you want to describe it). But that's not too much of a problem, with the following: 1) Consider a tailcall op a recommendation--but have the VM do a regular call, if necessary. 2) Regular calls create continuations, so you can't do a tail call out of a function, if you've already done a regular call inside that function, _unless_ we have an (efficient) way to tell if any such continuation was saved. You can figure some of that out at compile-time (whether a regular call could have been already made), but you'd need runtime checks for other cases, unless you just forego a tail call any time you _could_ have already done a regular call (which avoids the runtime checks, but allows less actual tail calls). Do any existing languages have both tail calls and continuations? Scheme mandates that anything which looks like a tail call, must be optimized as expected (other languages like Lisp merely permit it), but I don't know of Scheme having continuations. Scheme cares, of course, so that you can have ostensibly unlimited recursion, without running out of stack space (or really, memory). Tail calls and continuations seem a bit like opposites--one preserves state, the other destroys it. Just thought I'd send out these thoughts, since the topic was mentioned recently. JEff
Re: Tail calls and continuations
On Nov 10, 2004, at 3:08 PM, Leopold Toetsch wrote: Jeff Clites [EMAIL PROTECTED] wrote: But there's one wiggle: If you've created a continuation previously (and it still exists), then any call has to preserve the frame Well, that's what continuations are doing. They do preserve the frame, which they have taken a snapshot of. And they preserve any frames up the call chain. This is needed and implemented and working. But here's the part I'm thinking about: Continuations only copy the interpreter context, which contains the register bp, but of course the actual contents of the register frame are not duplicated. (This is as it should be.) The contents of the register frame are effectively preserved because in sub-invoke() we allocate a new register frame, and change the bp to point there. (That's fine too.) But, in a tail-call-optimization case, we don't need to call new_register_frame() and copy_regs()--ex hypothesi, we can re-use the register frame already in-place. That's a big savings--that's the optimization I'm after. But of course, we can only do that if we know that a real continuation hasn't also captured the context. But in light of what you say here... The concept of a return continuation (which recycles itself and the frame it returned from) doesn't have an influence on tail calls. Whenever you see a continuation on the surface, it's one that is preserving the frames and the context. Eventually there isn't even a RetContinuation object but just a pointer to the calling frame. But whenever you can see a continuations it's a real one ...it sounds like we have an easy way to tell if a real continuation has a claim on the register frame, because creating such a real continuation can mark the frame, and we can check for the mark in our tail-call implementation, and if it's marked then fall back to new_register_frame/copy_regs. (In fact that mark should be a reference count, keeping track of how many continuations have a claim on the register frame. That way, if a real continuation is created, but GC claims it before our tail-call attempt, we can still use our optimization.) But maybe we're already doing something like that, and I missed it. AFAIK the only problem with tailcalls and tail-recursive functions is to prperly detect them in some optimizer, or to convert to such tailcalls. Seems like that shouldn't be too bad--we only need to know that there's effectively no code between the function call and subsequent return. (Though maybe telling call from return will be tricky.) JEff
Re: Basic compilation example (a + b)?
On Nov 8, 2004, at 11:42 PM, Leopold Toetsch wrote: Jeff Clites wrote: new P16, 32 # .PerlInt add P16, P18, P17 That's what worries me, and what prompted the question. You don't know at compile-time that the return type should be a PerlInt. Yes, I've already stated that this needs fixing and I've proposed a scheme how to fix it. Fine then, but in the scheme you recently mentioned, you were explicitly null-ing out the return register. That shouldn't be needed--unnecessary overhead. That is, unless we're assuming the semantics of references types--that the return register would hold a reference object to store into, always. But I don't think we can mix the behaviors--unless there's some special flag on a PMC to indicate that it's a reference type, but that seems awkward. But since the pie-thon output doesn't do that, I'll assume the plan is not to use such reference types, if that's meant to be the canonical example. JEff
Re: calling conventions, tracebacks, and register allocator
On Nov 8, 2004, at 11:15 AM, Matt Fowles wrote: Dan~ On Mon, 8 Nov 2004 13:45:08 -0500, Dan Sugalski [EMAIL PROTECTED] wrote: The calling conventions and code surrounding them will *not* change now. When all the sub stuff, and the things that depend on it, are fully specified and implemented... *then* we can consider changes. Until then, things stand the way they are. I missunderstood. I though you were saying that what is currently in is final and will *never* be changed. Thanks for the clarification. Nevertheless, this is a legitimate topic for discussion, and the issues are fresh in people's minds. That's independent of any impediments that might block implementing changes at the current time. JEff
Re: No Cpow op with PMC arguments?
On Nov 8, 2004, at 3:08 AM, Leopold Toetsch wrote: Jeff Clites [EMAIL PROTECTED] wrote: No. The binary operations in Python are opcodes, as well as in Parrot. And both provide the snytax to override the opcode doing a method call, that's it. I guess we'll just have to disagree here. I don't see any evidence of this UTSL please. The code is even inlined: ,--[ Python/ceval.c ] | case BINARY_ADD: | w = POP(); | v = TOP(); | if (PyInt_CheckExact(v) PyInt_CheckExact(w)) { | /* INLINE: int + int */ | register long a, b, i; | a = PyInt_AS_LONG(v); | b = PyInt_AS_LONG(w); | i = a + b; | if ((i^a) 0 (i^b) 0) | goto slow_add; | x = PyInt_FromLong(i); ` But I said, from an API/behavior perspective. How the regular Python interpreter is implemented isn't the point--it's how the language acts that's important. And I can't think of any user code in which a+b and a.__add__(b) act differently, and I think that intentional--an explicit languages design decision. The impl. above of BINARY_ADD is most likely an optimization--the code for BINARY_MULTIPLY (and exponentiation, division, etc.) looks like this: case BINARY_MULTIPLY: w = POP(); v = TOP(); x = PyNumber_Multiply(v, w); Py_DECREF(v); Py_DECREF(w); SET_TOP(x); if (x != NULL) continue; break; And again, what about Ruby? If you believe in matching the current philosophy of the language, it won't use ops for operators (but rather, method calls), and won't do the right thing for objects with vtable/MMDs, and no corresponding methods. Not actually MMD in Python--behavior only depends on the left operand, it seems. It's hard to say what Python actually does. It's a mess of nested if's. Just look at the behavior--that's what's important: Behavior depends only on left operand: == class Foo: ... def __add__(a,b): return 7 ... == x = Foo() == x + x 7 == x + 3 7 == x + b 7 == x + (1,2) 7 All the following are error cases. Error statement varies depending on left operand only: == 3 + b Traceback (most recent call last): File stdin, line 1, in ? TypeError: unsupported operand type(s) for +: 'int' and 'str' == 3 + x Traceback (most recent call last): File stdin, line 1, in ? TypeError: unsupported operand type(s) for +: 'int' and 'instance' == 3 + (1,2) Traceback (most recent call last): File stdin, line 1, in ? TypeError: unsupported operand type(s) for +: 'int' and 'tuple' == b + 3 Traceback (most recent call last): File stdin, line 1, in ? TypeError: cannot concatenate 'str' and 'int' objects == b + x Traceback (most recent call last): File stdin, line 1, in ? TypeError: cannot concatenate 'str' and 'instance' objects == b + (1,2) Traceback (most recent call last): File stdin, line 1, in ? TypeError: cannot concatenate 'str' and 'tuple' objects == (1,2) + 3 Traceback (most recent call last): File stdin, line 1, in ? TypeError: can only concatenate tuple (not int) to tuple == (1,2) + b Traceback (most recent call last): File stdin, line 1, in ? TypeError: can only concatenate tuple (not str) to tuple == (1,2) + x Traceback (most recent call last): File stdin, line 1, in ? TypeError: can only concatenate tuple (not instance) to tuple null dest dest = l + r should produce a *new* dest PMC. Yes, it's a separate issue, but it's pointing out a general design problem with these ops--their baseline behavior isn't useful. It *is* useful. If the destination exists, you can use it. The destination PMC acts as a reference then, changing the value in place. But in case of Python it's not of much use Right, changing the value in-place would do the wrong thing, for Python. (It depends on whether the arguments to the op are references, or the actual values. If they're references, then it can work correctly, but then we don't want to be MMD dispatching on the (reference) types, but rather on the types of what they're pointing to.) except for the inplace (augmented) operations. Yes, but that ends up being just for the two-argument forms (and even those don't work for Python--a += 3 doesn't really update in-place in Python, but returns a new instance). In-place operators tend to only take one argument on the right side, so the p_p_p forms aren't useful for this. ..., but for PMCs this could compile like a = b.plus(c). but you don't need add_p_p_p, just method invocation. Why should we do method invocation with all it's overhead, if for the normal case a plain function call we'll do it? Ah, that's the key. Method invocation
Register allocation/spilling and register volitility
From other threads: Now we are placing arguments or return values in registers according to PDD03 and the other end has immediately access to the placed values, because the register file is in the interpreter. With the indirect addressing of the register frame, this argument passing is probably the most costly part of function calls and returns, because we have to copy the values. and Anyway, if you can pop both register frames -and- context structures, you won't run GC too often, and everything will nicely fit into the cache. I thought about that too, but I'd rather have registers adjacent, so that the caller can place function arguments directly into the callers in arguments. OTOH it doesn't really matter, if the context structure is in the frame too. We'd just need to skip that gap. REG_INT(64) or I64 is as valid as I0 or I4, as long as it's assured, that it's exactly addressing the incoming argument area of the called function. A problem with this is that you can't be sure that you can actually have the next frame of registers adjacent to the current frame--they might already be taken. Imagine A calls B, then B creates a continuation and stores it in a global, then returns. If A makes another function call, it can't just use the adjacent register frame--it's still taken. (I'm referring here to the sliding register window idea of course.) But this is reminding me of another idea I've had. I've been thinking for a while about another idea to decrease some of the copying which we are now doing for function calls, as well as the allocation of register sets during subroutine calls, and which would as a side-effect allow us to reduce the need for register spilling. First, a code snippet. Consider the following C code: int a(int x) { return b(x); } int b(int x) { return c(x + 1); } int c(int x) { return x + 7; } As compiled on the PPC, this code is very compact--each function is implemented by just one or two instructions, and only one register is used. There are two key factors here: (1) no register preservation is needed, and (2) the call from a() to b() is just a branch, since the call to a() has already loaded the registers with the correct values for the call to b(). By my thinking, register usage falls into three basic categories: 1) Registers used for parameter passing and returns values for function calls. 2) Registers used to hold values which need to be preserved across function calls. 3) Registers used to hold values which do not need to be preserved across function calls. Really, (1) and (3) are similar--in either case, you have registers whose values are allowed to change across function calls. (For register which hold return values that's obvious, and for those used to pass parameters, it seems fair that these be expected to change across a function call.) So we really have two cases--volatile and non-volatile register usage. In the PPC ABI, half the registers are treated as volatile, and the other half as non-volatile. For the PPC, this corresponds to caller-preserved v. callee-preserved. Because of continuations[1], parrot can't have callee-preserved registers, but it could still have volatile v. non-volatile registers. Here's what I'm thinking: Keep the old-scheme registers inside the interpreter structure, *as well as* the new indirect registers. Call the registers in the interpreter I0 to I31, and the indirect registers I32 to whatever.[2] I'll call these the direct and indirect registers. By their very nature, the direct registers are volatile, and the indirect registers are non-volatile. What does this buy us? The following: 1) Parameter passing and return occur via the established calling conventions, in the volatile registers. No extra copying is needed upon function call or return--the volatile registers are physically the same for the caller and the callee. 2) Temporary calculations can use the volatile registers. Values which need preserving can use the non-volatile registers directly. 3) In functions which need no non-volatile registers, there's no need to allocate the indirect registers at all. 4) Cases like my example code above can compile just as compactly as the PPC case. With the volatile Parrot registers mapped to volatile CPU registers (in the PPC case at least), this would be highly efficient: you'd end up with asm similar to what you have in the C case above, and you'd not have a need to save the CPU registers back to Parrot registers (other than those used in the calling conventions) for Parrot function calls out of JIT code. 5) Because this opens things up to more than 32 registers, code could use as many registers as needed. You'd never have register spilling per-se in order to accommodate local variables, though you'd still want a smart register allocator which minimized the number of registers needed (for memory locality as well as minimizing allocation). But a very simple
Re: No Cpow op with PMC arguments?
On Nov 8, 2004, at 12:50 AM, Leopold Toetsch wrote: Jeff Clites [EMAIL PROTECTED] wrote: On Nov 5, 2004, at 9:40 AM, Leopold Toetsch wrote: In Python, semantically you know that you'll end up doing a method call (or, behaving as though you had), so it's very roundabout to do a method call by using an op which you know will fall back to doing a method call. Clearer just to do the method call. No. The binary operations in Python are opcodes, as well as in Parrot. And both provide the snytax to override the opcode doing a method call, that's it. I guess we'll just have to disagree here. I don't see any evidence of this from an API/behavior perspective in Python. I think the existence of a separate Python opcode is just a holdover from a time when these infix operators only existed for built-in types (just inferring this). I can't find any case where Python would act differently, if these were compiled directly to method calls. And for Ruby, it's *explicit* that these operators are just method calls. And languages like Java don't have these operators at all, for objects. The only thing that's special is that there are certain built-in classes, and some of them implement __pow__, but that's not really anything special about __pow__. Yes. And these certain *builtin* classes have MMD functions for binary opcodes. Not actually MMD in Python--behavior only depends on the left operand, it seems. And even the ops we currently have are broken semantically. Consider a = b + c in Python. This can't compile to add_p_p_p, because for that op to work, you have to already have an existing object in the first P register specified. But in Python, a is going to hold the result of b + c, which in general will be a new object and could be of any type, and has nothing to do with what's already in a. That's a totally different thing and we have to address that. I have already proposed that the sequence: null dest dest = l + r should produce a *new* dest PMC. That's quite simple. We just have to pass the address of the dest PMC pointer instead of the PMC to all such operations. Warnocked. Yes, it's a separate issue, but it's pointing out a general design problem with these ops--their baseline behavior isn't useful. The result of l + r will not depend on what's to the left of the = by HLL semantics, for any case I can think of. (Perl cares about context, but that's not really the same thing.) ... I think we should create PMC-based ops only if one of the following criteria are met: (a) there's no other reasonable way to provide some needed functionality, So, this is already a perfect reason to have these opcodes with PMCs. a = b + c behaves differently, if b and c are plain (small) integers or overflowing integers or complex numbers and so on. You can't provide this functionality w/o PMCs. I don't understand this example. Certainly you need PMCs, but if b and c are I or N types, of course you'd use add_i_i_i or add_n_n_n, but for PMCs this could compile like a = b.plus(c). Of course you have to know I v. N v. P at compile-time, and there's no reason that I/N v. P pasm must look identical, for similar-looking HLL code. You need PMCs, but you don't need add_p_p_p, just method invocation. For complex numbers and such, I'd want to be able to define classes for them in bytecode. For that to work, ops would eventually have to resolve to method calls anyway. (You can't create a new PMC and vtable/MMDs in bytecode.) Why not skip the middle-man? JEff
Re: Register allocation/spilling and register volitility
On Nov 8, 2004, at 1:34 AM, Leopold Toetsch wrote: Jeff Clites [EMAIL PROTECTED] wrote: OTOH it doesn't really matter, if the context structure is in the frame too. We'd just need to skip that gap. REG_INT(64) or I64 is as valid as I0 or I4, as long as it's assured, that it's exactly addressing the incoming argument area of the called function. A problem with this is that you can't be sure that you can actually have the next frame of registers adjacent to the current frame--they might already be taken. Imagine A calls B, then B creates a continuation and stores it in a global, then returns. Please read the proposal summary by Miroslav Silovic, keyword watermark. If frames aren't adjacent, normal argument copying can be done anyway. This would seem to require the same types of runtime checks that you are objecting to below, unless user code is expected to explicitly check whether it's supposed to be assigning to I3 v I(3 + 32x?). And that still seems to require copying, in my case of a function a() which calls a function b() with the exact same arguments. Keep the old-scheme registers inside the interpreter structure, *as well as* the new indirect registers. Call the registers in the interpreter I0 to I31, and the indirect registers I32 to whatever. That would need two different addressing modes depending on the register number. That'll lead to considerable code bloat: we'd have all possible permutations for direct/indirect registers. Doing it at runtime only would be a serious slowdown. I discussed that in item (1) under Further notes. It's not needed. I've a better scheme in mind, which addressess efficieny as well as argument passing. And spilling? JEff
Re: Basic compilation example (a + b)?
On Nov 8, 2004, at 2:47 AM, Leopold Toetsch wrote: Jeff Clites [EMAIL PROTECTED] wrote: What pasm is supposed to correspond to this snippet of Python code (assume this is inside a function, so these can be considered to be local variables): a = 7 b = 12 c = a + b Run it through pie-thon. It should produce some reasonable code. For leaf-functions (w/o introspection i.e. calls to locals()), the lexical handling would get dropped. And a better translator would use lexical opcodes by index and not by name. It doesn't do-the-right-thing in the cases I'm interested in: % cat pythonClass.py class A: def __add__(x,y) : return boo x = A() y = x + 3 print y % python pythonClass.py boo % perl pie-thon.pl pythonClass.py | ./parrot --python - Can't find method '__get_number' for object 'py::A' It looks like it's trying to get the float-value of x, rather than calling x's __add__ method. But the part I was really wondering about is the a + b. This is what pie-thon.pl produces for that (you can just run it on the code fragment a + b--it doesn't matter the context): $P1 = new PerlInt # BINARY_ADD $P1 = a + b corresponding pasm: find_global P18, a find_global P17, b new P16, 32 # .PerlInt add P16, P18, P17 That's what worries me, and what prompted the question. You don't know at compile-time that the return type should be a PerlInt. It could be anything--it's really up to a. This is regarding my concern that the p_p_p ops aren't very useful (in Python at least), and I can't figure out what we should be using instead. So I'm still left wondering, how *should* this compile? JEff
Re: No Cpow op with PMC arguments?
On Nov 5, 2004, at 9:40 AM, Leopold Toetsch wrote: Jeff Clites [EMAIL PROTECTED] wrote: a) As Sam says, in Python y**z is just shorthand for y.__pow__(z)--they will compile down to exactly the same thing (required for Python to behave correctly). I don't think so (and you can replace with add, sub, ... FWIW). All these binops compile to Parrot opcodes. I'm saying that, by Python semantics they do the same thing--it's an open question how they should compile, that's what we're discussing. It is true that in the real Python they do compile to different Python ops (probably for historical reasons), but semantically they are identical. And importantly, not only do they produce the same results, but also there's never a case in which one works and the other produces an error. These call the MMD dispatcher. Now depending on the type of left and right, we e.g. call into a function living in classes/*.pmc that does the Right Thing. A plain PMC does the binop (if it can do it). If it's a an object derived from a PMC it either delegates to the PMC or, if provided by the user to the __pow__ function. If it's an object it does a full method lookup for that object ... In Python, semantically you know that you'll end up doing a method call (or, behaving as though you had), so it's very roundabout to do a method call by using an op which you know will fall back to doing a method call. Clearer just to do the method call. And currently, PerlInts used in Python code would handle a + b, but fail for a.__add__(b) (since that method isn't defined, currently). That breaks an invariant of Python--a + b should work if-and-only-if a.__add__(b) would, and should product the same result. Compiling to ops, this isn't guaranteed, and currently in fact doesn't happen. Real Python behaves as though infix operators are just an alternate syntax for certain method calls. And Ruby explicitly defines them this way--infix notation is just a syntax trick. (Note: not true for *all* operators, but at least for the mathematical infix operators.) ... Since __pow__ isn't special, we don't need anything to support it that we wouldn't need for any other arbitrary method, say y.elbowCanOpener(z). No. One is builtin and one isn't. These are really different. The only common thing is that the former can be overridden, while the latter is always provided by user code. I wouldn't describe that at can be overridden. If I define my own class, __pow__ is implemented on that class if-and-only-if I define it, just like any other method. I'm not overriding any default behavior--if I don't define it, it's just not there for my class. The only thing that's special is that there are certain built-in classes, and some of them implement __pow__, but that's not really anything special about __pow__. So I don't think an op really gives us what we want. This sentence would obviously be true for all binops then. It isn't. Yes, I think it's true for most of our PMC ops. And even the ops we currently have are broken semantically. Consider a = b + c in Python. This can't compile to add_p_p_p, because for that op to work, you have to already have an existing object in the first P register specified. But in Python, a is going to hold the result of b + c, which in general will be a new object and could be of any type, and has nothing to do with what's already in a. I don't seem to have any way to add 2 PMCs, and have as the result a third PMC, whose type is not known at compile-time. So yes, I am pointing out what I think is a larger design problem in Parrot--it's not specific to pow at all. And I think that we should not define PMC ops just because corresponding I or N ops exist. I think we should create PMC-based ops only if one of the following criteria are met: (a) there's no other reasonable way to provide some needed functionality, (b) there is some significant performance benefit to providing it as an op, or (c) it's needed to provide some interpreter functionality like invoke, but I think that this case is already covered by (a) and (b). Using methods when possible would let us get rid of most of the PMC ops, including things such as pow which make no sense for most PMC types, and leave us with a smaller set that makes more sense across PMC types. JEff
Re: No Cpow op with PMC arguments?
On Nov 5, 2004, at 10:03 AM, Sam Ruby wrote: Jeff Clites wrote: a) As Sam says, in Python y**z is just shorthand for y.__pow__(z)--they will compile down to exactly the same thing (required for Python to behave correctly). Since __pow__ isn't special, we don't need anything to support it that we wouldn't need for any other arbitrary method, say y.elbowCanOpener(z). [snip] So I don't think an op really gives us what we want. (Specifically, we could define a pow_p_p_p op, but then Python wouldn't use it for the case Sam brought up.) I'd apply the same argument to many of the other p_p_p ops that we have--they don't gives us what we need at the HLL level (though they may still be necessary for other uses). It is my intent that Python *would* use this method. What is important isn't that y**z actually call y.__pow__(z), but that the two have the same effect. Let's take a related example: y+z. I could make sure that each are PMCs [Not really a need for a check--everything will be a PMC, right? Or if not, you need to know before emitting the op anyway. Side issue, though.] and then find the __add__ attribute, retrieve it, and then use it as a basis for a subroutine call, as the semantics of Python would seem to require. Or I could simply emit the add op. How do I make these the same? I have a common base class for all python objects which defines an __add__ method thus: METHOD PMC* __add__(PMC *value) { PMC * ret = pmc_new(INTERP, dynclass_PyObject); mmd_dispatch_v_ppp(INTERP, SELF, value, ret, MMD_ADD); return ret; } ... so, people who invoke the __add__ method explicitly get the same function done, albeit at a marginally higher cost. There are three problems I see with this: 1) If you have 2 PerlInts in Python code, then a + b will work, but a.__add__(b) won't, since PerlInts won't inherit from your Python base class. To my mind, that breaks an invariant of Python. 2) If your __add__ method above were somehow in place even for PerlInts, it would produce a PyObject as its result, instead of a PerlInt, which is what I would have expected. That's a basic problem with our p_p_p ops currently--the return type can't be decided by the implementation of the MMD method which is called. and... Now to complete this, what I plan to do is to also implement all MMD/vtable operations at the PyObject level and have them call the corresponding Python methods. This will enable user level __add__ methods to be called. Prior to calling the method, there will need to be a check to ensure that the method to be called was, in fact, overridden. If not, a type_error exception will be thrown. 3) As described by Leo, the op would call the MMD dispatcher, which would ultimately do the method call, then your method above would call the MMD dispatcher, so you'd get an infinite loop, right? And if you avoid this by adding some check (as you mentioned) to make sure you only call __add__ if it was overridden (really, implemented at the user level), then your default implementation above will never be called, right? If instead you just compile infix operators as method calls, then all of those problems go away, and it's much simpler conceptually and in terms of implementation. And for Ruby (the language), it's explicit that infix operators are just an alternate syntax for method calls, so compiling them as ops is even more semantically problematic there. JEff
Re: No Cpow op with PMC arguments?
On Nov 4, 2004, at 5:24 AM, Sam Ruby wrote: [Referring to infix operators as an alternate syntax for a named method call] What's the downside of compiling this code in this way? If you are a Python programmer and all the objects that you are dealing with were created by Python code, then not much. However, if somebody wanted to create a language independent complex number implementation, then it wouldn't exactly be obvious to a Python programmer how one would raise such a complex number to a given power. Either the authors of the complex PMC would have to research and mimic the signatures of all the popular languages, or they would have to provide a fallback method that is accessible to all and educate people to use it. Yes, and I think that compiling using ops makes things worse, because of languages such as Java which don't have operator overloading, so you'd have to make all functionality available as method calls anyway, so why bother with the ops? Methods are much more flexible, and don't bloat the VM. Ultimately, Parrot will need something akin to .Net's concept of a Common Language Specification which defines a set of rules for designing code to interoperate. A description of .Net's CLS rules can be found in sections 7 and 11 (a total of six pages) in the CLI Partition I - Architecture document[1]. I think that ultimately, code will break down into two categories: 1) Code designed with multiple langauges in mind. 2) Code designed with only one language in mind. Code in case (2) will be awkward to use in other languages, but it should definitely be possible to use it somehow. For case (1), we need to make this easy for library authors to do. In terms of method naming, we may want to do something automatic (if you name your method such and such, it will appear to Python named this, and Ruby named that, and Perl named...), or it may be better to provide an explicit way to create language-specific method aliases. Some cases are simple (__mul__ in Python and * in Ruby and multiply in Java should all map to the same method for mathematical objects, probably), but others are more subtle (adding two array-like things means something different in different languages, potentially: append v. componentwise add v. componentwise add only if they have the same length). Doing something automatic saves a bunch of redundant work in the former case, but could cause problems in the latter.[1] And whatever the approach, it should be possible (even easy) for someone to take a library designed for only one language, and provide some cross-language mapping info and turn it into a nice cross-language library, without necessarily having to dig into the source code. (I'm thinking here of being able to specify the mapping in a document separate from the source code.) This is sort of treating method names as part of the interface, and not the implementation. [1] Automatic mapping could also cause problems in the case where I design a library and intend it to be cross-language, but where I want the API to look identical across langauges--I don't want to accidentally trip over a method name which happens got get translated for me, when I don't intend for that to happen. (For instance, I might name a method pow which controls some power level, not intending anything about exponentiation.) But, this wouldn't be a problem if the automatic approach were to let me somehow register a method as filling a certain role (my method 'blah' should be called to perform numeric addition), rather than inferring to from the method name. Probably a tricky balance between convenience and control/flexibility. JEff
Re: Streams and Filters (Ticket #31921)
On Nov 7, 2004, at 2:25 AM, Jens Rieks wrote: On Sunday 07 November 2004 09:48, Leopold Toetsch wrote: * where exactly is the mismatch coming from? Unix uses \n to indicate end-of-line, windows uses \r\n. The problem is, that the perlhist.txt file is checked in as a text file. I'll recommit it as a binary file in the hope that it fixes the problem. The root of the problem is the different line ending, I have no idea how parrot can deal with it, or if it should deal with it at all. Ah, just to amplify what you've said, it sounds like the problem is that CVS on Windows is translating the line endings upon checkout, since it's a text file. It sounds like the test is fine, just a bit tricky to get the perlhist.txt file in place unmodified. An alternative would be to create the test file on-the-fly, but flagging it as binary for CVS is probably simpler. (In my opinion, the line-ending issue isn't a problem for Parrot in terms of this test--it's legitimate to expect that you can process a particular file, not matter what platform you happen to be on, and no matter if the file happens to have been created on another platform.) JEff
Re: No Cpow op with PMC arguments? (Modified by Jeff Clites)
[missed cc-ing the list when I sent this originally] On Nov 5, 2004, at 10:39 AM, Brent 'Dax' Royal-Gordon wrote: Jeff Clites [EMAIL PROTECTED] wrote: There are a myriad of interesting mathematical types and operations, but they don't need dedicated ops to support them. But we're not talking about adding pow_bignum, pow_complex and pow_matrix. We're talking about adding pow--a fundamental operation by most standards--and a bignum, complex number, or matrix can Do The Right Thing when pow is called on it. What's bugging me is that PMCs are not meant to be specifically mathematical types--of the 72 in classes/*.pmc, only a few are. pow isn't a fundamental operation, by my thinking, on PMC types--it makes no sense for most of them. (Similarly for other mathematical operations.) Modeling these as method calls, rather than ops, seems to be a better fit conceptually. And if you look across languages, it makes even more sense (specifically in the case of mathematical infix operators): Java doesn't have operator overloading, and would use method calls anyway; Ruby explicitly treats infix operators as an alternative syntax for method calls; and Python semantically behaves this way as well. (And even the seemingly obvious cases aren't: There are at least three different operations on vectors which could be called multiplication. I don't think the mul op should be used for any of them.) I would assume mul on a matrix would perform the same matrix multiplication the public school system tortured me with in Algebra 2 and Precalculus. But I said vectors, not matrices. For vectors, you have dot product (inner product), cross product (outer product), and component-wise product (not often used in math, but useful in programming). Three different things which have equal claim on mul. There's not a one-to-one correspondence between fundamental operations on ints/floats and other mathematical types. What I *can* see a case for is removing *all* binary ops from their current special pseudo-vtable status; instead, create special names for them that won't conflict with anything, and turn them into normal methods (or normal multimethods, as the case may be). Yes, that's my thought. I think that we should only have PMC ops which make sense for most PMC types (or which are needed for basic interpreter functionality, like invoke)--it would make for a fairly short list. I don't think that we should have an op for PMCs just because we have a corresponding op for ints and floats. And no matter what, in terms of methods naming we'll need some mechanism for exposing a given method with different names for different languages--what shows up as __mul__ in Python should show up as multiply in Java and * in Ruby. That's something which hasn't been addressed/discussed yet. But that can't happen until we have N-ary multimethod support which scales well enough that we don't have to worry about the multimethod table becoming too big. Though in the case of Python, these don't act as multimethods--they dispatch on the left operand only. JEff
Re: The Saga of Building Parrot : ICU
On Nov 7, 2004, at 3:20 PM, Jack J. Woehr wrote: Nicholas Clark wrote: On Sun, Nov 07, 2004 at 01:17:28PM -0700, Jack J. Woehr wrote: Okay, I now configured perl Configure.pl --prefix=/usr/local/uplevel --cc=gcc --cxx=gcc --icuplatform=sparc-sun-solaris2.9 The build bombs in: string_set_data_directory: ICU data files not found(apparently) for directory [/usr/local/uplevel/blib/lib/icu/2.6.1]make: *** [runtime/parrot/include/parrotlib.pbc] Error 40 Does the directory /usr/local/uplevel exist at this point? Yes, it's empty, just created it the purpose of the build. Try building without specifying a --prefix to Configure.pl. The problem is that in the last part of the build process, parrot gets run to compile some stuff into bytecode, but since you haven't installed yet, this fails because some necessary files have not yet been moved into their final location. I think that's what's going on. It's definitely a problem with the configuration process, but the install functionality is new, and still a bit rough. So, doing make install without first doing make might work in the case of a --prefix specified, because it might take this into account. But if you just make and don't install (and don't specify a --prefix), then it should work--it's just that your build products will still be inside of your source tree, rather than being installed anywhere. But for experimenting, this should be okay. JEff
Basic compilation example (a + b)?
What pasm is supposed to correspond to this snippet of Python code (assume this is inside a function, so these can be considered to be local variables): a = 7 b = 12 c = a + b Just a basic compilation question, brought to mind by a recent thread, because the answer isn't obvious to me. JEff
Re: The Saga of Building Parrot : ICU
On Nov 7, 2004, at 12:10 PM, Leopold Toetsch wrote: Jack J. Woehr [EMAIL PROTECTED] wrote: Okay, I now configured perl Configure.pl --prefix=/usr/local/uplevel --cc=gcc --cxx=gcc --icuplatform=sparc-sun-solaris2.9 The build bombs in: string_set_data_directory: ICU data files not found(apparently) for directory [/usr/local/uplevel/blib/lib/icu/2.6.1]make: *** [runtime/parrot/include/parrotlib.pbc] Error 40 Argh (sorry not you), we have really to do something against that ever and ever resurrecting ICU build troubles. What's the state of ICU 3.0? ICU 3.0 worked fine for me as a drop-in replacement (built independently, not as part of the Parrot build), but I thinks that's not the main issue--it's the at-runtime location of resources that is the problem. I have an idea about how we could be doing that better, in particular in the case of using a dynamic libparrot (in which case parrot-specific resources should really be located relative to the library, not the executable). Basically, dladdr() lets you find out at runtime the path to a linked library, and that can be the base used for resource location (not just for the ICU data file(s), but also for the runtime directory, etc.). JEff
Re: No Cpow op with PMC arguments?
On Nov 4, 2004, at 5:24 AM, Sam Ruby wrote: From a Python or Ruby language perspective, infix operators are not fundamental operations associated with specific types, they are syntactic sugar for method calls. A the moment, I'm compiling x=y**z into: x = y.__pow__(z) There is nothing reserved about the name __pow__. Any class can define a method by this name, and such methods can accept arguments of any type, and return objects of any type. They can be called explicitly, or via the infix syntax. Of course--I should have realized that. I knew that's how Python handles +, etc.--don't know why I assumed exponentiation would be different. So scratch what I said. I should have said this: Languages tend to take one of the following two approaches when it comes to generalizing operations on basic types (numbers, strings) into operations on object types. 1) Generalization via conversion to a basic type. As an example, some languages generalize numeric addition, obj1 + obj2, as being syntactic sugar for something like, obj1.floatValue() + obj2.floatValue(). (That is, you do a basic operation on non-basic types by converting them into the relevant basic types, then performing the operation on those.) This is how Perl5 handles string concatenation--you string-concatenate two objects by string-ifying them, and concatenating those strings. 2) Generalization by method call. Some languages treat obj1 + obj2 as syntactic sugar for something like obj1.add(obj2). That is, you generalize in the obvious o.o. way. This is how Python (and C++) treats infix operators. Different languages choose (1) v. (2), and can certainly mix-and-match (take one approach for some operations, another for others). Another way a language may mix-and-match is to do (2) if such a method is defined on the object, and fall back to (1) if it isn't. Now from a Parrot perspective: Case (1) is already handled by Parrot--it's just an exercise in code generation by a compiler. For case (2), I think these operations correspond to method calls on objects (in the Parrot sense--the stuff in src/objects.c), not MMD or vtable operations accessed via custom ops. Here are a couple of examples why: a) As Sam says, in Python y**z is just shorthand for y.__pow__(z)--they will compile down to exactly the same thing (required for Python to behave correctly). Since __pow__ isn't special, we don't need anything to support it that we wouldn't need for any other arbitrary method, say y.elbowCanOpener(z). b) I can define arbitrary Python classes with arbitrary implementations of __pow__, and change those implementations on-the-fly, on a per-class or per-instance basis. These aren't new PMC-classes, and I don't think that the op-plus-MMD approach gives us the ability to handle that. Summary: Both cases (1) and (2) are syntactic sugar, and case (1) is sugar for casting/conversion, and case (2) is sugar for object-method calls. So I don't think an op really gives us what we want. (Specifically, we could define a pow_p_p_p op, but then Python wouldn't use it for the case Sam brought up.) I'd apply the same argument to many of the other p_p_p ops that we have--they don't gives us what we need at the HLL level (though they may still be necessary for other uses). JEff
Re: No Cpow op with PMC arguments?
On Nov 4, 2004, at 10:30 PM, Brent 'Dax' Royal-Gordon wrote: On Thu, 4 Nov 2004 21:46:19 -0800, Jeff Clites [EMAIL PROTECTED] wrote: On Nov 4, 2004, at 8:29 PM, Brent 'Dax' Royal-Gordon wrote: This is true. But how do you define a number? Do you include floating-point? Fixed-point? Bignum? Bigrat? Complex? Surreal? Matrix? N registers don't even begin to encompass all the numbers out there. Floating point, and possibly integer. Those are the numeric primitives of processors. Other aggregate mathematical types are always defined in terms of those (in a computing context), one way or another. Yes, but your decomposition (N2=P2; N3=P3; N1=N2+N3; P1=N1) doesn't take anything but the primitives into account. Yes--see my subsequent message (just sent a moment ago), as that decomposition isn't what I meant. But you asked how I define a number--that's what I was answering above, from a computing perspective, not a more general question. My point there, when I said one way or another, was that for example, even in mathematics, you define addition, multiplication, etc. on complex numbers in terms of such operations over the real numbers. (That is, you define them in terms of operations on a pair of real numbers.) Based on the design of processors, and of Parrot, there's a good performance reason to define basic ops on integers and floating point numbers--namely, they'll often JIT down to single instructions. Such ops on PMCs won't have that performance benefit--they'll still involve table lookups and multiple function calls to execute. So a mechanism other than an op may be more appropriate there. There are a myriad of interesting mathematical types and operations, but they don't need dedicated ops to support them. (And even the seemingly obvious cases aren't: There are at least three different operations on vectors which could be called multiplication. I don't think the mul op should be used for any of them.) JEff
Re: No Cpow op with PMC arguments?
On Nov 3, 2004, at 8:09 AM, Dan Sugalski wrote: At 11:04 AM -0500 11/3/04, Sam Ruby wrote: A single pow_p_p_p op backed by a (non-MMD) vtable entry would make it easier to support code like the following: def f(x): return x**3 print f(3), f(2.5) Yeah, it would. I know I'm going to regret asking, but... any reason *not* to make it MMD? (Though I have no idea what happens if you square a matrix) I feel like we have op-itis and vtable-itis. I would think that rather than a pow_p_p_p, you'd compile x**y as something like: set N0, P0 set N1, P1 pow N2, N0, N1 new P2, .PythonNumber assign P2, N2 I.e., PMCs don't inherently exponentiate--numbers do, and you can exponentiate PMCs by numberizing them, exponentiating, and creating a PMC with the result. Or, if we must have an op, implement it something like this (without needing a new vtable entry, or MMD): inline op pow(out PMC, in PMC, in PMC) :base_core { $1 = pmc_new(interpreter, $2-vtable-type(interpreter, $2)); $1-vtable-set_number_native(interpreter, $1, pow( $2-vtable-get_number(interpreter, $2), $3-vtable-get_number(interpreter, $3))) goto NEXT(); } Those would probably JIT to about the same thing, given register mapping. This is the viewpoint that pow() isn't a fundamental operation that makes sense for all types; it's a numeric operation, which can be extended in a straighforward manner to types which know how to represent themselves as numbers. (e.g., it's gibberish to raise a ManagedStruct to a ParrotIO power, except that you can stretch and interpret such a thing as just implicit num-ification of the arguments.) JEff
Re: No Cpow op with PMC arguments?
On Nov 4, 2004, at 8:29 PM, Brent 'Dax' Royal-Gordon wrote: Jeff Clites [EMAIL PROTECTED] wrote: I.e., PMCs don't inherently exponentiate--numbers do, and you can exponentiate PMCs by numberizing them, exponentiating, and creating a PMC with the result. This is true. But how do you define a number? Do you include floating-point? Fixed-point? Bignum? Bigrat? Complex? Surreal? Matrix? N registers don't even begin to encompass all the numbers out there. Floating point, and possibly integer. Those are the numeric primitives of processors. Other aggregate mathematical types are always defined in terms of those (in a computing context), one way or another. JEff
Re: more vtables
On Nov 2, 2004, at 12:41 AM, Leopold Toetsch wrote: When we have objects with finalizers, we have to run the finalizers in order from most derived down the parent chain. Maybe, but not necessarily. The case of loops means that we cannot always do this cleanly (no top of the chain), and the fact that finalizers may modify the topology of the object graph also makes this a bit ill-defined. There are several possible approaches, ranging from very conservative to very aggresive. Two real-world approaches already used are: make no guarantees about finalization order--only guarantee that a finalizer will be called at some point after an object becomes unreferenced, with no guarantees about ordering, and no guarantees that it will not be called if the object has already be re-rooted (Java); or, do graph-ordered finalization, and never call the finalizers of objects which are in loops (the Boehm collector does something along these lines, I believe). So as you state, we've to defer destruction, store these objects with finalizers somewhere, sort them, run user code to finalize objects and so on. Other objects are impacted as well--any object reachable from a finalizable object must wait for reclamation until finalizers have fired. Doing that all in the destroy vtable is tedious especially, when user code is involved too. The finalize vtable is overridable, the destroy isn't. A split makes this functionality much cleaner. I think what's bothering me about this is that I think part of the idea is to move closing of filehandles and such into destroy()--right? That sounds good--don't get rid of this resource until it's certain nothing will try to use it (eg, things which want to log a message during their finalization would use a filehandle). But I don't think this really solves it. For one thing, there will be similar-in-functionality code coming up in HLL's (doing some orderly shutdown of connections, which involves more than just closing the filehandle), and this will have to happen in a finalizer (since that's the only HLL option), which will feel too early in some cases. And even in a destroy(), you might want to do some I/O, and whether this will work will depend on whether the filehandles have been closed yet, and the same ordering concerns re-emerge. So I think the split would help, in a sense, but only be a partial solution to an unsolvable problem, and therefore maybe is not very elegant. So I'm a bit on the fence. (And in my parlance, just to be clear: finalizers are called after an object becomes un-referenced, and could re-root an object; destructors are called when the object is actually having its memory re-claimed--when it is going away for sure. No user code can be called from a destructor in the Parrot case, because of the possibility of re-rooting, and because of other constraints which may exist at destroy-time.) JEff
Re: case mangling and binary strings
On Nov 2, 2004, at 10:46 AM, Dan Sugalski wrote: At 1:42 PM -0500 11/2/04, Sam Ruby wrote: I don't care if Parrot uses ICU on any platform. I do care that Parrot supports utf-8 on every platform. Ah, OK. Yes, we will support all the unicode encodings, as well as the unicode character set, on all platforms. I'll point out, for the record, that it make no sense to have strings holding binary data--the observation that fundamental string API make no sense for such strings is an indication that this is an attempt to use the wrong data type for the wrong purpose. Just thrown an exception is not a magic cure--it's an indication that things are being modeled incorrectly. It shouldn't be necessary. In terms of ICU specifically, I don't much care what underlying library we use. That said, if we want to support all the unicode encodings, we're going to have to provide all of the relevant functionality that ICU gives us. Trying to re-implement this, separately, from scratch, is a mistake--a waste of resources. A better approach (for Parrot, and for the computing community in general) would be to put our efforts toward getting ICU working on the platforms on which it is having problems. Or start with some other well-established library--but ICU seems to be the best around. JEff
Re: Performance Statistics (and Graphs!)
On Nov 2, 2004, at 7:10 PM, Matt Diephouse wrote: Joshua Gatcomb and I have been working a little under a week to set up an automated build system for parrot that tracks performance (with help from Dan's box). We have collected benchmark data for regular and optimized builds with and without JIT from June 1st through October. With some help from Perl and GD, we've up several pages of graphs: http://www.sidhe.org/~timeparrot/graphs/ Both the build process and the creation of the graphs will (hopefully) be run daily (automated). There are plans to do some other things with the data as well (some notification scripts). Very cool. A useful thing to do would be to preserve all of the builds, so that newly-created benchmarks could be back-filled. (That way, if we realize that we've not been effectively testing something, we can devise an appropriate benchmark, and evaluate the consequences of past design changes against it.) JEff
Re: Are we done with big changes?
On Nov 1, 2004, at 6:14 AM, Dan Sugalski wrote: Because I need to get strings working right, so I'm going to be implementing the encoding/charset library stuff, which is going to cause some major disruptions. Please tag cvs before checking this in. Thanks, JEff
Re: Parrot on AMD64
On Oct 29, 2004, at 11:29 PM, Brent 'Dax' Royal-Gordon wrote: I recently got my hands on money for an upgrade, and got an AMD64 and motherboard and installed them. I'm still using 32-bit Windows, but I've also installed a 64-bit Gentoo on some previously unpartitioned space. ... Failed Test Stat Wstat Total Fail Failed List of Failed --- t/library/streams.t1 256211 4.76% 11 3 tests and 53 subtests skipped. Failed 1/124 test scripts, 99.19% okay. 1/1959 subtests failed, 99.95% okay. Any thoughts on this? Known failure currently for all platforms--see recent message [CVS ci] indirect register frame 9 - go. So it looks like you are passing all expected tests! JEff
Re: Mostly a Perl task for the interested
On Oct 30, 2004, at 12:58 AM, Leopold Toetsch wrote: Nicholas Clark [EMAIL PROTECTED] wrote: On Fri, Oct 29, 2004 at 05:47:55PM +0200, Leopold Toetsch wrote: * The created C code could benefit from #line directives to track where C code came from the input .pmc file, so that compiler errors are reported for the original .pmc file. Perl 5's xsubpp does this well, using #line directives to switch between foo.c and foo.xs, depending on whether that section of code was human written, or autogenerated. It makes things much easier while developing. Yep, thanks. The hooks are there, as well as a command-line option to turn line numbers off, which is sometimes useful. Anyway most line number stuff is already inside, but it's broken and needs fixing. FYI I'm fiddling with classes/pmc2c2.pl, in connection with your ccache-ish feature request, but I'm not (much) touching Pmc2c.pm, so I shouldn't conflict with anyone working there. JEff
[PATCH] PPC JIT failure for t/pmc/threads_8.pasm
I was getting a failure under JIT on PPC for t/pmc/threads_8.pasm, and the problem turned out to be that emitting a restart op takes 26 instructions, or 104 bytes, and we were hitting the grow-the-arena logic just shy of what would have triggered a resize, then running off the end. The below patch fixes this; really that magic number (200, now) needs to be bigger than the amount of space we'd ever need to emit the JIT code for a single op (plus saving registers and such), but with the possibility of dynamically loadable op libs (with JIT?), it's hard to say what number is guaranteed to be large enough. Or, we can pick a reasonable, largish number that works for the built-in ops (empirically determined, as now), and document that loadable JITted ops which could take more than this, need to make sure to grow the arena as necessary. (And we could provide a utility function to make this easy.) JEff Index: src/jit.c === RCS file: /cvs/public/parrot/src/jit.c,v retrieving revision 1.95 diff -u -b -r1.95 jit.c --- src/jit.c 25 Oct 2004 10:24:14 - 1.95 +++ src/jit.c 29 Oct 2004 07:50:09 - @@ -1395,7 +1395,7 @@ while (cur_op = cur_section-end) { /* Grow the arena early */ if (jit_info-arena.size -(jit_info-arena.op_map[jit_info-op_i].offset + 100)) { +(jit_info-arena.op_map[jit_info-op_i].offset + 200)) { #if REQUIRES_CONSTANT_POOL Parrot_jit_extend_arena(jit_info); #else
AIX PPC JIT warning
Recently config/gen/platform/darwin/asm.s was added, containing Parrot_ppc_jit_restore_nonvolatile_registers(). Corresponding code also needs to be added to config/gen/platform/aix/asm.s -- Parrot should fail to link on AIX currently, without this. I didn't try to update the AIX asm.s myself, since I wasn't confident that I could do this correctly without having a way to test. So, someone with AIX asm expertise, please take a look. Thanks, JEff
Re: Perl 6 and byte code
On Oct 27, 2004, at 6:24 AM, Gisle Aas wrote: How about the code JITed from the bytecodes. Will it be shared? The JITed code can't be shared directly--in it's current form, it (intentionally) includes absolute addresses which wouldn't be valid for other processes. But the exec core allows (or, will allow) bytecode to be compiled into a native executable (or presumably, library), so that would allow the advantages of native, memory-shared code. JEff
[PATCH] Re: JIT and platforms warning
On Oct 22, 2004, at 3:57 AM, Leopold Toetsch wrote: Jeff Clites wrote: On Oct 22, 2004, at 1:01 AM, Leopold Toetsch wrote: [JIT changes] I just finished tracking down the source of a couple of JIT test failures on PPC--due to recent changes but only indirectly related, and pointing out things which needed fixing anyway (float register preservation issues). I'll send it in tomorrow after I've had a chance to clean it up and add some comments. Please make sure to get a recent CVS copy and try to prepare the patch during my night ;) Of course! I've changed the PPC float register allocation yesterday, because it did look bogus - i.e. the non-volatile FPRs were allocated but not saved. That should be fixed. Yep, that was the core of the issue. There's no free lunch--if we use the nonvolatile registers, we need to preserve/restore them in begin/end, but if we use the volatile registers, we need to preserve them across function calls (incl. normal op calls). So I added code to do the appropriate save/restore, and use the non-volatile registers for mapping--that should be less asm than what we'd have to do to use the volatile registers. (The surprising thing was that we only got 2 failures when using the volatile registers--I'll look into creating some tests that would detect problems with register preservation.) The other tricky part was that saving/restoring the FP registers is one instruction per saved register, so saving all 18 was exceeding the asm size we allocate in src/jit.c (in some cases), since we emit Parrot_end for all restart ops. The fix for this was to pull this asm out into a utility routine, and just call that from the asm. (This is only done for restoring the registers so far--I should do it for the preservation step too, but right now that's just inline.) The attached patch also contains some other small improvements I'd been working on, and a few more jitted ops to demonstrate calling a C function from a jitted op. While you're investigating PPC JIT: JIT debugging via stabs doesn't work at all here. I get around the gdb message (missing data segment) by inserting .data\n.text\n in the stabs file, but that's all. After this change and regenerating file.o I can load it with add-symbol-file file.o without any complaint, but the access to the memory region the jitted code occupies is not allowed, disassemble doesn't work ... I'll take a look and see if I can figure it out. I remember gdb being uncooperative about disassembling, frustratingly, and I've moved to the habit of using something like x/300i jit_code as a workaround. Clearly it can access the memory region, so it seems like a gdb bug. See attached the patch, plus the new asm.s file. JEff config/gen/platform/darwin/asm.s: asm.s Description: application/applefile asm.s Description: application/text ppc-jit-preserve-fp.patch Description: application/text
Re: [PATCH] Re: JIT and platforms warning
On Oct 23, 2004, at 4:20 AM, Leopold Toetsch wrote: Jeff Clites [EMAIL PROTECTED] wrote: See attached the patch, plus the new asm.s file. Doesn't run, segfaults on even mops.pasm - please check. I can't reproduce that here; parrot -j works for me with examples/{benchmarks,assembly}/mops.pasm, and all 'make testj' tests pass. (They were passing before, and I updated to pick up changes since I sent the patch, and still all passes.) I also tried building against a system ICU (was building against the parrot-supplied version), in case there were issues with calling into shared lib code (long shot), and no difference. Below is myconfig--I'm not doing any special configuration (just running 'perl Configure.pl', and not building optimized). Do you have any uncommitted local changes which might be involved? I'm up-to-date from CVS, and have no uncommitted changes except for those in the patch. JEff Summary of my parrot 0.1.1 configuration: configdate='Sat Oct 23 13:06:30 2004' Platform: osname=darwin, archname=darwin jitcapable=1, jitarchname=ppc-darwin, jitosname=DARWIN, jitcpuarch=ppc execcapable=1 perl=perl Compiler: cc='cc', ccflags='-g -pipe -pipe -fno-common -no-cpp-precomp -DHAS_TELLDIR_PROTOTYPE -pipe -fno-common -Wno-long-double -I/sw/include', Linker and Libraries: ld='c++', ldflags=' -flat_namespace -L/sw/lib', cc_ldflags='', libs='-lm -lgmp' Dynamic Linking: share_ext='.dylib', ld_share_flags='-dynamiclib', load_ext='.bundle', ld_load_flags='-bundle -undefined suppress' Types: iv=long, intvalsize=4, intsize=4, opcode_t=long, opcode_t_size=4, ptrsize=4, ptr_alignment=1 byteorder=4321, nv=double, numvalsize=8, doublesize=8 % gcc -v Reading specs from /usr/libexec/gcc/darwin/ppc/3.3/specs Thread model: posix gcc version 3.3 20030304 (Apple Computer, Inc. build 1495)
Re: [PATCH] Re: JIT and platforms warning
On Oct 23, 2004, at 3:42 AM, Leopold Toetsch wrote: Jeff Clites [EMAIL PROTECTED] wrote: Yep, that was the core of the issue. There's no free lunch--if we use the nonvolatile registers, we need to preserve/restore them in begin/end, but if we use the volatile registers, we need to preserve them across function calls (incl. normal op calls). Good point and JIT/i386 does it wrong in core.ops. But normal non-JITted code is not the problem - the framework preserves mapped registers or better - it has to copy all mapped registers to Parrot registers so that C code is able to see the actual values. Ah, good. The problem I was seeing was caused by Cset_s_sc, since my jit_emit_call_func was written with the assumption that we should be mapping only non-volatile registers (even though we were actually mapping volatiles, but at the end of the list, and even though we weren't yet preserving the appropriate float registers). But a recent check-in had us mapping the volatile float registers only, which exposed the problem. ... So I added code to do the appropriate save/restore, and use the non-volatile registers for mapping--that should be less asm than what we'd have to do to use the volatile registers. (The surprising thing was that we only got 2 failures when using the volatile registers--I'll look into creating some tests that would detect problems with register preservation.) Well, allocation strategy on PPC (or I386) is to first use the non-volatile registers. PPC has 14 usable registers. To provoke a failure you'd need e.g. 15 different I-registers and then a JITted function call like Cset_s_sc. We were allocating the volatile float registers first (or, only)--so Cset_s_sc was blowing away an N-register, even with only one in use. That's why I was surprised there weren't more failures. Anyway, I think we need a more general solution. ... Solution: The JIT compiler/optimizer already calculates the register mapping. We have to use this information for JIT pro- and epilogs. That makes sense--I hadn't initially realized we were tracking this, but since we are, we should use it. Things may be a bit tricky fixup-wise, since the size of the prolog will depend on how many float registers we need to preserve. (We can save a variable number of int registers with just a single instruction in the PPC case, as long as we are using registers in order, or we don't mind saving a few extra if we are not.) The allocation strategy should be adjusted and depend on code size and register usage: - big chunks of compiled code like whole modules should use the non-volatile registers first and then (if needed) the volatile registers. - small chunks of code should use the volatile registers to reduce function startup and end sizes. Yep, makes sense. And there may be a case for not mapping at all for very small chunks, but I'd have to experiment to know if that's be a win. 2) JITed functions with calls into Parrot The jit2h JIT compiler needs a hint, that we call external code, e.g. CALL_FUNCTION(string_copy) This notion is needed anyway for the EXEC core, which has to generate fixup code for the executable. So with that information available, the JIT compiler can surround such code with: PRESERVE_REGS(); ... RESTORE_REGS(); The framework can now depending on the register mapping save and restore volatile registers if needed. Sounds good. The other tricky part was that saving/restoring the FP registers is one instruction per saved register, so saving all 18 was exceeding the asm size we allocate in src/jit.c (in some cases), since we emit Parrot_end for all restart ops. Yep, that's suboptimal. I've done that on i386 because it was just easy. But you are right, the Parrot_end() code should really be there only once. Yep, instead of emitting it multiple times and conditionally jumping over it, we can emit it just once and conditionally jump to it. ... , frustratingly, and I've moved to the habit of using something like x/300i jit_code as a workaround. Ah, yes - forgot that. At least stepping does the right thing now, with your fix for the stabs file. Very nice to be able to do p I0 and such. One other question: Your P_ARITH optimization in jit/ppc/core.jit. I can't come up with a case where this kicks in, since in the tests I've tried, prev_op is always NULL when JITting if_i_ic or unless_i_ic. If I set things to not map any int registers, then I don't hit the cases in jit_emit.h which set prev_op to 0, but build_asm is doing it--I can't come up with a small test case which isn't being treated as multiple sections, it seems. There's something there w.r.t. sections which I don't understand, but anyway I just wanted to know how to set things up so that I can see your optimization in action. Thanks, JEff
Re: A small Perl task
On Oct 23, 2004, at 5:14 AM, Leopold Toetsch wrote: First, if you don't have it yet done, install ccache. Thanks for the tip--seems awesome. HOW IT WORKS The basic idea is to detect when you are compiling exactly the same code a 2nd time and use the pre- viously compiled output. You detect that it is the same code by forming a hash of: ... So you get really fast recompiles *except* for classes/*.c. What I'd like to have is (based on ccache's philosophy) a cache for classes/*.dump and classes/*.c files. Currently they are recreated permanently. I started playing with this, got confused, and finally figured out why I'm confused. In the dump case, you basically (it seems) need to parse the pmc file in order to determine the parent (chain), in order to determine the dependencies, in order to figure out what you need to put into your digest. (In the ccache case, it's the preprocessor doing this for you.) But by then, you've done most of the work I'd think (by parsing the pmc file). So at that point, you'd end up calculating a digest over a bunch of files, to avoid the Dumper overhead, and the latter may likely be faster. So this might not be a win for us, since it doesn't look like it would let us skip much. I think this will only be worth doing if we have a faster way to determine the dependencies (basically the parents of the pmc in question). The Makefile implicitly knows this, so it might be possible for it to pass this into the script. But I have the caching and use-the-best-digest-module logic worked out, I'm just stuck on figuring out what work it can skip. Let me know if you have any ideas, or if I'm missing something. I guess maybe the *.{c,h} case is simpler, since there you only depend on your *.dump files (plus the script itself, etc.), though there's more work to figure out what files you're trying to generate (in the library case, etc.). JEff
Re: JIT and platforms warning
On Oct 22, 2004, at 1:01 AM, Leopold Toetsch wrote: [JIT changes] I just finished tracking down the source of a couple of JIT test failures on PPC--due to recent changes but only indirectly related, and pointing out things which needed fixing anyway (float register preservation issues). I'll send it in tomorrow after I've had a chance to clean it up and add some comments. JEff
Re: Register stacks, return continuations, and speeding up calling
On Oct 20, 2004, at 12:09 PM, Leopold Toetsch wrote: Dan Sugalski wrote: 'Kay, now I'm confused. I thought we were talking about removing the registers from out of the interpreter structure, which'd leave us needing two pointers, one for the interpreter struct and one for the registers. Ok, short summary of future layout of JIT regs: itemPPC i386 interpreter r13 -16(%ebp) frame pointer r16%ebx Register addressing is done relative to the frame pointer, which will be in a register. The interpreter isn't used that often inside integer JIT code, so it isn't in an register in i386 but is easily reloaded into one. Currently the frame pointer and the interpreter are the same. Just to clarify: This is the approach wherein each frame gets a fresh set of registers, and function call and return (or continuation invocation) copy the relevant registers between the register sets? And this isn't quite the scheme from the towards a new call scheme thread, in which we'd be duplicating the interpreter context for each frame, right? (And the latter was what you did in Proof of concept - hack_42 (was: the whole and everything), right?) Just trying to sort out all of the ideas. JEff
Re: Pathological Register Allocation Test Generator
On Oct 20, 2004, at 11:24 PM, Leopold Toetsch wrote: Bill Coffman [EMAIL PROTECTED] wrote: And of course, lexicals and globals already have a storage, you don't need to spill them. I'm not sure that's true. If there's no 'eval' in scope, lexicals don't have to live in pads--they could purely exist in registers. And with tied namespaces and such, it may not be legitimate to re-fetch a global (ie, to fetch it multiple times, if the code appears to only fetch it once) -- one could pathologically have a global whose value appears to increase each time it's fetched, for instance, or you could end up with multiple round-trips to a database. ... Another thing that might be worth checking, after parrot gets out of alpha, is if reducing or increasing the number of registers will help performance. Just a thought. The 4 x 32 is pretty good. It matches recent hardware too. But if a good register algorithm shows that 4 x 16 is enough, we can of course decrease the register number. Increasing shouldn't be necessary. Matching hardware is probably not too significant--even though the PPC has 32 int registers, we can't map to all of them in JIT (some are dedicated to holding the interpreter pointer, etc.), and we'd really need 3 x 32 hardware int registers to accommodate all we'd like (I, S, and P registers). So even currently it's a loose match. JEff
Re: Pathological Register Allocation Test Generator
On Oct 21, 2004, at 4:13 AM, Leopold Toetsch wrote: Jeff Clites [EMAIL PROTECTED] wrote: On Oct 20, 2004, at 11:24 PM, Leopold Toetsch wrote: And of course, lexicals and globals already have a storage, you don't need to spill them. I'm not sure that's true. It should read: if there are lexical or global opcodes, lexicals and globals have a storage. Ah yes, true. tied namespaces and such, it may not be legitimate to re-fetch a global (ie, to fetch it multiple times, if the code appears to only fetch it once) -- one could pathologically have a global whose value appears to increase each time it's fetched, Then there's something horribly wrong with that usage of tie: not the value is refetched from the var - the var is refetched from the namespace. I think there'll be two types of tie--tied variables (like Perl has already), and tied namespaces (as supposedly some people really need, though I don't fully know why). But even without the above pathological case: with tied namespaces, a namespace fetch potentially has unknown overhead, and a compiler can't know if re-fetching is better than spilling. But on the other hand, maybe that's just part of the deal with tied namespaces--they may be fetched from more often than the code would imply, so the tied namespace needs to be prepared for that. JEff
Re: Register stacks, return continuations, and speeding up calling
On Oct 21, 2004, at 8:34 AM, Leopold Toetsch wrote: Dan Sugalski wrote: In that case I won't worry about it, and I think I know what I'd like to do with the interpreter, the register frame, and the register backing stack. I'll muddle it about some and see where it goes. JIT/i386 is up to date now that is: it doesn't do any absolute register addressing anymore. So what next: * deprecate the usage of allmost all register stack push and pops? I think we don't need them anymore. Register preservering is done as part of the call sequence. Are we still planning to move the current return continuation and current sub, out of the registers and into their own spots in the interpreter context (in order to avoid the PMC creation overhead in the common case, etc.)? (Or, have we already done this?) JEff
Re: C89
On Oct 21, 2004, at 11:51 AM, Dan Sugalski wrote: At 11:25 AM -0700 10/21/04, Bill Coffman wrote: I read somewhere that the requirement for parrot code is that it should be compliant with the ANSI C'89 standard. Can someone point me to a description of the C89 spec, so I can make sure my reg_alloc.c patch is C89 compliant? I don't think the ANSI C89 spec is freely available, though I may be wrong. (Google didn't find it easily, but I don't always get along well with Google) If the patch builds without warning with parrot's standard switches then you should be OK. (ANSI C89 was the first big rev of C after the original KR C. If you've got the second edition or later of the KR C book, it uses the C89 spec) Also, if you're compiling with gcc, then you can pass -std=c89 to the compiler to enforce that particular standard. (Apparently--though I haven't tried it.) I believe -ansi does the same thing. JEff
Re: [perl #32036] [BUG] t/pmc/signal.t fails
On Oct 19, 2004, at 12:42 AM, Leopold Toetsch wrote: Will Coleda [EMAIL PROTECTED] wrote: t/pmc/signal...Hangup I saw that once too: looks like the test script got the signal. That's what my patch from last week was supposed to fix--I'm surprised it's still happening. We should currently be making sure not to kill the harness. (However, if there's another copy of parrot running--for instance, stopped in a debugger somewhere--then that may get the signal erroneously.) JEff
Re: [Proposal] JIT, exec core, threads, and architectures
On Oct 19, 2004, at 1:56 AM, Leopold Toetsch wrote: Jeff Clites [EMAIL PROTECTED] wrote: On Oct 17, 2004, at 3:18 AM, Leopold Toetsch wrote: Nethertheless we have to create managed objects (a Packfile PMC) so that we can recycle unused eval-segments. True, and some eval-segments are done as soon as they run (eval 3 + 4), whereas others may result in code which needs to stay around (eval sub {}), and even in the latter case not _all_ of the code generated in the eval would need to stay around. It seems that it may be hard to determine what can be recycled, and when. Well, not really. As long as you have a reference to the code piece, it's alive. Yes, that's what I meant. In the case of: $sum = eval 3 + 4; you don't have any such reference. In the case of: $sub = eval sub { return 7 }; you do. In the case of: $sub = eval 3 + 4; sub { return 7 }; you've got a reference to the sub still, but the 3 + 4 code is no longer reachable, so wouldn't need to stay around. But it's possible that Parrot won't be able to tell the difference, and will have to keep around more than is necessary. And we have to protect the packfile dictionary with mutexes, when this managing structure changes i.e. when new segments gets chained into this list or when they get destroyed. Yes, though it's not clear to me if all eval-segments will need to go into a globally-accessible dictionary. (e.g., it seems the 3 + 4 case above would not.) It probably depends on the generated code. If this code creates globals (e.g. Sub PMCs) it ought to stay around. Yes, that's what I meant by not all--some yes, some no. JEff
Re: [Proposal] JIT, exec core, threads, and architectures
On Oct 17, 2004, at 3:18 AM, Leopold Toetsch wrote: Jeff Clites wrote: On Oct 16, 2004, at 4:47 AM, Leopold Toetsch wrote: Nethertheless we have to create managed objects (a Packfile PMC) so that we can recycle unused eval-segments. True, and some eval-segments are done as soon as they run (eval 3 + 4), whereas others may result in code which needs to stay around (eval sub {}), and even in the latter case not _all_ of the code generated in the eval would need to stay around. It seems that it may be hard to determine what can be recycled, and when. And we have to protect the packfile dictionary with mutexes, when this managing structure changes i.e. when new segments gets chained into this list or when they get destroyed. Yes, though it's not clear to me if all eval-segments will need to go into a globally-accessible dictionary. (e.g., it seems the 3 + 4 case above would not.) OTOH it might be better to just toss all the constant table access in all instructions, except: set_n_nc set_s_sc set_p_pc # alias set_p_kc This would reduce the interpreter size significantly (compare the size of core_ops_cgp.o with core_ops_cg.o). Reducing the size is good, but this doesn't overall reduce the number of accesses to the constant table, just changes which op is doing them. Not quite, e.g: set P0[foo], bar set S0, P0[foo] are 3 accesses to the constant table. It would be set S1, foo set S2, bar set P0[S1], S2 set S0, P0[S1] with 2 accesses as long as there is no pressure on the register allocator. Sure, but you can do this optimization today--narrowing it down to just those 3 ops isn't required. But it only helps if local re-use of the same constants is frequent, and it may not be. (But still, it's a good optimization for a compiler to implement--it just may not have a huge effect.) Also, there's some subtlety. This: set S1, foo set S2, foo isn't the same as: set S1, foo set S2, S1 but rather: set S1, foo clone S2, S1 since 'set' copies in the s_sc case. (That's not a problem, just something to keep in mind.) The assembler could still allow all constant variations of opcodes and just translate it. For this we'd need a special register to hold the loaded constant, so that we don't overwrite a register which is in use. No, just the registers we have anyway, For PIR yes, but the PASM assembler can't know for sure what register would be safe to use--the code could be using its own obscure calling conventions. JEff
Re: [Proposal] JIT, exec core, threads, and architectures
On Oct 16, 2004, at 12:26 AM, Leopold Toetsch wrote: Jeff Clites [EMAIL PROTECTED] wrote: ... But, we use this currently, because there is one issue with threads: With a thread, you don't start from the beginning of the JITted code segment, This isn't a threading issue. We can always start execution in the middle of one segment, e.g. after an exception. That's already handled on almost all JIT platforms and no problem. The code emitted in Parrot_jit_begin gets the Ccur_opcode * as argument and has to branch there, always. I was remembering wrong--we do this on PPC too. On Oct 16, 2004, at 4:47 AM, Leopold Toetsch wrote: String, number (and PMC) constants are all addressed in terms of the compiling interpreter. ... When we do an eval() e.g. in a loop, we have to create a new constant table (and recycle it later, which is a different problem). Running such a compiled piece of code with different threads would currently do the wrong thing. The correct constant table depends on the code segment, rather than the specific interpreter, right? That means that referencing the absolute address of the const table entry would be correct for JIT code no matter the executing thread, but getting the const table from the compiling interpreter is wrong if that interpreter isn't holding a reference to the corresponding code segment. Access to constants in the constant table is not only a problem for the JIT runcore, its a lengthy operation for all code, For a string constant at PC[i]: interpreter-code-const_table-constants[PC[i]]-u.string These are 3 indirections to get at the constants pointer array, and worse they depend on each other, emitting these 3 instructions on an i386 stalls for 1 cycle twice (but the compiler is clever and interleaves other instructions) For the JIT core, we can precalculate the location of the constants array and store it in the stack or even in a register (on not so register-crippled machines like i386). It only needs reloading, when an Cinvoke statement is emitted. For PPC JIT, it seems that we are putting in the address of the specific const table entry, as an immediate. OTOH it might be better to just toss all the constant table access in all instructions, except: set_n_nc set_s_sc set_p_pc # alias set_p_kc This would reduce the interpreter size significantly (compare the size of core_ops_cgp.o with core_ops_cg.o). Reducing the size is good, but this doesn't overall reduce the number of accesses to the constant table, just changes which op is doing them. The assembler could still allow all constant variations of opcodes and just translate it. For this we'd need a special register to hold the loaded constant, so that we don't overwrite a register which is in use. JEff
Re: [Proposal] JIT, exec core, threads, and architectures
On Oct 14, 2004, at 12:10 PM, Leopold Toetsch wrote: Proposal: * we mandate that JIT code uses interpreter-relative addressing - because almost all platforms do it - because some platforms just can't do anything else - and of course to avoid re-JITting for every new thread FYI, the PPC JIT does already do parrot register addressing relative to the interpreter pointer, which as you said is already in a CPU register. This is actually less instructions than using absolute addressing would require (one rather than three). We do still re-JIT for each thread on PPC, though we wouldn't have to (just never changed it to not). But, we use this currently, because there is one issue with threads: With a thread, you don't start from the beginning of the JITted code segment, but rather you need to start with a specific Parrot function call, somewhere in the middle. But you can't just jump to that instruction, because it would not have the setup code needed when entering the JITted section. So currently, we use a technique whereby the beginning of the JITted section has, right after the setup code, a jump to the correct starting address--in the main thread case, this is just a jump to the next instruction (essentially a noop), but in the thread case, it's a jump to the function which the thread is going to run. So right now the JITted code for a secondary thread differs by one instruction from that for the main thread. We'll need to work out a different mechanism for handling this--probably just a tiny separate JITted section to set things up for a secondary thread, before doing an inter-section jump to the right place. JEff
[PATCH] Re: [perl #31978] [BUG] dynclasses broken
On Oct 13, 2004, at 6:36 PM, Will Coleda (via RT) wrote: # New Ticket Created by Will Coleda # Please include the string: [perl #31978] # in the subject line of all future correspondence about this issue. # URL: http://rt.perl.org:80/rt3/Ticket/Display.html?id=31978 One of the recent changes has broken: cd dynclasses make Which did, in fact, work for all of one checkout for me. =-) This is on a fresh checkout (not update), configure, make, cd dynclasses and... ... /Users/coke/research/parrot2/include/parrot/interpreter.h:61: error: conflicting types for `typedef struct Parrot_Interp*Parrot_Interp' /Users/coke/research/parrot2/include/parrot/platform_interface.h:124: error: previous declaration as `struct Parrot_Interp' The below patch should fix this. The problem is that ${ld} for Mac OS X became 'c++' because that's needed to link libparrot.dylib since ICU contains C++ code. But, dynclasses/build.pl tried to take a shortcut, and compile and link in one step. This caused a problem, because the c++ compiler was choking on an odd typedef of ours. So we need to build with just a C compiler. (We could probably link with one in this case too, but we don't have separate configs for linking the stuff in dynclasses, v. everything else.) So the fix here is just to compile and link as separate steps, always, rather than only when we need to combine multiple .o files. At the same time, I'm not sure why we need this construct in a header: struct Parrot_Interp; typedef struct Parrot_Interp *Parrot_Interp; I'm not surprised that chokes a C++ compiler, but I don't know why a C compiler tolerates it either. Not sure why this is necessary--I wonder if we could do something a bit less tricky? JEff Index: config/gen/makefiles/dynclasses_pl.in === RCS file: /cvs/public/parrot/config/gen/makefiles/dynclasses_pl.in,v retrieving revision 1.2 diff -u -b -r1.2 dynclasses_pl.in --- config/gen/makefiles/dynclasses_pl.in 8 Oct 2004 07:08:31 - 1.2 +++ config/gen/makefiles/dynclasses_pl.in 14 Oct 2004 06:08:27 - @@ -13,14 +13,6 @@ our $PMC2C = $PERL ${build_dir}${slash}classes${slash}pmc2c2.pl; # Actual commands -sub compile_loadable_cmd { -my ($target, $source) = @_; -$LD $CFLAGS $LDFLAGS $LD_LOAD_FLAGS . -${cc_o_out} . $target . . --I${build_dir}${slash}include -I${build_dir}${slash}classes . -$source; -}; - sub compile_cmd { my ($target, $source) = @_; $CC $CFLAGS . @@ -58,14 +50,8 @@ } elsif ($mode eq 'compile') { my ($group_files, $pmc_group) = gather_groups(@pmcs); -my @grouped_pmcs = grep { exists $pmc_group-{$_} } @pmcs; -my @ungrouped_pmcs = grep { ! exists $pmc_group-{$_} } @pmcs; - -# Convert X.c - X.so for all non-grouped X.c -compile_loadable($_) foreach (@ungrouped_pmcs); - -# Convert X.c - X.o for all grouped X.c -compile($_) foreach (@grouped_pmcs); +# Convert X.c - X.o for all X.c +compile($_) foreach (@pmcs); # lib-GROUP.c for my $group (keys %$group_files) { @@ -80,6 +66,11 @@ partial_link($group, lib-$group, @$pmcs) or die partial link of $group failed ($?)\n; } + +# Link non-grouped PMCs individually +my @ungrouped_pmcs = grep { ! exists $pmc_group-{$_} } @pmcs; +partial_link($_, $_) foreach (@ungrouped_pmcs); + } elsif ($mode eq 'copy') { # Copy *.so - destination, where destination is the first # argument, given as --destination=DIRECTORY @@ -166,17 +157,6 @@ } } -sub compile_loadable { -my ($src_stem, $dest_stem) = @_; -$dest_stem ||= $src_stem; -if (needs_build($dest_stem$LOAD_EXT, $src_stem.c)) { -run(compile_loadable_cmd($dest_stem$LOAD_EXT, $src_stem.c)) - or die compile $src_stem.c failed ($?)\n; -} else { -print $dest_stem$LOAD_EXT is up to date\n; -} -} - sub partial_link { my ($group, @stems) = @_; my @sources = map { $_$O } @stems;
Re: dynamically loadable modules
On Oct 8, 2004, at 7:54 AM, Andy Dougherty wrote: On Thu, 7 Oct 2004, Steve Fink wrote: So what I need is names for these. At the moment, I'm mostly using $(SO) for shared lib extensions, $(DYNMOD) for d-l-modules. The buildflags I gneerally call $(LD_SHARED) or something with shared for shared libs, and something like $(LD_DYNMOD_FLAGS) for d-l-modules. Clearly, I'm not very experienced with dealing with these things across platforms, so I was hoping somebody (Andy?) might have a better sense for what these things are called. Sorry -- offhand I don't have any sense of any standard names I don't think it's common to have a split like the dylib v. bundle split on Mac OS X, so there's probably not a common convention. JEff
Re: ICU causing make to fail w/o --prefix=`pwd`
On Oct 8, 2004, at 9:03 AM, Steve Fink wrote: If I just do perl Configure.pl make right now, it builds the parrot executable ok but then fails when it tries to compile the library .imc files. It's looking for the icu data dir in $(prefix)/blib/lib/2.6.1. It works if I do perl Configure.pl --prefix=$(pwd) make or set PARROT_ICU_DATA_DIR, but this seems like an unfriendly default for developers. I have a similar problem with the search path for loadable modules. I think I probably broke this, btw, when I repaired 'make install'. I had previously bandaged over the problem by defaulting ${prefix} to the top-level directory. But I'm not sure how to fix it. It's probably the issue of what to use at build-time v. install-time; I'd expect that nothing should be looked for via ${prefix} at build time, and rather it would just be used at install time (and possibly on the link line when building the shared libparrot, to set the install name). JEff
Re: ICU causing make to fail w/o --prefix=`pwd`
On Oct 8, 2004, at 9:24 AM, Jeff Clites wrote: On Oct 8, 2004, at 9:03 AM, Steve Fink wrote: If I just do perl Configure.pl make right now, it builds the parrot executable ok but then fails when it tries to compile the library .imc files. It's looking for the icu data dir in $(prefix)/blib/lib/2.6.1. It works if I do perl Configure.pl --prefix=$(pwd) make or set PARROT_ICU_DATA_DIR, but this seems like an unfriendly default for developers. I have a similar problem with the search path for loadable modules. I think I probably broke this, btw, when I repaired 'make install'. I had previously bandaged over the problem by defaulting ${prefix} to the top-level directory. But I'm not sure how to fix it. It's probably the issue of what to use at build-time v. install-time; I'd expect that nothing should be looked for via ${prefix} at build time, and rather it would just be used at install time (and possibly on the link line when building the shared libparrot, to set the install name). Ignore what I said. I misread that as failure when building the ICU data files. Sorry about that. JEff
Re: Python bound and unbound methods
On Oct 6, 2004, at 11:49 PM, Leopold Toetsch wrote: Jeff Clites [EMAIL PROTECTED] wrote: 3) I won't mention the problem of languages which allow an object to have instance variables and instance methods of the same name (so that in Python, a.b would be ambiguous if a is an object from such a language). Well, Python has that very problem. By dynamically defining an instance variable, a method with that same name becomes inaccessible. Well, for Python itself that's not a problem per se, it's a fundamental part of the design--there's only one slot for a given name, and it can hold a sub or something else, but it's just one slot. (You don't really have named methods, you just have named slots that may or may not hold subs.) That's just the way Python objects work. But for objects coming over from another language, you might really have separate slots for subs v. data, but if these are accessed from Python code, you don't have a way to specify which one you mean. It's very similar to the namespace problem with language crossing. JEff
Re: Python bound and unbound methods
On Oct 6, 2004, at 1:11 AM, Leopold Toetsch wrote: Jeff Clites [EMAIL PROTECTED] wrote: To put it another way, the expression foo.bar() in Python doesn't really parse as invoke method bar on object foo, but rather as lookup attribute bar on object foo Well, there isn't much difference here. invoke method bar implies a method lookup that normally checks the method cache for existing (statically built) methods. If a method isn't found there, the dynamic plan B comes in. And if Python code overrides a method (attribute slot) the method cache has to be invalidated. A few things though: 1) In my case of a = x.printMe, we need to be prepared to return a function created by currying a method using that particular instance, and for b = printWrapper.printMe we need to return a function which does type checking on its first argument (then calls the relevant method on it)--certainly doable, just odd. 2) I'd expect the method cache to be per-class, but Python can change an attribute slot on a per-instance basis (as well as a per-class basis), so we can't really use a per-class method cache (or, we need a flag on particular instances which tell us not to use it for them). 3) I won't mention the problem of languages which allow an object to have instance variables and instance methods of the same name (so that in Python, a.b would be ambiguous if a is an object from such a language). More fun is probably to get the same inheritance (mro) as Python. Yes, true, and I guess related--Python must first look for things overridden on the particular instance, I think, even before looking at its class. JEff
Python bound and unbound methods
Python's method call semantics allows you to look up a method of an object as an attribute, store it in a variable, and later invoke it like a regular function. This works similarly if you do the lookup on the class object, but when you invoke the function you need to pass in an instance as an argument. Consider: class printWrapper: def __init__(self, message): self.message = message def printMe(self): print self.message x = printWrapper(foo) x.printMe() # prints foo a = x.printMe a()# bound method call; prints foo b = printWrapper.printMe b(x) # unbound method call; prints foo Do we have plans on how we might implement this via Parrot? To put it another way, the expression foo.bar() in Python doesn't really parse as invoke method bar on object foo, but rather as lookup attribute bar on object foo, and call the result as a function. That's an incredibly flexible and compact semantics, and let's you do stuff like this: class omniWrapper: def __init__(self, message): self.message = message def printMe(self): print self.message def __getattr__(self, attrname): return self.printMe z = omniWrapper(hello) z.printMe() # prints hello z.igloo() # also prints hello z.anythingAtAll() # anything prints hello That is, you can appear to be calling methods that aren't there at all. It seems odd, but makes perfect sense in light of the above description--method calls in Python are not what you might think, based on how other languages act. But that would seem to imply that Python would need a very different method call infrastructure than Perl -- a.b() in Python means something very different than $a.b() in Perl. (That of course brings up questions of how things act when Python tries to call a method on a Perl object--syntax, semantics.) JEff
Re: Namespaces, part 2
On Oct 4, 2004, at 8:25 AM, Dan Sugalski wrote: Okay, since we've got the *basic* semantics down (unified namespace, namespace entries get a post-pended null character) I'll ask again, what about subs? Do they get name-mangled too? $Px = find_global [key; key; key], 'name' As Leo pointed out in a thread of the same name last year, this is a new syntax--keyed access on nothing. I assume you mean for [key; key; key] to serve as a sort of literal syntax for an array of strings? If so, we should make the syntax for that sort of thing explicit. Or, we can do what Leo suggested (in a thread of the same name last year, and more recently), and write this as keyed access on a particular namespace (typically, the root namespace): $P0 = root_namespace $P1 = find_global $P0['Foo'; 'Bar'; 'Baz'], 'xyzzy' The case where the namespace is specified by string I think should go away. I assume by specified by string, you mean a string such as Foo::Bar? If so, I agree, and I'll note that it would be possible to write per-language utility functions to parse these apart--compilers will need to be able to process these, in some form, and so might programmers in general, but it doesn't need to be an op. That is, if I have the namespace Foo, Foo::Bar, and Foo::Bar::Baz, and then have a thing named xyzzy in Foo::Bar::Baz, I can get it by doing: $P1 = find_global ['Foo'; 'Bar'; 'Baz'], 'xyzzy' or $P0 = find_global ['Foo'] $P1 = find_global $P0, ['Bar'; 'Baz'] $P2 = find_global $P1, 'xyzzy' ... That is, if we don't specify the name of the thing in the namespace we get the namespace PMC itself, which we can then go look things up in. This is handy since it means code can cache the namespace PMC so if there are a dozen variables to go fetch out, we don't have to do the multi-level hash lookup each time. (That'd be nasty) Several things here: 1) By your name mangling scheme, it seems we don't need a special syntax for looking up a namespace--it's just something else in the namespace above it, so looking up Foo::Bar would be: $P0 = root_namespace $P1 = find_global $P0['Foo'], 'Bar' But if everything's lumped together and name-mangled in a non-segmented namespace, then you don't need find_global, you just need this (as Leo suggested): $P0 = root_namespace $P1 = $P0['Foo'; 'Bar'] # P1 holds Foo::Bar $P2 = $P1['Baz'; 'xyzzy'] # P2 holds xyzzy from Foo::Bar::Baz $P2 = $P0['Foo'; 'Bar'; 'Baz'; 'xyzzy'] # same thing I was formerly a proponent of keeping the final thing in a separate parameter, but with your current mangling-plus-flat-namespace proposal, I don't see a reason for it. 2) What's putting in the trailing null bytes (HLL programmer, compiler, or Parrot's implementation of the lookup)? I'd assume it would be the HLL compiler, so shouldn't the above be: $P2 = $P0['Foo\0'; 'Bar\0'; 'Baz\0'; '$xyzzy'] #or at least, shouldn't the PASM variant look like this (In particular, I noticed you said, a thing named xyzzy, but didn't include a sigil?) 3) Canonical decomposition: Consider the following: $P0 = root_namespace $P2 = $P0['Foo\0'; 'Bar\0'; 'Baz\0'; '$xyzzy'] v. $P0 = root_namespace $P0 = $P0['Foo\0'] $P0 = $P0['Bar\0'] $P0 = $P0['Baz\0'] $P2 = $P0['$xyzzy'] (The example's the same even with your other syntax.) These appear equivalent, _but_ with the former, the root namespace could take into consideration to whole list keys, and decide what to return based on that information; with the latter, it only sees 'Foo\0'. So in the general case (in light of namespace tie-ing), they could do different things. The upshot of this is that we need to make explicit the algorithm followed by a multi-keyed lookup--basically, iterative v. recursive, if you think it through. Recursive is more flexible and powerful (because any namespace along the way can short circuit the rest of the lookup), but means that HLL compilers must emit the first option above (with an iterative approach, where Parrot controls the algorithm and namespaces have less control, a compiler would have a choice). 4) Similar to (3), you can't do much caching of a namespace PMC, in light of tying and such. Consider: $a = $Foo::Bar::bar; somesub(); $b = $Foo::Bar::zoo; A compiler can't safely optimize this to cache the lookup of the Foo::Bar namespace, because somesub() might perform tying of one of the relevant namespaces, or otherwise rearrange the hierarchy, and you'd get the wrong $zoo. Even without somesub(), in light of namespace tying it's possible that the lookup of $bar actually caused a namespace rearrangement, so you can't even cache across adjacent lookups (or a lookup and a store, such as $Foo::Bar::bar = $Foo::Bar::zoo). 5) Python. Language crossing issues aside, there are issues with Python itself. Python does not know, at compile-time, what's
Re: Namespaces, part 2
On Oct 4, 2004, at 9:58 PM, Brent 'Dax' Royal-Gordon wrote: Tim Bunce [EMAIL PROTECTED] wrote: Now, with that out of the way, let's talk about overlaid namespaces. I don't think I ever read a description of what the purpose of this was. I get the what but not the why. Without the why it's hard to critique the how. Clearly I'm not Dan, but I think the idea here is that, for example, in the following code: module Foo::Bar { class Baz { $quux } } You can have the current namespace actually be [ ::Foo::Bar::Baz, ::Foo::Bar, ::* ] (or, for the last one, whatever the namespace that @*ARGS and friends are in is called), so that the search for $quux can be done very easily. This may have changed for Perl6, but at least for Perl5, non-lexicals are only ever looked for in the current package, the top-level (main) namespace, plus the pre-defined variables (don't know if these are handled at compile-time), according to the Camel. So having an arbitrary search list might not be used in the Perl case. JEff
Compilation paradigms
Unless we've got this somewhere already, I think it would be useful to have some examples of simple Perl6 code, and the bytecode we expect this to compile down to. This would help to fix some ideas, and make sure that everyone's on the same page. As an example, what bytecode goes with the following: # simple case sub example1($num) { my $x = 1; return $num + $x; } # nested lexical scopes sub example2($num) { my $x = 1; { my $x = 2; return $num + $x; } } # nested lexical scopes, with eval sub example3($str) { my $x = 1; { my $x = 2; my $y = $str; eval $str; } } JEff
Re: Namespaces again
On Sep 29, 2004, at 9:01 PM, Brent 'Dax' Royal-Gordon wrote: [Argh...] Chip Salzenberg [EMAIL PROTECTED] wrote: parrot_alias(a, 'b', # dest: Python is unified, no need for a category here a, 'b', 'scalar') # src: Perl is not unified, so source category is required parrot_alias(a, 'c', a, 'c', 'array')# here's a different category, to get '@c' or some such. Yes it's ugly. But if we can't fix a ditch, the least we can do is put a big friendly warning sign on it. Turns out the ugliness isn't needed anyway; Python already has a syntax to access a namespace entry by string name (via __dict__), and Python import copy anyway, rather than alias. That is, this: from os import WNOHANG creates a WNOHANG variable in the local namespace with the value of os. WNOHANG, but it's a copy--changing one changes the other. So, one could do this: foo = Perl.Blah.__dict__[some-japanese-variable-name] when the variable name isn't something Python can accommodate directly, and get the same result. Another way to do it would be to have each category actually be a namespace. In other words, Perl's namespaces are structured like this: File = package { ns = package { Find = package { sub = package { find = { ... } } }, Path = package { sub = package { BUILD = { ... }, DESTROY = { ... } } } ... }, scalar = package { some_idiot = put a variable in this package } } And Python would access them like so: File.ns.Path.sub.new() Not perfect, certainly, but it would work, and be reasonably elegant. Yep, this is exactly what I had in mind. Everything is hash-like anyway, so whether you call the category a namespace or just something namespace-like (since it's hash-like), doesn't matter so much, and on the Python side they're all just things which allow named attribute lookup. (This does pose a problem going the other way, but I suspect Perl could simply mark its own packages in some way, and fall back to a simpler scheme, such as ignore the sigil, when it's munging another language's namespaces.) I'd think that, when importing a module, you'd need a concept of what structure of namespace you're dealing with (which maps roughly to what language was used to implement the module, though flat namespace languages might all act the same). As I touched on in another post, I think you'd need to concept of a wrapper--for instance, when Python code pulls in a Perl module, it would get a wrapper which acts Pythonish, and acts as a go-between for the real namespace. (Probably, this would be via namespace tie-ing.) In this light, I think that going the other way is easy--all Python variables would look to Perl like scalars holding references (since this is the semantics that Python variables have). But see my recent post in the Namespaces, part 1 (new bits) thread for some caveats. JEff
Re: Namespaces again
On Oct 1, 2004, at 5:45 AM, Leopold Toetsch wrote: Jens Rieks [EMAIL PROTECTED] wrote: On Friday 01 October 2004 08:42, Leopold Toetsch wrote: sucks a lot less than making python programmers say import Foo.ns.Bar.scalar.baz But OTOH I can imagine that finally standard modules are present in Parrot that do the right thing. E.g. in Python import os would import a Parrot module that wraps these functions to Parrot. So basically, when you have: os.setsid() # Python POSIX::setsid(); # Perl these two functions would be the same, living in some Parrot lib and aliased to the language's equivalent. So the import or use syntax shouldn't be changed just to accomodate Parrot's namespace internals. Well, it wouldn't be a syntax change per-se, and it wouldn't be to accommodate Parrot--this is about language crossing. You'd say, in Python, import Perl.ns.Foo.ns.Bar.scalar.baz only in the case where the namespaces you are dealing with are of Perl origin. This is Python accommodating Perl (or really, Perl accommodating Python). Perl has more structure to its namespaces than Python does--that's just a fact, and this is one way to deal with it. In your example above, it looks like you're talking about one function, ostensibly living in two namespaces. That's fine--if you author a module in pasm (no HLL involved), then you get to structure your namespace any way you want. In this case, it sounds like you'd want your module to implement two different interfaces, one intended for Perl, and one intended for Python. That would be fine, and it would be optimal for both languages--at the cost of extra work for the module implementor, explicitly taking into account multiple languages. In the case of someone just writing a module in Perl, not thinking about other languages at all, the compiler will create a Perl-style interface for its namespace, and Python programmers will still be able to use this module, just with a somewhat odd-looking interface. But this isn't a new syntax for Python--from a Python perspective, that really is the structure of that module. It will just look like Perl programmers create really nested modules. JEff
Re: Namespaces, part 1 (new bits)
More detailed responses are below, but some general comments first: I think that no matter what the approach, there's an unavoidable mismatch between Perl and Python when it comes to variable naming, it's going to be a bit awkward to access Perl variables from within Python. I don't see any way around that. Either of the obvious approaches will have issues to deal with; these approaches are: 1) Treat Perl variables as having the sigil as part of the name. -or- 2) Treat Perl variables as not including the sigil in the name, but have multiple categories. For Python, the problem with (1) is that no Perl variables will have legal names from Python's point of view. With (2), the variable names are okay for Python, but Python doesn't have the concept of multiple categories of variables. But, that's not as bad as it sounds, and I'll spell out how either of those could work. But with either approach, the integration won't be seamless--using a Perl-originating module (namespace) in Python will involve more work on the Python side than is required when using a Python-originating module (in Python). Here's how either approach could work. Consider the following Perl(5) package as an example: # Perl5 package Gizmo; $foo = 1; @foo = ('a'); %foo = ('b' = 'c'); sub foo { return 1; } Approach 1) Sigil is part of the name. import Perl.Gizmo x = Perl.Gizmo.foo # error--not defined x = Perl.Gizmo.__dict__['$foo'] # x now holds the value of Perl's $foo Perl.Gizmo.__dict__['$foo'] = 'hello' # Perl's $foo now holds a Python string # similarly for hashes, arrays, etc.; subs would probably need a sigil, maybe '' That would actually work. It's awkward, but it reflects the fact that Python's symbol tables (like Perl's) can deal with entries which you couldn't create with the normal Python syntax. Approach 2) Sigil is not part of the name. Use categories. import Perl.Gizmo x = Perl.Gizmo.scalars.foo # x now holds the value of Perl's $foo Perl.Gizmo.scalars.foo = 'hello' # Perl's $foo now holds a Python string x = Perl.Gizmo.arrays.foo # x now holds the value of Perl's @foo Either approach would work from a Python perspective. The syntax of the second, to my eyes, is a bit less awkward. (Note that even with approach 2, you'd need to resort to the explicit Perl.Gizmo.scalars.__dict__[] style to access variables with non-ASCII names, but at least you wouldn't have to do that for *all* variables.) Both approaches require some special Pythonish behavior of the Perl-originated namespace. Parrot's namespace-tie-ing mechanism could probably be leveraged to provide this behavior and avoid putting the burden on the Perl side (i.e., the extra work should be on the import side, since it's reaping the benefit of the language crossing). What I'm thinking here is that when Python tries to import a Parrot module, it actually gets a namespace object which is a wrapper (or adaptor) around the actual Perl namespace. (So in the example above, the Perl.Gizmo that Python gets is actually a special namespace which mediates interaction with the real Perl Gizmo namespace.) Here's what the adaptor would need to do in each case: Case 1) The adaptor would need to expose a __dict__ attribute to provide a hash-like interface to the contents of the namespace. This is necessary because the real Perl namespace won't have the necessary __dict__ variable in place. This would be simple to implement--namespaces are already hash-like. Case 2) The adaptor would satisfy a request for Perl.Gizmo.scalars.foo by asking the real Gizmo namespace for the entry foo in the scalars category. This would also be simple to implement if we have category-structured namespaces. Either approach requires that the import mechanism have an awareness of the language of the imported namespace (i.e., Perl modules need to be wrapped but Python modules don't, for import from Python code), but that's supplied by the Perl. prefix anyway. Going from Python to Perl is easy--all Python variables look like Perl-scalars-holding-references, though we might want to special-case strings and numbers. See below for further notes on this. Further comments, some of which will be redundant with the above: On Sep 30, 2004, at 1:00 AM, Leopold Toetsch wrote: Jeff Clites [EMAIL PROTECTED] wrote: First off, Perl5 doesn't describe itself that way. The Camel states, Note that we can use the same name for $days, @days, and %days without Perl getting confused. While that's fine for Perl it doesn't help, if you want to access one distinct days from Python: import days from Perl.foo # which one? So it's true that $foo and @foo are different items, but that can be stated as, the scalar 'foo' and the array 'foo' are different items, with the same names. That does only work, if the context provides enough information, which one of the foos should be used. That information isn't present always. What I mean is, if someone
Re: Namespaces
On Sep 29, 2004, at 7:25 AM, Dan Sugalski wrote: Okay, after seeing all the back and forth, here's what we're going to do. Namespaces are going to be *simple*. They do two things, and only two things. 1) They provide a hierarchy for other namespaces 2) They bind names to PMCs That's it. No typing, no classification, no nothing. By postpending a null character, below, you _are_ doing typing/classification, of course. And, what about subs? If languages want to do that, then they'd better darned well do it themselves. Yes, this is going to make interoperability a pain at the variable level, but it's pretty clear that we're just not going to be able to do that. The next step, then, is to sketch out ops to query, read, and write name/PMC bindings and query/read/write/overlay hierarchies. (Which, I realize, many languages just won't use, and that's fine) I think it's best to go with a unified hierarchy/variable namespace, so we'll postpend a NUL to the end of non-variables. Printing NULs is a pain, so we should make sure that we've got good general-purpose binary printing routines people can use. As Larry said, it's best to prepend it. And then we can call it a sigil. And since we're name mangling, pick a printable character like : for the prefix, since we'll know the initial character is always really a throw-away encoding of the syntactic category. Then we're back to _exactly_ my scheme, with the syntactic category names as single characters, and as an implementation detail shoving the category and name together in a single string instead of keeping them separate. And of course this will all be awkward for languages with distinct syntactic categories, but without natural name mangling. And it will probably result in more intermediate strings being allocated. This is exactly like open(IN, file) v. open(IN, , file), and we're deciding that the former is vastly superior. And of course, you do realize that in Perl5, this: $data[1] refers to the variable @data, so that even for Perl5 there will be name-mangling (not just the ability to use the literal sigil-plus-namae as seen in the source). Yes, I know, we could do better. If anyone wants to do so, go lock Larry, Matz, and Guido in a room together. When they hash out the language semantics we'll let 'em out and then do better. :) We don't need Larry, Matz, and Guido to agree on anything here. I explained a scheme which would work for all of their languages, as well as Scheme, Common Lisp, Objective-C, Java, C, and probably anything else. It just requires acknowledging that different languages have different sets of syntactic categories, rather than pretending things are simpler than they are. JEff
Re: Namespaces, part 1 (new bits)
On Sep 29, 2004, at 2:53 AM, Leopold Toetsch wrote: Dan Sugalski [EMAIL PROTECTED] wrote: Okay, so we've got two points of dispute: 1) Jeff doesn't think the sigil should be part of the variable name Which isn't practicable. We can't strip off the sigil for perl5. It's part of the variable name, $foo and @foo are different items. Those statements don't follow from one another. :) First off, Perl5 doesn't describe itself that way. The Camel states, Note that we can use the same name for $days, @days, and %days without Perl getting confused. I'm asserting that it works perfectly well (and seems to have been the original intent) to say that Perl allows for a scalar, a hash, and an array all named foo, and the grammar always makes it clear which one you mean (mostly, via sigils). So it's true that $foo and @foo are different items, but that can be stated as, the scalar 'foo' and the array 'foo' are different items, with the same names. (Just like there can be both a person named April and a month named April in English.) Here are some further demonstrations that the seeming intent of Perl5 is not to treat the sigil as actually part of the name, but as a feature of the grammar which indicates the syntactic category of the name: $hello and ${hello} are the same thing, but $hel{lo} is not -- the name can separate from the sigil $array[1] refers to @array, not $array -- the sigil changes depending on context, even for a given item Secondly, Perl just clouds the issue, since it _could_ work either way. For other languages, you have distinct syntactic categories, but without the name decoration. For example, in Common Lisp, this: (foo foo) means, call the function foo, and pass the variable foo as an argument. So if the function foo doubles numbers, and the variable foo is set to 11, then the above evaluates to 22. The reason I'm taking this as important, despite Common Lisp's not being a target language for Parrot, is that it's pointing out that namespaces across languages have the concept of dealing with multiple syntactic categories, though the number of such categories varies between languages. Ruby and Python are simple--they have unified namespaces, or so it seems--and Perl has a syntax which allows us to pretend that it has a unified namespace, although I think that's stretching the truth. (And, I think there's still an issue with namespace names and sub names, since they don't have sigils in the grammar--at least not as normally written.) But other languages aren't so simple, and if we oversimplify our treatment of namespaces, then we end up with something less elegant and less flexible that we could have. If you want to use a perl5 module from Python which has both $foo and @foo exported, we can just pitch a fit. We should be able to handle accessing either, if Python provides a syntax for doing so. And: we can't attach hints to the namespace lookup because you just don't know, if Python wants the scalar foo or the array foo. There is exactly one foo object that Python can use, that's it. That's not accurate, and it's not a hint, it's a demand--the programmer should know exactly which one he wants (which is especially true if you are trying to think of the sigil as part of the name). Possible syntaxes within Python: a = lookupPerlScalar(foo); b = lookupPerlSub(foo); c = lookupPerlArray(foo); or its: a = lookupFromPerl(scalar, foo); or it's: a = lookupInParrotNamespace(Perl, scalar, foo); Since Python deals in references only, assignment syntax could work just fine for this, but if someone wanted a more aliasing-like syntax, it could work like what Chip suggested in another thread: parrot_alias(a, 'b', # dest: Python is unified, no need for a category here a, 'b', 'scalar') # src: Perl is not unified, so source category is required parrot_alias(a, 'c', a, 'c', 'array')# here's a different category, to get '@c' or some such. Yes it's ugly. But if we can't fix a ditch, the least we can do is put a big friendly warning sign on it. I've lost hope for transparent aliasing, but it would work partially so say, imports from Perl to Python can automatically alias scalars to Python variables of the same name, but arrays and hashes you have to pull in manually, and alias to an explicitly-specified name on the Python side. It will never just work for all things, since this is a valid identifier in Common Lisp: *foo-foo* So to exploit the full power of Parrot, languages will need to have syntaxes/functionality/API to access foreign namespaces, but things could be made transparent for at least a subset of variables. (And as an example, not _all_ identifiers in Common Lisp are so exotic.) Python allows only bare names in the import statement: from a import foo [ as a_foo ] but not: from a import @foo [ as a_foo ] This seems to make things worse for treating the sigil as part of the name,
Re: Namespaces again
On Sep 27, 2004, at 8:55 AM, Dan Sugalski wrote: Okay, I've come to realize that it really helps if I'm clear about what I want, which kinda requires being clear about what I want. There are two things in the namespaces I'm concerned about. First are the actual objects one grabs out. Variables, sub objects, class objects -- actual stuff. Languages seem to fall into two categories -- those that group subroutines in with data and those that don't. (Most generally do) Second there's the structure of the namespace. I really do want Foo::Bar to be handled with a Foo entry in the top-level namespace, and a Bar entry in the Foo namespace, and nothing that anyone does to any Foo object at the top level can damage that. What I proposed in my 9/26/2004 post to the Namespaces, part 1 (new bits) thread directly accommodates all that. Here is how that looks in data structure terms. (I'll use hash notation here, though the actual implementation could optimize.) Let's say that all you have around are $Foo and $Foo::Bar::baz -- that is, a Foo variable at the top level, and a bar variable inside a Bar namespace inside a Foo namespace. The namespaces involved could look like this: top-level namespace (say this is namespace #1): { variables = { Foo = PerlScalar PMC (or whatever) }, namespaces = { Foo = PerlNamespace PMC, call namespace #2 } } namespace #2 (the namespace from above): { namespaces = { Bar = PerlNamespace PMC, call namespace #3 } } namespace #3 (the namespace from immediately above): { variables = { baz = PerlScalar PMC } } It doesn't matter if the sigils are treated as part of the name--the structure's the same, you just have $Foo and Foo:: up there instead. (Though Perl maybe could have done without the structure for variable v. namespace, other languages don't have a natural syntax to distinguish them.) And of course, you'd have a subs = ... section if we had any subs in our example. My above-mentioned post explained what you'd do for Python, and how it fits well into this. And for languages which need more sections, they would be accommodated as well. The fact that the Foo class has an object named Foo at the top level which you can fool with is separate from the fact that there's a Foo *namespace* at the top level. Using a filesystem analogy, I want to have a directory and file named Foo, with file operations not touching the directory, and directory operations not touching the file. Yep, that does that. It seems like the simplest thing to do is to have a *real* unified system and go with a prefix or suffix scheme for namespaces that the namespace ops automatically pre/post-pend on the names. I don't think there's any need for name mangling--that's the sort of thing you do when you need to shoehorn extra information into a structure that wasn't designed to hold it. If you want to name mangle, _and_ not arbitrarily restrict what characters a language might allow in identifiers, then it makes sense to give everything a prefix, and it's unambiguous that the prefix is everything up to your first separator character (say, slash); for example: namespace/Foo, variable/Foo--there's no ambiguity or restriction, for instance variable/foo/bar is just the foo/bar variable in some language which allows slashes inside of variable names. And just to close the loop, you'd still express your $Foo::Bar::baz lookup like: lookupVariableInNamespace P1, [Foo; Bar], baz # the things in the [...] are always namespace names Logically, that's identical to what I have above--it's just another way to express the same structure, just an internal implementation detail, with some pros (maybe one less hash lookup) and cons (more intermediate strings created). You'd still say a given namespace has different sections to accommodate different categories of entities. JEff
Re: Namespaces again
On Sep 28, 2004, at 7:02 AM, Aaron Sherman wrote: Rather than trying to shuffle through the keyboard and find that special character that can be used, why not have each language do it the way that language is comfortable (e.g. place it in the regular namespace as a variable like Python or place it in the regular namespace, but append noise like Perl or hide it in some creative way for other languages). For the most part, there's no performance penalty in having a callback that the language/library/compiler provides because access to the objects in question will be via a PMC, and only LOOKUP of that PMC will be via namespace, no? In that way, you could: namespace_register perl5_namespace_callback, Perl5 namespace_register python_namespace_callback, Python [...] namespace_lookupP6, F\0o::Bar, Perl5 namespace_lookupP7, foo.bar, Python the namespace callback could take a string and return whatever Parrot needs to look up a namespace (Array PMC?), having encoded it according to Parrot's rules. That's similar in spirit to what I proposed of allowing PMC-subclassing of the default ParrotNamespace, so that namespaces created from different languages (often implicitly) could have different behaviors. But I'd keep the pulling-apart of F\0o::Bar into [F\0o; Bar] a compile-time task, so that at runtime the work's already been done (since the compiler knows what language it's compiling). JEff
Re: Namespaces again
On Sep 28, 2004, at 8:58 AM, Jeff Clites wrote: And just to close the loop, you'd still express your $Foo::Bar::baz lookup like: lookupVariableInNamespace P1, [Foo; Bar], baz # the things in the [...] are always namespace names Here are more examples, just to be clear: (and the actual op names would be different, but I'm trying to be unambiguous here): # $Foo::Bar::baz, or presumably Python's Foo.Bar.baz lookupVariableInNamespace P1, [Foo; Bar], baz # $::foo, or foo as a variable in the top-level namespace lookupVariableInNamespace P1, [], foo # Foo::bar lookupSubInNamespace P1, [Foo], bar Now, the above are shortcuts (or optimizations) for the generalized op: lookupInNamespace P1, [Foo; Bar], .VARIABLE, baz And, they're also shortcuts for doing things more manually: # $Foo::Bar::baz lookupNamespaceInNamespace P0, [], Foo # or rootNamespace P0 lookupNamespace P1, P0, Bar #lookup namespace Bar in namespace in P0 lookupVariable P2, P1, baz #lookup variable baz in namespace in P1 And if we must treat the sigil as part of the name for Perl6 at the parrot level, we just get stuff like: # $Foo::Bar::baz lookupVariableInNamespace P1, [Foo::; Bar::], $baz But, it seems better to not do that. JEff
Re: Namespaces again
On Sep 28, 2004, at 9:54 AM, Chip Salzenberg wrote: According to Jeff Clites: Let's say that all you have around are $Foo and $Foo::Bar::baz ... top-level namespace (say this is namespace #1): { variables = { Foo = PerlScalar PMC (or whatever) }, namespaces = { Foo = PerlNamespace PMC, call namespace #2 } } I'm a bit confused by this example. Don't you mean: variables = { '$Foo' = PerlScalar PMC (or whatever) }, i.e. with the '$' in the key? Sorry, accidentally sidestepped the array/scalar/hash issue. Given $foo, @foo, and %foo, we could either do: variables = { '$foo' = whatever, '@foo' = whatever, '%foo' = whatever } -or- scalars = { 'foo' = whatever }, arrays = { 'foo' = whatever }, hashes = { 'foo' = whatever } The infrastructure's the same, it just depends on whether we want to (at the Parrot level) think of Perl as having one category of variables, or three. We could even have: scalars = { '$foo' = whatever }, arrays = { '@foo' = whatever }, hashes = { '%foo' = whatever } ...if we wanted to model Perl as having three categories, but still treat the sigils as part of the name. I think the second option has the most potential for smooth interoperability with other languages. Presumably, all of Pythons stuff would go in a single bucket in a Python-originating namespace. So it might look like: { whatever = { a = namespace, b = object, c = sub } } Where there's only one category, and whatever could be references or objects or variables or scalars or whatever made the most sense from a cross-language standpoint. (I think the different semantics of Perl v. Python variables means that it might be appropriate for the categories to be different between the two languages, though Python and Ruby might match one another. But, as all Python variables are references, they're sorta-like Perl scalars, so maybe the category for Perl-scalars should be the same as that for Python-references--just depends on what works best for cross-language access, but the infrastructure's the same.) And what I mean here is that how many categories are inside a given namespace depends on where it was created (in Perl code v. Python code, for example), but after that point the usage is the same. JEff
Re: Namespaces again
On Sep 28, 2004, at 11:26 AM, Chip Salzenberg wrote: According to Jeff Clites: top-level namespace (say this is namespace #1): { variables = { Foo = PerlScalar PMC (or whatever) }, namespaces = { Foo = PerlNamespace PMC, call namespace #2 } } I think I get it. You're replacing sigil characters and associated name mangling, turning it into explicit named categories, thus avoiding lots of anticipated string tricks. Yep, exactly. And it works for Perl, but also fits languages which don't have a built-in mangling-via-sigils, yet still have different categories (like Common Lisp, which allows functions and variables of the same name). And the named categories don't necessarily imply an additional hash lookup--with specialized ops for the common categories, we can fast-path getting to the correct hash to do the final lookup. JEff
Re: Namespaces again
On Sep 28, 2004, at 12:28 PM, Chip Salzenberg wrote: According to Dan Sugalski: At 11:58 AM -0700 9/28/04, Jeff Clites wrote: On Sep 28, 2004, at 11:26 AM, Chip Salzenberg wrote: According to Jeff Clites: top-level namespace (say this is namespace #1): { variables = { Foo = PerlScalar PMC (or whatever) }, namespaces = { Foo = PerlNamespace PMC, call namespace #2 } } I think I get it. You're replacing sigil characters and associated name mangling, turning it into explicit named categories, thus avoiding lots of anticipated string tricks. Yep, exactly. And unfortunately dies a horrible death for languages that don't categorize the same way as perl. :( The horrible death you fear is unavoidable. The variable categories are an impedance mismatch that namespaces can't paper over. Name spaces have different dimensionality, if you will, in different languages. We can't *fix* that without changing the languages. You said on IRC that the Perl/Python gap would have to be bridged with one-shot namespace connection, something like Exporter. (Perl $a::b = Python 'a.b', Perl @a::c = Python 'a.c', etc.) So the import process could use Jeff-style categories to get some handle on what's being imported. e.g.: parrot_alias(a, 'b', # dest: Python is unified, no need for a category here a, 'b', 'scalar') # src: Perl is not unified, so source category is required parrot_alias(a, 'c', a, 'c', 'array')# here's a different category, to get '@c' or some such. Yes it's ugly. But if we can't fix a ditch, the least we can do is put a big friendly warning sign on it. Exactly. If Python want to alias to a Perl variable, it's going to have to specify the category one way or another. Doing string-fiddling by putting a sigil on the front of the string is one way, but just specifying a category explicitly is another. The latter is a much more natural thing to do, in regard to languages which don't themselves have sigils. And the string-fiddling seems like more of a hack--why try to shove the category and name information together, into a single string, rather than leaving them separate? I'm not even suggesting an implementation here really; it's just, namespaces need to handle multiple syntactic categories, and we need to have API to reflect this. And you hit the nail on the head that these are *syntactic* categories. These syntactic categories should always be known at compile-time (which is what makes them syntactic categories); for languages with sigils, its easy, and for languages without them (most), the grammar of the language defines it. And in terms of adaptors, I think that aliasing from Perl to Python is going to end up aliasing '[EMAIL PROTECTED]', and never '@c' really (since Python always holds references--really aliasing to '@c' would imply that assignment within Python would copy); conversely, going from Python to Perl would always end up as a scalar on the Perl side, since everything from Python would look like a reference to Perl. (At least, this seems like the most natural approach.) JEff
Re: Why lexical pads
On Sep 25, 2004, at 10:27 PM, Larry Wall wrote: On Sat, Sep 25, 2004 at 10:01:42PM -0700, Larry Wall wrote: : We've also said that MY is a pseudopackage referring to the current : lexical scope so that you can hand off your lexical scope to someone : else to read (but not modify, unless you are currently compiling : yourself). However, random subroutines are not allowed access : to your lexical scope unless you specifically give it to them, : with the exception of $_ (as in 1 above). Otherwise, what's the : point of lexical scoping? Note that this definition of MY as a *view* of the current lexical scope from a particular spot is exactly what we already supply to an Ceval, so we're not really asking for anything that isn't already needed implicitly. MY is just the general way to invoke the pessimization you would have to do for an Ceval anyway. A mildly interesting thought would be for Ceval to take additional parameters to make explicit what's visible to the eval'd code--essentially making the running of the code like a subroutine call. So the traditional Ceval would turn into something like eval $str, MY, but you could also have eval $str, $x, $y, or just eval $str, which would execute in an empty lexical scope. That would allow additional optimizations at compile-time (and make MY the sole transporter of lexical scope), since not every Ceval would need what MY provides, but even more importantly, it would allow the programmer to protect himself against accidentally referencing a lexical he didn't intend, just because the code in his string coincidentally used the same variable name. More optimization opportunities, and more explicit semantics. But that's now a language issues, so I'm cc-ing this over to there. JEff
Re: Why lexical pads
On Sep 25, 2004, at 10:14 AM, Dan Sugalski wrote: At 7:43 PM -0700 9/24/04, Jeff Clites wrote: On Sep 24, 2004, at 7:32 PM, Dan Sugalski wrote: At 7:28 PM -0700 9/24/04, Jeff Clites wrote: On Sep 24, 2004, at 6:51 PM, Aaron Sherman wrote: However, the point is still sound, and that WILL work in P6, as I understand it. Hmm, that's too bad--it could be quite an opportunity for optimization, if you could use-and-discard lexical information at compile-time, when you know there's no eval around to need it. Even if not it's going in anyway. The introspection abilities are more than worth the extra memory that the name hashes use. It's a compiler issue. Well... sorta. It's a specification issue. *Not* having pads of some sort, whether they're dull, bog-simple activation frames which're just arrays, or the more complex ordered hash system we've gone with, makes continuations untenable. You need a stable backing store, otherwise resuming a continuation's a dodgy thing. Wait--I think we're talking about two different things. I'm talking about optimizing away nested scopes _within_ a sub; for instance, consider: sub foo { my $x; ... $x = 2; if(blah) { my $x; ... $x = 3; if(goo) { my $x; ... $x = 4; } ++$x; } ++$x; } At each curly-brace, a new pad needs to be pushed, and something stored into its x slot (maybe just a pointer to a PerlReference also stored in a register), and at the end the pad needs to be popped (and presumably, later garbage-collected). But, if the semantics of your language are such that you can't look up lexical by name, then the fact that all 3 of those variables have the same name doesn't matter--you could change the names to $x, $y, $z, and create code would behave identically. That is, all the consequences of the lexical structure can be worked out at compile-time, and it doesn't need to be preserved at run-time. So a compiler could compile the above into just some register stores and branches--no need to push new lexical pads while inside the body of the sub, and that should be _much_ faster without the pad manipulation overhead. (But if there could be something like eval inside of those ... sections, then you _do_ have to save it all.) [And if, for continuations, those registers need to be stored away to something else, that's still only one pad needed per sub, rather than one needed per lexical scope within a sub.] But by, It's a compiler issue, I mean that parrot doesn't get to decide if those curly braces are supposed to mean something at runtime--the compiler make the decision as to whether it needs to emit pad-manipulation ops. (And it goes without saying that the compiler is bound by the semantics of the language it's compiling) As I already said, I'm not arguing that Parrot shouldn't have lexical pads, just that languages with appropriate semantics can benefit greatly in cases where they don't have to use them. (And for example, I believe I've heard of languages which have something like eval, but for which the execution of the eval'd string doesn't occur in the lexical scope where it's invoked, so they never need by-name lookups of lexicals. That's essentially what you'd get in Perl5 if you had sub myEval { eval $_[0] }, and only ever called myEval, and not eval directly.) It also makes up-call lexical peeking and modification impossible. This is something Larry's specified Perl 6 code will be able to do. That is, any routine should be able to inspect the environment of its caller, and modify that environment, regardless of where the caller came from. If I'm interpreting that correctly, then it may just have the consequence that the Perl6 compiler can perform fewer optimizations than others. (And if the body of a sub can reference and modify lexicals in its caller, then it sounds like they're not really *lexicals*, which is confusing) Hmm, that also precludes part of what tail-call optimization usually gives you, since you can't reuse the current stack frame, if the called sub is supposed to be able to look back at the caller's state, so you lose the potential for unbounded recursion. If that's all correct, Perl6 is giving up a lot for that peeking feature. (And I'm less worried about the memory than I am about all of the pushing and popping and by-name stores and lookups, which could optimized away to just register usage.) There shouldn't be much, if any, pushing and popping for stuff like this, and access to lexical pads should be an O(1) operation. Remember, you know at compile time what lexical variables are in scope which means if you have a data structure which can be accessed by name and index (like, say, the OrderedHash PMC clas...) most, if not all, of the access
Re: Why lexical pads
On Sep 25, 2004, at 11:15 AM, Dan Sugalski wrote: At 2:10 PM -0400 9/25/04, Chip Salzenberg wrote: According to Dan Sugalski: Leaf subs and methods can know [their call paths], if we stipulate that vtable methods are on their own, which is OK with me. So, given this sub and tied $*var: sub getvar { my $i = rand; $*var } the FETCH method implementing $*var might not be able to see $i? Which implies that there may be no pad and $i could be in a register? Yeah, I think that's OK. I'm certainly OK with it, though there is an appeal to introspection. (Since you might want to fiddle with things in a debugger) But for a debugging, you'd want to be able to compile with optimizations disabled, so there's no real problem there. (And also, a clever-enough debugger might be able to let you do by-name manipulations of things stored in registers--you just need to preserve enough information at compile-time, in a form the debugger can use, like a separate symbols file). JEff
Re: towards a new call scheme
On Sep 24, 2004, at 1:13 AM, Leopold Toetsch wrote: Piers Cawley [EMAIL PROTECTED] wrote: I could be wrong here, but it seems to me that having a special 'tailinvoke' operator which simply reuses the current return continuation instead of creating a new one would make for rather faster tail calls than fetching the current continuation out of the interpreter structure, then invoking the target sub with with this new continuation (ISTM that doing it this way means you're going to end up doing a chunk of the work of creating a new return continuation anyway, which rather defeats the purpose.) Yep, that's true. We already have (unimplemented) Ctailcallmethod opcodes. A plain Ctailcall for functions is still missing. Anyway, if the current continuation has its location in the interpreter context, its simple to implement these opcodes. You mentioned: Well, basically, if an interpreter template is used (hanging off the subroutine), the previous interpreter template is the return continuation. So this occurred to me: Not sure if it would be useful, but since you'd be able to trace back up the call stack, a tailcall for functions could take a numeric parameter, to let you invoke a continuation from further back in the stack, without having had to pass it in as an explicit parameter. So for instance: return_cc == alias for call_cc_indexed 0 tailcall == alias for call_cc_indexed 1 call_cc_indexed 2 == invoke the continuation from 2 frames back, as though it had been passed down It's just a generalization--it lets you do the equivalent of grabbing the current continuation, and passing that to a function which passes it to a function which passes it to a function...which invokes it, without having to ever create a real PMC. But I'm not sure if that's a useful design pattern. JEff
Re: Namespaces, part 1
On Sep 23, 2004, at 9:53 AM, Dan Sugalski wrote: At 12:06 AM -0700 9/23/04, Jeff Clites wrote: On Sep 22, 2004, at 8:13 PM, Dan Sugalski wrote: At 7:32 PM -0700 9/22/04, Jeff Clites wrote: *) If a language wants different types of variables to have the same name, it has to mangle the names. (So you can't have an Array, a String, and an Integer all named Foo) The only language I know of that does this is Perl, and we just need to include the sigil as part of the name and we're fine there. This seems antithetical to cross-language functionality. Why? Not to be snarky here, I'm curious. Just that if I set a global $foo = 5 in Perl, I'd want to be able to change it from Python as, foo = 5. The problem there is deciding whether foo is $foo, @foo, or %foo, all of which may exist simultaneously. Yep, exactly my point/worry. If they aren't named $foo, @foo, and %foo, but instead are all foo, then we're back to globs and globs are a place we don't want to be at. Nah, it's no different from what we're already proposing with variable/sub/namespace partitioning--typeglobs are just an implementation choice. In particular, there are 2 obvious ways we could implement what we've got so far: a given namespace object manages 3 hashes--one to lookup variables, one for subs, and one for other namespaces. Or, a namespace could manage a single hash, in which you lookup a little structure which has one slot to hold a variable, one slot to hold a sub, and one to hold a namespace. The latter approach is basically a typeglob. You wouldn't necessarily expose that, but as an implementation detail it has some advantages, if there'd ever be a situation where you wanted to (say) look for foo by retrieving the variable if there is one, else a sub if there isn't a variable, else a namespace (advantage because there'd only be one hash lookup). Whether you've got 3 conceptual sections or 5, you'd have the same sorts of implementation choices. In particular, a nice approach would be to allow a given namespace object to decide on its semantics based on the context (language probably) in which it was created--so a namespace created in Perl code might have 3 or 5 sections, and one created in another language might have only 1. If the namespace lookup API (ie, how you do a lookup once you've got hold of a particular namespace object) had a signature such as, lookup(namespace object, name of thing, type indicator), then the namespace could decide what it wants to do if it doesn't have a section for that type of thing--it could return null, fall back to a different type, signal an error, whatever. Also, don't forget Perl5 (for Ponie)--it's never been characterized as having the sigil as part of the variable name. (At least, judging by the Camel.) (Not to mention the cross-language issues you really run into, for languages that don't do a simple scalar/hash/array split, but use different basic classifications, or none at all) So even if we did force perl to do name mangling, we get other problems in return. From Python, I can't set it using $foo = 5, since that isn't syntactically valid in Python, and it's no fun at all to have to do something introspective like, 'setValueOfGlobal($foo, 5)'. Yep. But, then, from Python you won't be able to set $nOl either, so I'm not sure it's that big a deal. Yeah, but as I said before, I'm less worried that there will be _some_ such cases, and more bothered that even if you try to coordinate, you can't create in Perl a variable which is legal in Python, or create one in Python which is legal in Perl. It's bad if our approach leaves zero overlap. (And if we do have zero natural overlap between Perl6 and Python, I'd probably argue that we should partition Python and Ruby apart, so that introspective lookups would have to specify a language. But that's all just awkward.) ('specially since I'm not sure my mail client will let me either...) Ha, yep, I assume you meant: $. (Your client specified the wrong encoding [ISO-8859-1 instead of Shift_JIS] in the MIME headers--but I know mine won't. :) Besides, Python's OO and we all know global variables are evil things one doesn't use in OO code, right? :) True, they're evil-ish, OO or not, but more benign in short scripts, which P/P/R tend to favor. (And of course globals inside namespaces are still globals, but better behaved since they have fancier::looking::names.) (And one legitimate use of inside-a-package-but-still-global variables is for constants relevant to the package. Java does that a lot. As a made-up example, stuff like java.lang.math.PI) For me, I want to be able to load some compiled bytecode module, and not care what language it was written in. Alas, there will always be problems. We can make interoperability *possible*, but that doesn't mean we can make it *easy*. Yes, but we should *try* to make it easy and elegant, and not resign ourselves too easily to giving up on it. (Besides
Re: Why lexical pads
On Sep 24, 2004, at 8:07 AM, Aaron Sherman wrote: On Fri, 2004-09-24 at 10:03, KJ wrote: So, my question is, why would one need lexical pads anyway (why are they there)? They are there so that variables can be found by name in a lexically scoped way. One example, in Perl 5, of this need is: my $foo = 1; return sub { $foo ++ }; Here, you keep this pad around for use by the anon sub (and anyone else who still has access to that lexical scope) to find and modify the same $foo every time. In this case it doesn't look like a by-name lookup, and once optimized, it probably won't be, but remember that you are allowed to say: perl -le 'sub x {my $foo = 1; return sub { ${foo}++ } }$x=x();print $x-(), $x-(), $x-()' Which prints 012 because of the ability to find foo by name. Ha, you're example is actually wrong (but tricked me for a second). Here's a simpler case to demonstrate that you can't look up lexicals by name (in Perl5): % perl -le '$x = 2; print ${x}' 2 % perl -le 'my $x = 2; print ${x}' (printed nothing) The first case prints 2 because $x is a global there; in the second case, it's a lexcial, and ${x} is looking for a global. In your example, ${foo} is actually addressing the global $foo, not your lexical. To demonstrate, both this: perl -le 'sub x {my $foo = 7; return sub { ${foo}++ } }$x=x();print $x-(), $x-(), $x-()' and this: perl -le 'sub x { return sub { ${foo}++ } }$x=x();print $x-(), $x-(), $x-()' also print 012. (Your example should have printed 123, if the lexical $foo had been what the closure was incrementing.) Someone else suggested that you need this for string eval, but you don't really. You need it for by-name lookups, which string evals just happen to also need. If you can't do by-name lookups, then string eval doesn't need pads (and thus won't be able to access locals). String eval is special because without it, you can tell at compile time all of the places where a lexical is used (ie, you can trace all of the places it's used back to which declaration matches them), and obviate the need for a by-name lookup (since Perl5 doesn't allow explicit by-name lookups of lexicals). For string evals to work in the lexical scope in which they occur, you do have to have by-name lookups behind-the-scenes. I believe that's all correct. And where this could be a savings is that, in any lexical scope in which there is no eval visible (looking down the tree of nested lexical scopes), then you don't need to save the name-to-variable mapping in nested pads. Add a call to eval, and you need to save a lot more stuff. JEff
Re: Why lexical pads
On Sep 24, 2004, at 6:51 PM, Aaron Sherman wrote: On Fri, 2004-09-24 at 12:36, Jeff Clites wrote: Ha, you're example is actually wrong (but tricked me for a second). Here's a simpler case to demonstrate that you can't look up lexicals by name (in Perl5): You are, of course, correct. If I'd been ignorant of that in the first place, this would be much less embarassing ;-) No need to be embarrassed--it's easy to trick yourself. (I had forgotten myself, until I recently tried it while thinking whether lexical pads really needed a by-name API.) However, the point is still sound, and that WILL work in P6, as I understand it. Hmm, that's too bad--it could be quite an opportunity for optimization, if you could use-and-discard lexical information at compile-time, when you know there's no eval around to need it. JEff
Re: Why lexical pads
On Sep 24, 2004, at 7:32 PM, Dan Sugalski wrote: At 7:28 PM -0700 9/24/04, Jeff Clites wrote: On Sep 24, 2004, at 6:51 PM, Aaron Sherman wrote: However, the point is still sound, and that WILL work in P6, as I understand it. Hmm, that's too bad--it could be quite an opportunity for optimization, if you could use-and-discard lexical information at compile-time, when you know there's no eval around to need it. Even if not it's going in anyway. The introspection abilities are more than worth the extra memory that the name hashes use. It's a compiler issue. You're right that no matter what, you need lexical pads as a feature in Parrot for...those cases where you need lexical pads. But it's nice to have stuff that a compiler can optimize away in a standard run, and maybe leave in place when running/compiling a debug version--but that's a matter of the semantics of the language. (And I'm less worried about the memory than I am about all of the pushing and popping and by-name stores and lookups, which could optimized away to just register usage.) JEff
Re: Namespaces, part 1
On Sep 22, 2004, at 8:13 PM, Dan Sugalski wrote: At 7:32 PM -0700 9/22/04, Jeff Clites wrote: One problem: Some languages (Scheme, for example, and arguably C) have a unified namespace for subs and variables. What to do there? The easiest thing would be to allow the languages to store into multiple sections at once, or leave it to them to do the right thing as they need to. The risk is that if they store into both, then setting a variable in Scheme code might blow away a sub in perl, and other than that it seems you'll have big-time ambiguity. But I don't know of an obvious solution. Either way works OK for me, since neither Scheme nor C are really our target languages. Well, my point here is that namespace semantics varies a lot between languages, and quite reasonably people seem to want to target a lot of different languages to Parrot, besides just Perl/Python/Ruby. And I'm not sure where Python and Ruby stand, in terms of sub/variable name overlap. *) Sub and method names *will* collide, so be careful there. I'm not sure that they will. I'd really, *really* rather they did. Not because I'm up for collisions (I'm not) but because if you *do* leave methods in namespaces it allows you to do all sorts of very interesting things with lexically overlaid namespaces. To me, it doesn't make sense to muddle basic semantics to allow tricks. In particular, for method invocations on objects, it makes sense that the lookup is very directly object to class, lookup method in class, rather than object to class, get class name, lookup method in global namespace. There's an efficiency issue, and the latter precludes truly anonymous classes (since they should have no name with which to do a lookup). I tend to think of this as classes live in namespaces, methods live in classes--and of course anonymous classes aren't stored in a namespace, but are only usable via a reference to the class (like lexical v. global variables). [Side note: Oh, and of course that reminds me--class names might need to be their own section of the namespace, or maybe they do just live in the, uh, namespace-namespace. My worry here is that the lookup API should really know the type of thing it's looking for--that is, have ops like lookupsub v. lookupvariable v. lookupnamespace--and it could be awkward if there is one that might return a namespace or might return a class, because those two things would have at least somewhat different API.] And for existing libraries in languages which allow subs and methods with the same name, we can't retroactively change it--it's too late to be careful. But again, I'm not sure what it means to have a sub in a class, so they might just automatically stay out of each other's way. (For example, you could say Java had only methods, no functions, and Perl5 has only functions, no real methods.) *) If a language wants different types of variables to have the same name, it has to mangle the names. (So you can't have an Array, a String, and an Integer all named Foo) The only language I know of that does this is Perl, and we just need to include the sigil as part of the name and we're fine there. This seems antithetical to cross-language functionality. Why? Not to be snarky here, I'm curious. Just that if I set a global $foo = 5 in Perl, I'd want to be able to change it from Python as, foo = 5. From Python, I can't set it using $foo = 5, since that isn't syntactically valid in Python, and it's no fun at all to have to do something introspective like, 'setValueOfGlobal($foo, 5)'. For me, I want to be able to load some compiled bytecode module, and not care what language it was written in. That's the goal of a generic object runtime. If the module's docs say, set the global 'foo' to control..., then I'd want to do that in a way that is natural for whatever language I'm using ($foo in Perl, foo in Python...). Strictly speaking, the names can be anything, which is fine. Nulls, control characters, punctuation, grammatically incorrect Bengali insults... we don't put any limits on 'em. And the only place that it makes any difference at all is when you're either doing the equivalent of C's extern or explicitly walking the namespace. In the latter case you ought to be ready for anything (since with Unicode I fully expect to see the full range of characters in use) and in the former case, well... you ought to be ready for anything since Unicode's going to put the full range of characters into use. :) Sure, that fine--there will certainly be cases where one language allows variable names that another can't access naturally, or maybe even at all. But in the Perl case, we'd end up having _all_ variable names turning out illegal in Python (and I suppose Ruby), which seems bad. (Though it's somewhat academic. Besides leading to globs, and we *so* don't want to go there, Larry's declared that perl variable names include the sigil) Of course, I'm not talking about Perl
Re: S5 updated
On Sep 22, 2004, at 5:06 PM, Edward Peschko wrote: How do you do that? Generation and matching are two different things algorithmically. yes, but they are intimately linked. just like the transformation of a string into a number, and from a number to a string. Two algorithmically different things as well, but they'd damn-well better be exact inverses of the other. But they're not: 3 foo -- 3 -- 3 My point is that if inputting strings into grammars is low level enough to be an op, why isn't generating strings *from* grammars? Maybe, because it's a less common thing to want to do? (Which is a bit ironic, since technically grammars are typically characterized as sets of rules for how to generate all the acceptable strings of the language they define, and parsing is sort of running that in reverse.) But you seemed to be saying (to which Luke replied the How do you do that? above) that they should somehow share an implementation, so that they can't accidentally diverge. But algorithmically it seems they can't share an implementation, so making them both fundamental ops doesn't achieve the goal of ensuring parity. JEff
Re: S5 updated
On Sep 23, 2004, at 5:27 PM, Edward Peschko wrote: On Thu, Sep 23, 2004 at 08:15:08AM -0700, Jeff Clites wrote: just like the transformation of a string into a number, and from a number to a string. Two algorithmically different things as well, but they'd damn-well better be exact inverses of the other. But they're not: " 3 foo" -- 3 -- "3" I'd say that that's a caveat of implementation, sort of a side effect of handling an error condition. Nope, I'd call it fundamental semantics--it allows common idioms such as "0 but true" in Perl5, for example. It's just an explicit part of the rule for how Perl (and C's strtol/atoi functions) assign numerical values to strings. But you might like this example better, which I assume will work in Perl6: "3" -- 3 -- "3" (In case your email viewer doesn't render that, the first string contains the "fullwidth digit three", a distinct, wider version of a 3, used in some Asian languages.) By your criteria there are very few inverses - you could say that multiplication isn't an inverse of division because of zero, for example. I'm reacting here to your saying, "exact inverses". But for this example, it's not my criteria--to a mathematician, multiplication over the real numbers (or over integers) is in fact not invertible. If you add the further caveat that everything in the string to be converted has to be an integer, then they *are* direct inverses. Yes, the operation is invertible, if restricted to a domain over which it's invertible My point is that if inputting strings into grammars is low level enough to be an op, why isn't generating strings *from* grammars? Maybe, because it's a less common thing to want to do? Well, there re two responses to the "that's not a common thing to want to do": 1) its not a common thing to want to do because its not a useful thing to do. 2) its not a common thing to want to do because its too damn difficult to do. I'd say that #2 is what holds. *Everybody* has difficulties with regular expressions - about a quarter of my job is simply looking at other people's regex used in data transformations and deciding what small bug is causing them to fail given a certain input. Yeah, but when a regex isn't acting how I expected it to, I know that because I've already got in-hand an example of a string it matches which I thought it wouldn't, or one it fails to match which I thought it should. What I want to know is *why*--what part of the regex do I need to change. Generating strings which would have matched, wouldn't seem to help much. And you might be underestimating how many strings can be generated from even a simple regex, and how uninformative they could be. For example, the Perl5 regex /[a-z]{10}/ will match 141167095653376 different strings, and it would likely be a very long time before I'd find out if this would match any strings starting with "x". I'd probably be left with the impression that it would only match strings starting with "a". Running a regular expression in reverse has IMO the best potential for making regexes transparent - you graphically see how they work and what they match. How graphically? Why shouldn't that be reflected in the language itself? Maybe because if it's likely to be used mostly for debugging, and can be implemented in a library, then it doesn't need to be implemented as an operator, and contribute to the general learning curve of the language's syntax. JEff
Re: Namespaces, part 1
On Sep 22, 2004, at 10:58 AM, Dan Sugalski wrote: *) There are three things that can be in a namespace: Another namespace, a method or sub, and a variable. *) The names of namespaces, methods subs, and variables do *not* collide. You may have a namespace Foo, a sub Foo, and a variable Foo at the same level of a namespace. The easiest way to say these two point may be just to say that each namespace has three sections, and a given lookup is always a lookup in a specific section. One problem: Some languages (Scheme, for example, and arguably C) have a unified namespace for subs and variables. What to do there? *) Sub and method names *will* collide, so be careful there. I'm not sure that they will. Classes are namespace-like, but aren't precisely namespaces. Since methods live in classes (and subs don't), this may just work itself out. (To give a concrete example, for a class Foo with an instance method bar, I never really look up any Foo::bar--instead, I invoke bar in an instance of Foo, which doesn't need to go looking for the relevant namespace--the namespace is directly attached to the Foo class.) It's a different story for class methods, which really are basically just subs (not really methods). But we still may have a problem if there are any languages which allow class and instance methods of the same name, in the same class. *) If a language wants different types of variables to have the same name, it has to mangle the names. (So you can't have an Array, a String, and an Integer all named Foo) The only language I know of that does this is Perl, and we just need to include the sigil as part of the name and we're fine there. This seems antithetical to cross-language functionality. Treating the perl sigil as part of the variable name is cute and all, but if we take that to heart at the Parrot level, then we really cut off cross-language access in any natural way. So I think Perl's $foo needs to look like foo to Python, or else everything will be nasty looking. Which will loop back to how handle $foo and @foo at the same time. You're right that functions' lacking sigils makes this less of a problem, but that seems like a fragile truce. *) Namespaces, by default, support notifications. What does that mean? Notification of what? JEff
Re: [perl #31682] [BUG] Dynamic PMCS [Tcl]
On Sep 22, 2004, at 5:30 PM, Will Coleda (via RT) wrote: # New Ticket Created by Will Coleda # Please include the string: [perl #31682] # in the subject line of all future correspondence about this issue. # URL: http://rt.perl.org:80/rt3/Ticket/Display.html?id=31682 ... $ make realclean $ export LD_LIBRARY_PATH=.:blib/lib $ perl Configure.pl $ make $ make shared $ cd dynclasses; make On OSX 10.3.5, I end up with: ld: /Users/coke/research/parrot/blib/lib/libparrot.dylib is input for the dynamic link editor, is not relocatable by the static link editor again ... As for the next error... huh? Yeah, this is going to be a fun config dance. On Mac OS X, there are two distinct types of dynamic libraries--those which are meant to be linked against and automatically loaded at runtime (dylibs), and those which are meant to be manually loaded (bundles). libparrot needs to be built as a dylib things in dynclasses (or other plug-in-like stuff) need to be built as bundles The Makefiles don't have this conceptual split currently, and so things (including libparrot) are being built as bundles, but getting the dylib extension (running the 'file' command on the library reveals the truth). A bit Frankenstein. I worked out in the past the right configs to build parrot shared on Mac OS X, but I don't think I'd yet wired the Makefiles (and config process) to do that, and still have the other stuff build as bundles. I'll dig this up and see what I did. JEff
incremental collector and finalization
Hi Leo: I was reading over you incremental GC posts from about a month ago, and read the referenced paper--quite nice work you've done in implementing the ideas there. I have one question: What about finalizers? I may have just missed it, but it would seem that calling finalizers would require another sweep over the items newly resident on the free list (the left-over items in the from-space), which would (unfortunately) take time proportional to the number of freed object. BUT, a nifty thing would be to actually delay finalization until an object is about to be re-used off of the free list. That is, treat ecru items as only-maybe free, and as they are pulled off of the free list for re-use, check to see if a finalizer needed to be called and if so, call it and move on to the next item on the free list (as the first one may now be referenced, and should be marked grey). This would allow finalization to be treated incrementally, at the cost of it happening later than it would otherwise (which I think is fine). But maybe this is what you had in mind already. This doesn't give ordered finalization, but that may be okay. JEff
Re: No Autoconf, dammit!
On Sep 18, 2004, at 2:09 AM, [EMAIL PROTECTED] wrote: * Nicholas Clark [EMAIL PROTECTED] [2004-09-08 17:37:52 +0100]: The probing is going to *have* to get written in something that compiles down to parrot bytecode to work on the autoconf-deprived systems, so with that as a given there's no need for autoconf ahead of that. How feasable would it be to write a sh - parrot bytecode compiler? :) Ha, I'm sure it could probably be done, but of course most of what the shell does it invoke other programs, so in the common case it still wouldn't give you portability to non-Unix-like platforms. JEff
Re: Namespaces
On Sep 13, 2004, at 1:07 AM, Luke Palmer wrote: Jeff Clites writes: On Sep 12, 2004, at 8:43 PM, Luke Palmer wrote: Jeff Clites writes: On Sep 7, 2004, at 6:26 AM, Dan Sugalski wrote: *) Namespaces are hierarchical So we can have [foo; bar; baz] for a namespace. Woo hoo and all that. It'd map to the equivalent perl namespace of foo::bar::baz. How does this hierarchical nature manifest? I ask because I don't know of any languages which actually have nested namespaces, Other than, um, well, Perl. As an implementation detail yes, but I can't think of any Perl code (except for explicit introspection) which reveals this. Does: print ${$Foo::{Bar::}{baz}} Count as explicit introspection? Well, that's another one of those cases where I don't see that it gives the programmer power that you don't already get by being able to do: print ${${Foo::Bar::}{baz}}. Now it's true that if you think it's really important to tie namespaces such that you can take over the name resolution for [Foo], then you'd want the former. But I don't see that as really terribly useful A debugger. I'm not sure exactly what you mean--if you're referring to being able to have a debugger intercept name resolution for only certain namespaces, then all you need to get this is the ability to intercept all name resolution--and just fall back to the default for namespaces you don't care about. If you're talking about the ability to have the debugger dump out all namespaces with Foo at the top, or do a hierarchical printout, then a debugger can certainly do this without having truly nested namespaces. That's just output formatting/logic, really. And of course, that's introspection. Sure. That's not the idea behind the heirarchical namespaces either. A lot of power comes out of being able to treat namespaces as a tree-like data structure (I can actually attest to this: see Class::Closure), as was pointed out before, being able to treat namespaces as a filesystem. For me, the filesystem metaphor plays out a little differently. I think of an individual namespace as being like a DB file. It's convenient to organize files via a hierarchical filesystem--but files are not themselves nested (you don't have files inside of other files), and the API of files (open/close/read/write/seek/truncate) has nothing to do with anything hierarchical. So for me the analogy means that it's convenient to have hierarchical organization so that humans can keep CPAN organized, but that doesn't have runtime consequences. Also, Perl 6's classes will behave more heirarchically: module Foo; class Bar {...} my $x = Bar.new; # Actually works, while Bar is Foo::Bar. But Java has something similar (inner classes), and does it without nested namespaces. In fact it (I believe) unwinds stuff like this completely at compile-time, so that your last line compiles down to the same thing as my $x = Foo::Bar.new;. What might do it for me, would be if stuff like this worked: module Foo; $x = 7; class Bar {...} # now, in a different lexical scope... sub Foo::Bar::blah { ++$x; } # Resolves to $Foo::x, because Bar is inside Foo # now, elsewhere at runtime ...some syntax which moves Bar from module Foo into module Zoo at runtime then the above sub is Zoo::Bar:blah, and $x resolves to $Zoo::x, if that exists Not that I think any of that would necessarily be a good thing to have, but it would be the sort of thing which would naturally be implemented via nested namespaces. (And, if any significant language allows this sort of run-time namespace hierarchy rearrangement, then I'd see that as a good reason to nest--so that we don't cut out a whole language from Parrot-targetablility.) But, that said, I think I'm pretty much the only person who thinks that namespaces shouldn't nest conceptually. And I'm really more worried about the other issue I brought up, about how do deal with interoperability between different languages which have different ideas about namespace segmenting between different types of entities. That's a bigger problem. JEff
Re: Namespaces
On Sep 12, 2004, at 8:43 PM, Luke Palmer wrote: Jeff Clites writes: On Sep 7, 2004, at 6:26 AM, Dan Sugalski wrote: *) Namespaces are hierarchical So we can have [foo; bar; baz] for a namespace. Woo hoo and all that. It'd map to the equivalent perl namespace of foo::bar::baz. How does this hierarchical nature manifest? I ask because I don't know of any languages which actually have nested namespaces, Other than, um, well, Perl. As an implementation detail yes, but I can't think of any Perl code (except for explicit introspection) which reveals this. That is, seemingly nothing about Foo::Bar comes out of Foo, nor is it necessarily related to Foo::Baz. so I'm not sure what this is meant to imply. In particular, if this implies cascading lookup (meaning that a symbol looked up in [foo; bar; baz] would fall back to looking in [foo; bar] if it didn't find it), than that's not how Perl5 packages work. No, the lookup is not cascading downwards. It is cascading upwards, however, so that in: [ foo; bar; baz ] The [ foo; bar ] part can be implemented differently. This means that Python's namespaces and Perl's namespaces can have different semantics. Picture a PythonStash and PerlStash pmc. Sure, if they're nested then the baz namespace is looked up inside of bar which is looked up inside of foo--I get that. But what I'm wondering is, let's say you have $Foo::Bar::someVariable (in Perl5 syntax). That means look up $someVariable, inside of the Foo::Bar namespace. Now, let's contrast two ways this namespace name might compile down: [Foo; Bar] v. [Foo::Bar] Now, what would behave differently in those two cases, in terms of the behavior of $Foo::Bar::someVariable? I can't think of anything. Now it's true that if you think it's really important to tie namespaces such that you can take over the name resolution for [Foo], then you'd want the former. But I don't see that as really terribly useful--in the examples I've heard of (e.g., turning the namespace lookup into an Oracle fetch) have seemed to not provide much that a tied hash doesn't. And there are currently a ton of, for instance, Text::Blah modules on CPAN--but that doesn't actually mean that they have anything to do with a Text namespace--that's not the intention, and most of these modules have nothing to do with each other. That naming is to keep them conceptually organized, not functionally. I can certainly see having different namespace semantics for different languages, but that doesn't imply arbitrary nesting. That might imply separate top-level namespaces, each with its own behavior (so overall, two levels of namespace), or you could just have one-level namespaces, but each with its individualized semantics, or all namespace might be represented by the same type of data structure, which is just traversed differently from different languages. (And I'm not sure that the semantics do differ between languages--given a namespace and a name, there's seemingly only one thing to do.) And again, I don't so much object to the idea of nested namespaces--I just feel that they'll slow down symbol lookups, without giving us much in return. I'm afraid we're adding complexity we don't need. JEff