Re: [drlvm] vote on class unloading design (was Class unloading support - tested one approach)
The http://wiki.apache.org/harmony/ClassUnloading wiki page is Immutable. How can I contribute to it? [Among other things, I'd like to add that the design can also potentially apply to SableVM and , and make other suggestions / changes.] Do I have to submit JIRA bugs? If yes, how do I make the patches? (svn diff?) Thanks for helping me! Etienne Aleksey Ignatenko wrote: Actually there is one additional 4-th approach: *Mark and scan based approach *wich I described in the first letter. Java heap trace is performed by VM Core and GC is not affected at all. So the list is: 1. vtable objects( Aleksey ) 2. per heap space/generation boolean marker on the classloader instance( Etienne ) 3. class reachability marker byte in vtable (Robin ) 4. Mark and scan based approach. I agree that we need to structure all appraches somehow, like description for every approach in wiki. Aleksey. On 11/10/06, Rana Dasgupta [EMAIL PROTECTED] wrote: Identifying an end to end alogirithm is of course necessary, and the discussions on the other thread are great. But what were we voting on here? My understanding was that we were voting on the approach of using: 1. vtable objects( Aleksey ) 2. per heap space/generation boolean marker on the classloader instance( Etienne ) 3. class reachability marker byte in vtable (Robin ) and not on three end-to-end algorithms since Robin's description was not one. I am OK if we now want to cancel the vote, but next time we vote on a technical issue it may be a good idea for the caller to summarize the contending solutions on the thread and then call for the vote. Thanks, Rana On 11/9/06, Weldon Washburn [EMAIL PROTECTED] wrote: It looks like I called for a vote too soon. The continuing discussion on class unloading design is uncovering many important issues. This is excellent as it is much better to deal with design issues at this stage rather than during implementation. I propose the following: 1) Cancel the current vote on design. 2) Someone put together a complete class unloading design based on Etienne/Robin's approach. Include pseudo code and post to harmony-dev. 3) We call for a new vote once the comments on the documented design indicate it is ready. On 11/8/06, Robin Garner [EMAIL PROTECTED] wrote: Pavel Pervov wrote: Really BIG -1 from me. As Aleksey (Ignatenko) described in original thread, j/l/Class'es and j/l/ClassLoader's are always available in rootset, so even if no objects of a class exist, this class will be reachable. Actually, some sort of class unloading prototype exists in DRLVM code, which implements the scheme, which is very close to what is currently voted. It was integrated with GC v4 and is not supported by other GCs. This prototype traces up to class loader. Robin's approach is way faster then prorotype is. Unfortunately, that approach requires up to 3 GC cycles to complete in DRLVM. In a full-heap STW collector, my proposal would require 1 GC to collect unused classloaders. In a generational STW collector, 1 full-heap GC, and would depend on the particular invariants enforced by an incremental/concurrent collector, but would be 1 complete cycle of any of the standard algorithms (I guess up to 3 GCs if the sweeps happened at the wrong places). BTW, voted approach does not describe proof-of-full-collection algorithm (at least I didn't find one). The only one I think of is full-heap-collection, which _requires_ STW. My approach simply requires the underlying collector to have a notion that periodically it can say that 'every reachable object allocated since time 't' is now marked reachable. If the class-unloader can ensure that one full epoch of this invariant has passed, then it can safely perform unloading. Although automatic anloading brings some additional requirements for GC (weak roots (references) support and pinned allocation), it is proven to work (patch available) and, also, is the most natural algorithm for DRLVM. What is the run-time cost of it ? And where is it described ? I was only aware of Etienne's proposal as a full class-unloading scheme. With the best regards, -- Robin Garner Dept. of Computer Science Australian National University -- Weldon Washburn Intel Enterprise Solutions Software Division -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [drlvm] vote on class unloading design (was Class unloading support - tested one approach)
Salikh Zakirov wrote: Etienne Gagnon wrote: The http://wiki.apache.org/harmony/ClassUnloading wiki page is Immutable. How can I contribute to it? It is immutable for anonymous guests, you need to register and login somewhere. All pages are editable by all registered users. Silly me! I must have missed some FM (as in RTFM) where this is written... Thanks, Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [drlvm] Class unloading support - tested one approach
Salikh Zakirov wrote: Technically, it should not be too difficult to add an additional field to the VTable structure, and require GC to write 1 there during object scanning. However, if the VTable mark is located in the same cache line as gcmap, it may severely hit parallel GC performance on a multiprocessor due to false sharing, as writing VTable mark will invalidate the gcmap pointers loaded to caches of other processors. objectVTable gcmap +++---++--+ | VT ptr |---| gcmap ptr |---| offset of ref #1 | | ... ||...|| offset of ref #2 | +++---+| ...| |0 | +--+ If you go that far for every scanned object (!), then you could simply place the class unloading bit in the gc map, at index -1) to minimize disruption of current code... objectVTable gcmap +--+ +++---+| cl.un. mark bit | | VT ptr |---| gcmap ptr |---| offset of ref #1 | | ... ||...|| offset of ref #2 | +++---+| ...| |0 | +--+ This gets rid of the cache line hazard... Why don't you also investigate using SableVM's bidirectional object layout? Dayong Gu (a Ph.D. student tha I co-supervise) found that it was quite simple to implement in JikesRVM. I don't see why it should be harder to implement in drlvm... It would save you this nasty indirection for scanning objects! [See my Ph.D. thesis for an explanation of the layout. You can get in touch with Dayong through the coordinates at http://www.sable.mcgill.ca/people/ ]. Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [drlvm] Class unloading support - tested one approach
OK. My latest proposal (a few messages ago) was assuming that the nursery was empty when the end of epoch collection is launched. If it is not, you can do 2 things: a) do a minor collection to empty it, or b) i - use a finalization-like list of references to class loader objects ii - launch gc, which might mark a previously unmarked vtable iii- do a finalization-like rescuing for resuscitated class loaders b) should really have a minimal performance impact. As for its apparent complexity, I would say that this is a non-issue; similar code must already exist in drlvm for implementing finalization. Just for clarification: b) implies a combined nursery + mature space collection. Actually, for the mature space part, you could get away with a smaller collection if you premature all class loaders and classes to a specific mature space area; the you only need to collect that space (in addition to the nursery). Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [drlvm] Class unloading support - tested one approach
Salikh Zakirov wrote: I have another concern though: just before starting final unloading collection, we scan vtable marks and identify the candidates for unloading. During the final unloading collection, the candidate classloader roots are reported as week. At the end of the trace, we need to rescan vtable marks and revive the classloader which were found in possession of live objects. This techniques is exactly the same as the one used for object finalization. However, in contrast with finalization, we will need to repeat reviving classloaders which have non-0 vtable marks until the process converges, and no new classloaders are revived. (* in finalization, both dead and live objects in finalization queue are revived, and thus the revival converges in just 1 step *). Revival is only needed if you use the finalization-like approach. If you only do class-unloading GC when the nursery is empty, then no revival is needed. In this case, after GC you only need to revert weak references to hard ones. Nulled weak references relate to dead class loaders for which you can definitely free the native resources. Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [drlvm] Class unloading support - tested one approach
Etienne Gagnon wrote: Salikh Zakirov wrote: I have another concern though: just before starting final unloading collection, we scan vtable marks and identify the candidates for unloading. During the final unloading collection, the candidate classloader roots are reported as week. At the end of the trace, we need to rescan vtable marks and revive the classloader which were found in possession of live objects. This techniques is exactly the same as the one used for object finalization. However, in contrast with finalization, we will need to repeat reviving classloaders which have non-0 vtable marks until the process converges, and no new classloaders are revived. (* in finalization, both dead and live objects in finalization queue are revived, and thus the revival converges in just 1 step *). In case you chose the finalization-like + revival way, then I don't see any significant performance hit of multiple-step convergence! For one thing, you'll probably agree with me that it is quite unlikely to take more than 1 step to converge in most cases, and the additional work in the other cases is still quite insignificant relative to the remaining collection work! Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [drlvm] Class unloading support - tested one approach
Salikh Zakirov wrote: Ah, I think I got it. Yep. 3) trace the heap 4) scan vtable marks and revive marked class unloaders, by adding the strong root from the previously collected unload list. Remove the revived classloaders from unload list. 5) repeat steps (3) and (4) until there is no classloaders to revive As long as it is understood that the repeated (3) is not a full trace. It's only a trace starting from the revived roots. [This is important in evaluating the total work done]. The voting definitely was premature, as it turns out that even the design under discussion can be elaborated to multiple substantially different designs. Yes, you're right. Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [drlvm] Class unloading support - tested one approach
Ivan Volosyuk wrote: We will get rid of false sharing. That's true. But it still be quite expensive to write those '1' values, because of ping-ponging of the cache line between processors. I see only one solution to this: use separate mark bits in vtable per GC thread which should reside in different cache lines and different from that word containing gcmap pointer. The only thing that a GC thread does is write 1 in this slot; it never writes 0. So, it is not very important in what order (or even when) this word is finally commited to main memory. As long as there is some barrier before the end of epoch collection insuring that all processors cache write buffers are commited to memory before tracing vtables (or gc maps). You don't need memory coherency on write-without-read. :-) Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [drlvm] Class unloading support - tested one approach
Salikh Zakirov wrote: 7) let the GC finish collection and reclaim unreachable objects -- this reclaims java objects Just a bit of a warning... This should be integrated within the weak/soft/phantom + finalization framework. We definitely don't want the native resources of a class loader to be freed and later have finalization revive the class loader... :-) [Luckily, nothing special needs to be done for JNI code; CallTYPEStaticMethod does require a native reference to a class object. Yay! ] Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [drlvm] Class unloading support - tested one approach
Ivan Volosyuk wrote: We will get rid of false sharing. That's true. But it still be quite expensive to write those '1' values, because of ping-ponging of the cache line between processors. I see only one solution to this: use separate mark bits in vtable per GC thread which should reside in different cache lines and different from that word containing gcmap pointer. Thinking about it... Doesn't the object vtable suffer from the same problem, anyway? It's probably worse, as it will be quite unfeasible to try to locate them in the right cache lines! Yep, another point against object-vtables... Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [drlvm] vote on class unloading design (was Class unloading support - tested one approach)
Aleksey Ignatenko wrote: So the list is: [...] 2. per heap space/generation boolean marker on the classloader instance( Etienne ) 3. class reachability marker byte in vtable (Robin ) [...] Unless Robin disagrees(?), I would say that 2 and 3 have evolved into a single merged proposal that comes with various minor variations for different collection needs: simple stop the world (stw), generational stw, concurrent, etc. You could call it the epoch-based vtable marking approach. So, that bring us back to 3. Robin: Do you agree? :-) Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [drlvm] Class unloading support - tested one approach
[First, let me say that, as I am not contributing a class unloading *implementation* to drlvm, I will understand if the project was more inclined to chose an actually contributed piece of code over a design without contributed implementation. :-)] There was a -1 vote... Hmmm... As I voted +1, I will enter the arena... :-) Before getting farther in this class unloading discussion, I would like to get some clarifications about 3 things: 1- How does drlvm implement object finalization? Doesn't this require some object rescuing, much similarly to my proposal for correctly implementing class unloading (the class loader rescuing thing)? If it does, then can somebody explain to me what's wrong with my proposal of setting, in normal operation, a hard reference to class loader objects, and then temporarily using weak, rescuable reference to enable class loader collection? I don't see a performance hog there. Rescuing a few class loaders (if any) and their related classes once per epoch shouldn't cost much! I have yet to see a convincing explanation of how continuous collection of object-vtables would be more efficient... Really, even with Robin's proposal, this would work. If a class loader gets into an unloadable state, then most likely, the class loader and all classes it has loaded will have migrated to an old generation. So, as long as we set then end of a class unloading epoch at an old generation collection, then we can simply weaken the class loader reference during collection (only when the bit of all related vtables are unset), then apply the finalization-like rescue dance to class loaders. [*]This wouldn't affect any other operation, during other GC cycles, as Robin's unconditional bit/byte/word vtable write only serves to tell us whether a class loader has had living instances of its classes during the epoch. 2- I would like to read some full description of the object-vtable proposal. In particular, how does this proposal deal with the following: a) Preventing unloading of a class which has an active static method. b) Preventing unloading of a class which has an unsynchronized instance method active, which has overwritten local variable 0 (or for which the liveness analysis has detected the death of the reference to this). 3- Why would it be so hard to add an unconditional write operation during collection (e.g. during copying or marking of an object) in drlvm? A detailed technical explanation is welcome. :-) So far, this latest point (3-) seems the sole argument in favor of using the object-vtables approach. Wouldn't the right fix, if it's currently really impossible to implement an unconditional write, be to extend the drlvm GC interface? Isn't this a design deficiency in the GC interface? No other argument, so far, seems to be in favor of the object vtable approach, unless I missed some. As for Robin's attempt to deal with weak/hard reference to the class loader object using a Reference Queue, I am not yet convinced of the correctness of the approach... [off the top of my head: potential problem with static synchronized methods]. And, this would be probably more work intensive (so, less efficient) than my proposal in 1-[*] above. It might also be tricky to identify all possible situations such as Object.getClass() where some special code is needed to deal with a problem situation. I prefer clean, scope-limited code for dealing with class unloading. It's easier, that way, to activate or deactivate class loading [dynamically!]. In summary, I would like to be convinced of the completeness and of the correctness of all competing approaches. I personally am, so far, in favor of Robin's unconditional vtable bit/byte/word write idea, along with an adapted version of my proposal for dealing with class loader death (such as proposed in 1-[*] above). Also, if somebody was able to find a correctness deficiency in my proposal, then please let us know, so that we make sure this deficiency is eliminated from all competing proposals. By the way, what are the currently competing proposals? 1- object vtables 2- Robin/Gagnon proposal (still finishing up some details ;-) 3- Is there a 3rd? Which ones have existing implementations? How correct/complete are they? Do we have access to some human readable (i.e. non-code) full description of the algorithm? Etienne Robin wrote: On Thu, 2006-11-09 at 02:01 +0300, Ivan Volosyuk wrote: Robin, thank you for detailed description of the algorithm. IMHO, this was the most complicated place of the whole story: how to have a weak reference to classloader and still be able to get it alive again. This shouldn't be performance critical part and is quite doable. I absolutely agree with your estimations about tracing extra reference per object. The approach you propose is more efficient and quite elegant. -- Ivan Thanks :) On 11/8/06, Robin Garner [EMAIL PROTECTED] wrote: Robin Garner wrote: Aleksey
Re: [drlvm] Class unloading support - tested one approach
I was making it more complex than it needs... Here's an improvement... 1- During normal operation, the VM keeps hard references to all class loader instances. [This prevents any premature class loader death]. 2- At the start of an epoch (or just before), all vtable bits (or byte or word) are cleared. [From now on, I will use the bit terminology for simplicity. The bit may reside in an otherwise unused byte or even word, for efficiency purpose]. 3- The end of an epoch happens no sooner than when all generations / heap parts have been collected at least once since the epoch start. [One can cheat and visit objects of uncollected parts/generations to mark their vtables]. 4- An old generation collection is chosen as the end of an epoch. This is end of epoch collection. [As class loaders/classes are likely to have moved to older generations, there's no point trying to kill them in young collections]. 5- Just before starting the end of epoch collection, all the class-loader vtable lists are visited (and bits are cleared in prevision of the next epoch). All vm references to [candidate] class loaders with no surviving objects (nor active methods) (e.g. no vtable bit set) are made weak. 6- The end of epoch collection is launched. 7- There's actually no need for rescuing class loaders. The vm reference to any surviving [candidate] class loader is made hard again. Interesting fact: other candidate class loaders cannot have any instance (nor any active method) as GC doesn't create instances nor method calls. So, there's no need for a rescuing dance! The list of dying class loaders can be used for freeing related native resources. IMO: simple, clean, efficient... Etienne Etienne Gagnon wrote: If it does, then can somebody explain to me what's wrong with my proposal of setting, in normal operation, a hard reference to class loader objects, and then temporarily using weak, rescuable reference to enable class loader collection? I don't see a performance hog there. Rescuing a few class loaders (if any) and their related classes once per epoch shouldn't cost much! I have yet to see a convincing explanation of how continuous collection of object-vtables would be more efficient... Really, even with Robin's proposal, this would work. If a class loader gets into an unloadable state, then most likely, the class loader and all classes it has loaded will have migrated to an old generation. So, as long as we set then end of a class unloading epoch at an old generation collection, then we can simply weaken the class loader reference during collection (only when the bit of all related vtables are unset), then apply the finalization-like rescue dance to class loaders. [*]This wouldn't affect any other operation, during other GC cycles, as Robin's unconditional bit/byte/word vtable write only serves to tell us whether a class loader has had living instances of its classes during the epoch. -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [drlvm] Class unloading support - tested one approach
Note: For preventing collection of class loaders related to active method frames, there are various solutions. One could simply walk all method frame stacks just before the end of epoch collection (my preferred approach) and mark the bit of related vtables. Another approach would be to add an unconditional write on every method call (that would be a big tax to pay!). I'll let you imagine all the variations on that theme. :-) Etienne Etienne Gagnon wrote: I was making it more complex than it needs... Here's an improvement... -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [drlvm] vote on class unloading design (was Class unloading support - tested one approach)
For what it's worth, I'll add my humble +1. Etienne Xiao-Feng Li wrote: Agreed, so plus me. Thanks, xiaofeng On 11/8/06, Rana Dasgupta [EMAIL PROTECTED] wrote: On 11/7/06, Weldon Washburn [EMAIL PROTECTED] wrote: All, It looks like the debate on class unloading has concluded. Let's vote on implementing Robin's proposal. +1 Weldon PS -- When class unloading gets implemented and committed is a seperate issue. +1 I think it's the best proposal among those that were discuussed -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [drlvm] vote on class unloading design (was Class unloading support - tested one approach)
Rana Dasgupta wrote: It's worth a lot, since you have implemented this in SableVM yourself. But I haven't! I was simply discussing the general design so that I could suggest it as a term project in SableVM for my Virtual Machine course, next term. :-) Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [classlib] Preprocessor - out of the abstractness came forth concreteness
Tim Ellison wrote: I'll wait until you have a simple contrived example working and settled upon some syntax for mark-up, then I'll help to mark-up some real example and test the tooling. Let me know when you have something ready to try. Yes, I have to figure out some sane syntax for all this. I'll try to find some easily extensible syntax, so that we don't have to break backward compatibility every time we extend it. :-) Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [drlvm] dynamic object layout
Hi, In the SableVM project, all new feature development is done in sandboxes. This helps maintaining a robust trunk (as much as is possible). My experience is that this works well. Maybe you could do the same in Harmony. Isn't there already some sandboxes? It might or might not work well here, though. In SableVM, bugs are not usually reported against sandboxes. In contrast, in Harmony, almost everything is done through JIRA, including patch submission... Etienne Weldon Washburn wrote: On 11/4/06, Alexei Fedotov [EMAIL PROTECTED] wrote: So my suggestion is to create separate branches for new features which could be merged into the main branch when mature enough to achieve an appropriate level of stability. What do you think? As much as I hate it, I don't know how to avoid branching. Also, we probably need some sort of JIRA coding to reflect which branch has which patches. -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [classlib] Preprocessor - an attempt at [even more] clarity
Geir Magnusson Jr. wrote: [...] 2) Master is virtual, dev1..N are concrete, one dev form is Most Favored Nation Status so something coherent can be shoved into SVN (Q: is MFNS necessary? Nice, yes, but necessary?) I definitely think that it is necessary (where J2SE5 is the preferred devtarget), at least during the initial transition to using the processing tool. It's (I think) the least intrusive method for starting to slowly add non-J2SE5 code in the development trunk while letting existing J2SE5 developers continue developing without changing the way they've been using up to now. By leaving the preferred devtarget in svn, the hordes of existing J2SE developers will have the choice of continuing to use simple svn co/ci to work on the code and won't have to learn to use new, yet not fully tested tools. It also lets the smaller number of non-J2SE5 developers test the processing tools and help identify bugs (what!? :-P) and usability problems without forcing J2SE developers into doing the same. Maybe, in the longer term, it would make sense to have some kind of canonical master form, and store that in svn, but I am a big fan of the least disturbance path, when trying to put in place new processes in an ongoing development project. :-) Question: What would be the advantage (in the longer term) of putting, in the Subversion trunk, source code in non-compilable canonical master form, instead of compilable preferred devtarget form? Why is the existence of master so important to you? Etienne PS: Tim, Thanks so much for providing clear answers and examples explaining the approach! -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [classlib] Preprocessor - CHECKPOINT
Tim Ellison wrote: Agreed. It would be interesting to determine the most effective location for those markers (measured by reverse mapping accuracy vs. number of markers). I am a fan of accuracy... Yet, maybe it would be simple enough if a release was always based on a specific svn URL, then the mark could be totally exact using svn keywords... :-) You lost me here. I'm trying to define tie-points between the 'releasetarget' source and the canonical form. AIUI this will require structured comments in the 'releasetarget', right? Heh! I'm proposing another unusual markup that would be least intrusive. I'm proposing to simply use precise svn url/version identification as markup: file:XXX.java - /* * @ProcessIdentification(j2se,release) * @ProcessURL(https://svn.apache.org/repos/harmony/somepath/XXX.java) * @ProcessRevision(144) * * This tells us that this file is the result of: * process(svn cat -r revision url, j2serelease) */ /* releasetarget code, WITHOUT any additional markup:-) */ ... ... As long as we are *sure* that the releasetarget code was built from precisely that url/version (without any local workspace modification) then it should be as precise as having markups everywhere in the source code. :-) Of course, this adds a (small?) dependence on Subversion, but the gains in markup noise reduction seems to be worth the trouble. Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [classlib] Preprocessor - out of the abstractness came forth concreteness
Tim Ellison wrote: I'm inclined to say that we tackle the 1.5/1.6 differences as our example. Marking-up a module like NIO would be easy enough. That's what I was hoping for. One concrete example :) I'm thrilled to see something happen in the sandbox (that's really the only way to do it... I'd be -1 to trying anything in the trunk. I hope that you knew that...) Yes, of course. But I'll wait for Etienne to agree to the idea since he's signing up to do the hard work. I'll help to mark up the classlibs. Of course, I agree! Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [classlib] Preprocessor - CHECKPOINT
Tim Ellison wrote: For example, if class X of the main version is not part of j2me, process(j2me) would move this file to a subdirectory .streams/. Why would you move the files rather than exclude them? I was assuming that the processor would generate a whole new source tree for each process() target, so that you could work on the original checked-out file in it's 'canonicalized form' for Big Java work, or process(jme) into a new location and work in the non-canonical form your Little Java spectacles on. [...] Neat idea! I would put the following restriction, though: one should NOT modify more than one target at a time. So, you would probably need some way to prevent parallel modifications in spectacle views. One way to achieve this: process(X,target, destination) = Xtarget in a distinct location and X files are changed to read-only and some tracking file F tells us about Xtarget's location. So, you would also need: release(F) = asks all kind of questions (want to lose changes? delete files? etc.) Of course: revert(F?|destination?) = makes X read-write. [Hoping this was clear enough... It's not a very good explanation...] Agreed. It would be interesting to determine the most effective location for those markers (measured by reverse mapping accuracy vs. number of markers). I am a fan of accuracy... Yet, maybe it would be simple enough if a release was always based on a specific svn URL, then the mark could be totally exact using svn keywords... :-) Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [classlib] Preprocessor - CHECKPOINT
Geir Magnusson Jr. wrote: To release code, one would apply: process(X, release-target) = Y Now, it is important to understand that Y, in this case, is NOT suitable for doing any modification as revert(Y) = Kaboom! (The tool will simply report that it can't do it; it won't crash.) ... This is what I thought we were talking about all along - basically starting w/ the full source, and pre-process to the canonical source for the target version. However, I don't understand why I can't go backwards, modulo some manual merging if needed. For example, if I have X and release-version, I should be able to take a Y' and resolve back to X'. There are problematic cases. [...] It's the problematic cases I am trying to get entirely rid of by design, by having an exact transformation process, in contrast with the usual branch/merge inexact process. Of course, this means that development happens using complete source code (this is either the canonical form or a development target form, but not a release form). For applying user-provided patches, then it can still be done the good old way, using these little patches made from release code, but applying them directly to the canonical or dev-target code, resolving conflicts manually. [The smaller the patch, the easier it is to apply it on files with target-specific code.] The idea is that manual processing is never required when coding using canonical and dev-target modes. No inexact modifications, no difficult merges, etc. And, as a bonus, communication between parrallel target developers. I know, it's an unusual development processing tool, but, hey, why shouldn't Harmony innovate? :-) Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [classlib] Preprocessor - CHECKPOINT
Geir Magnusson Jr. wrote: [...] See, I'm hoping for something a tad different : 1) For building : process() (and revert() for fun) for cmd line use for the build scripts, so we just do ... 2) For development : IDE plugin where a) I can tell plugin that my project def/configuration is whatever, it using metadata in a file, only consider the code that is defined (or not excluded - whatever is easier) b) I can tell plugin to look at X, know that I want java5 and in situ, it shows me what Z would look like. I edit this, but I'm really editing X. And I want to be sure, I push a button or have a split screen that shows me what X really looks like. I can edit in X, or in virtual Z, or both at the same time, as I'm just really editing X. I have nothing against this! But, we have to make sure: 1- we don't lose the communication aspect of telling developers about parrallel development. 2- we have a robust language design; this is best achieved by first building the command-line based tool, then extend it into a full-blown Eclipse plugin... :-) Etienne -- Etienne M. Gagnon, Ph.D. [EMAIL PROTECTED] President http://www.sablevm.org/ J-Meg inc. http://www.sablecc.org/ -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [classlib] Preprocessor - CHECKPOINT
Sian January wrote: I may be totally off track here, but how about just having two copies of all the files that differ? I don't believe it would be that many, and it would save us from having complicated source files or having to use special tools or special IDE plug-ins. For me the value of having clearly readable source code and being able to use an IDE out of the box outweighs any extra effort there may be with this solution. I suspect that two copies might eventually become three, four, five copies... Also, having parrallel copies does not help when modifying shared code in these copies. Actually, it would be even worse than using the normal branching feature of Subversion, as you wouldn't get neat svn merge help... Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [classlib] Preprocessor - CHECKPOINT
Geir Magnusson Jr. wrote:
I have nothing against this! But, we have to make sure:
1- we don't lose the communication aspect of telling developers about
parrallel development.
I'm not sure I understand what problem you are trying to solve there. We
do parallel development on the source base right now - no one ever does
a blocking checkout - it's optimistic.
What tells you, when you modify method m() of class C that,in the middle
of it, there's different ways of processing something in j2se6?
e.g.
class C
{
.../* many, many methods */
private void m()
{
... /* canonical j2se5 code */
...
// @Process(j2se6)
// { do something additional }
...
...
}
}
So, if you were, for example, to move m() to become the first method,
and ALSO change its name...
class C
{
private void m2()
{
... /* canonical j2se5 code with modifications!! */
...
// @Process(j2se6)
// { do something additional }
...
...
}
.../* many, many methods */
}
Such changes are usually not dealt with very neatly with svn merge.
Now, if using the processign tool, you get at least 2 benefits over svn
merge:
1- No need to do any svn merge... The method was moved AND renamed
even for j2se6.
2- You were aware, when moving the method and changing its name that you
could possibly be making the life of j2se6 developers a little more
difficult.
The communication aspect of 2 can be quite helpful when doing
system-wide changes. Just think about the effect of simply doing a
system-wide reindentation of source code; this is a nightmare when
developing using branches, as diff/merge tools are line-based, not
syntax-based. So, if your new indenter moves things across lines,
you're in hell.
Believe me; I've lived through that trying to maintain a well indented
GNU Classpath in SableVM's repository.
I am not saying that using the tool should replace using branches for
all situations, but there are situations where it would be more
effective not to use branches when the code is mostly identical. Using
traditional pre-processing tools, in such cases, is often the solution
adopted by some people; the problems of simply-mided traditiona
pre-processing has been discussed earlier in this thread. I am
proposing, instead, to use an innovative syntax-based, revertable
processing. That's all. [I guess you'll need to see it working to
make sense of it... I'll work on a prototype. I've just submitted the
ICLA/ACQ].
Etienne
--
Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/
SableVM: http://www.sablevm.org/
SableCC: http://www.sablecc.org/
signature.asc
Description: OpenPGP digital signature
Re: [classlib] Preprocessor - CHECKPOINT
Geir Magnusson Jr. wrote: The communication aspect of 2 can be quite helpful when doing system-wide changes. Just think about the effect of simply doing a system-wide reindentation of source code; this is a nightmare when developing using branches, as diff/merge tools are line-based, not syntax-based. So, if your new indenter moves things across lines, you're in hell. So here's where I'm starting to believe I'm missing something important : Can you give me an example of the communication that you imagine taking place here? That's the part I'm just not grokking here. How about: j2se5 developers, seeing that his planned modification to some code will lead to utterly broken nearby j2se6 (commented) code fragments, steps on this mailing list and asks j2se6 developers to help him find the most harmless way to do the change (or, at least,warn them about the problem)? I am not saying that using the tool should replace using branches for all situations, but there are situations where it would be more effective not to use branches when the code is mostly identical. I want to avoid branches at all costs. [somewhat off-topic] I think that we should consider tools/approaches on their merit. Sometimes, branches are the right solution... You wouldn't want to use syntax-processing instead of branches for managing sandboxes. ;-) Clearly there's some confusion here. My goal is to 1) Avoid branches at all costs so we can share as much code, and get as much benefit for collaboration between different versions, and different platforms, if that happens. Agreed. 2) make it simple to work in either the 'master' code, or the 'target' code through tooling, including standard IDE activities like debugging Agreed. 3) make it easy for users to report bugs based on 'target' code, including patches and line numbers Ah! That's something to add to my requirements. Fine! I hadn't included the patches thing into account. It doesn't break what I've exposed so far; it simply adds to it. What I've suggested is : forward(X, platform) - Y reverse(X, platform, Y') - X' OK. I see this in *addition* to: forward(X, devtarget) - Y reverse(Y') - X' So, to simplify the discussion, let's rewrite your proposal as: forward(X, releasetarget) - Y reverse(Y',X,releasetarget) ~~ X' (possibly reporting conflicts/problems) That's neat. I like it. Yet, we would encourage developers to work and submit patches using devtarget code, instead of releasetarget code. In other words, here's how I see the distribution(s): - Binary j2se5 Harmony release: includes j2se5release src.jar for end-developers using debuggers to step through API code. - Source j2se5 Harmony release: API are in j2se5dev form. The build process uses the processing tool to generate j2se5release src.jar. - Binary j2se6 Harmony release: includes j2se6release src.jar for end-developers using debuggers to step through API code. - Source j2se5 Harmony release: API are in j2se6dev form. The build process uses the processing tool to generate j2se6release src.jar. Why? Because reverse works much better with devtarget than releasetarget code, and the communications benefit of devtarget that are lost in releasetarget code (because of stream code erasure). In addition, using smart IDEs, there wouldn't be much difference for developers between the visible formats of dev vs release code when working on it. for X being a single class. To produce a complete target source base, walk the single source tree as defined by a 'platform descriptor' : I don't really understand the platform descriptor thing, yet I think that it is, somehow, an orthogonal issue to the discussion above. [Remember my long message describing 2 levels of processing: file/directory-level and source-level.] My first prototype tool would attack source-level processing (discussed above), to validate the approach. Maybe you could try rephrasing your file/directory-level (right?) proposal? How does it differ from Tim's proposal? Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [drlvm] Class unloading support - tested one approach
Robin Garner wrote: - Allocate a byte (or word) in each vtable for the purpose of tracking class reachability. Yep, there's no reason to keep the bits (or words, for performance) in the class loader, even in the approach I've proposed. They could be moved to the vtable. Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [classlib] Preprocessor - CHECKPOINT
Geir Magnusson Jr. wrote: There's caching too, I think. LogCache4J What I meant was that it didn't seem like we came to a conclusion on it - that if we had a general pre-processing solution, we could use that too for logging, rather than have two. The actual use-cases will help figure this out. Here two typical some use cases, and some proposed solutions: Problem --- logging, and other situations where you really don't want to put the additional source code in the main source files Solution use aspects (Plug: you might want to give a look at the optimizing abc compiler) Problem --- supporting a different API specifications/versions, such as j2me and j2se1.4, in addition to the main version (e.g. j2se1.5) Solution This is a trickier problem. We can divide the problem at two main levels: 1- file/directory level 2- source-code level At the file/directory level, each version (e.g. j2me, j2se1.4, ...) might include additional files (relative to the main version), and might not include some files of the main version. In other words, j2me might not contain database APIs. Managing file inclusion/exclusion can be done in various ways. a) ant based: use distinct ant (sub-)files for each version. The problem is that IDEs (e.g. Eclipse) will most likely show some of the excluded files in its class/files browser. This wouldn't be very elegant. It also implies always compiling through ant files. Of course, one could develop Eclipse-specific configuration files to mimic the inclusion/exclusion of ant files, but then, this opens the door for discrepancies between ant vs eclipse inclusion/exclusion lists. I don't like it. b) custom-tool based: the custom processing tool I proposed could also parse inclusion/exclusion lists, and use these lists to move files around, in addition to processing the content of unmoved files. For example, if class X of the main version is not part of j2me, process(j2me) would move this file to a subdirectory .streams/. If a class Y is not in the main version (the one used for svn ci), it resides in subdirectory .streams in the trunk. process(j2me) moves this file into the normal directory. As for IDEs, now you can configure them to automatically exclude .stream/ directories. Inclusion/exclusion could be managed in two ways: 1- the processing tool could look for inclusion/exclusion list files, such as j2me.inclusion, j2me.exclusion, main.inclusion, main.exclusion, etc. This would lead to the best performance (for process(X)), yet it does require much manual update of inclusion/exclusion lists, with all the discrepancies that result from human mistakes while updating these files. 2- (my preferred way), directives, at the beginning of the source code would indicate whether a file is included in a version or not. Depending on target, the file would be moved to the appropriate directory (normal, .streams). Of course, there's also the problem of non-source files, i.e. resources. IMO, resources could be managed using specific directories (.main/, .j2me, .j2se1.4) and a .shared/ directory with symbolic links in the specific directories. As for source-level management, you would use my proposed processing tool to view the source files with the right spectacles [as Tim said so elegantly:-)]. For development targets, it is important that: revert(process(X, target)) = X By development target I mean a target that is meant for Harmony developers in prevision of reverting modified code to a format suitable for svn ci (i.e. revert to main target). For comfortable IDE development, one could imagine that the IDE editor can reduce to one-line visible comments (or better, specially formatted ones) so that it gives you the impression that you are really wearing target-specific spectacles. [I know Eclipse allows for such things already]. To release code, one would apply: process(X, release-target) = Y Now, it is important to understand that Y, in this case, is NOT suitable for doing any modification as revert(Y) = Kaboom! (The tool will simply report that it can't do it; it won't crash.) Yet, I think that it would be important that the processing tool leaves markers in Y, so that we could also have a tool to help finding the canonical source line of a reported bug: revertLine(Y, L') = L (where L' is a line reported by end-developer, and L the related line in svn). Markers would be short single lines comments, so the end-developer annoyance would be kept minimal. What do you think? I am really offering to develop this custom tool. Just help me identify various Harmony needs I might have missed! Of course, this tool is not the best solution to ALL problems, yet, so far, I think that it seems to best address the problem of supporting various API
Re: [classlib] Preprocessor - CHECKPOINT
Mikhail Fursov wrote: At the file/directory level, each version (e.g. j2me, j2se1.4, ...) ... Just my $0.02: IMO it's unreal to support J2SE 1.4 1.5 in the same source. Too many differences in the language due to generics. This example needs branches weekly manual merges (not a big problem imho) You're absolutely right! I wasn't thinking about it when I took that example. This is really a typical case (j2se 1.4) where using svn branches is the right solution. But, for j2me1.5, java for credit cards 1.5 and j2se1.6 (maybe, if they don't redesign the Java syntax, etc.), I think that a tool along the lines I was proposing would be best. Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [classlib] Preprocessor - CHECKPOINT
Mikhail Fursov wrote: On 11/1/06, Etienne Gagnon [EMAIL PROTECTED] wrote: For comfortable IDE development, one could imagine that the IDE editor can reduce to one-line visible comments (or better, specially formatted ones) so that it gives you the impression that you are really wearing target-specific spectacles. [I know Eclipse allows for such things already]. ... Etienne, What is 'comfortable IDE development' if you can't modify the Y? Am I missing something here? Maybe my text was wrongly formatted... You looked at the wrong example. Comfortable development happens only using development targets. E.g. 1- process(X, devtarget) - Z 2- edit Z in IDE using comfortable development, where you see a single commented line for every hidden stream code chunk, keeping you aware that other streams have related code there [you click on the + in Eclipse if you want to see the complete chunk]. Of course, you should never delete a chunk without consulting other stream developers first. So: edit Z - Z' 3- revert(Z') - X' this works, as long as devtarget is a stream code preserving target (a development target). 4- svn ci of X' :-) You wouldn't want to do the same with a release target. A release target is a target where you want to completely remove other streams source code from the processor output. This is so your typical J2SE end developer that looks at Harmony's source.zip code wouldn't have lots of non J2SE commented code in his face. [This was a concern expressed earlier in this thread]. The output of process(X, releasetarget) should NOT be used for development, not within, nor outside an IDE. It's only role is to give end developers clean source code to look at. :-) Etienne PS: Note how revert() does not expect a target argument. So, to switch from j2me to j2xx development, one must: process(revert(Z),j2xx) [where Z == process(X,j2me)]. This simplifies quite a few things... -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [classlib] Preprocessor - CHECKPOINT
[WARNING: Long!] Here's an idea. It would be quite easy (for me, at least ;-) ) to develop a line-preserving AND revertible processing tool so that: 1- source + javac = j2se 2- source + processor(target) + javac = target, where target is a parameter (j2me, etc.) 3- source + processor(target) = source for target, where line numbers are identical to source 4- source for target + reverseprocessor(target) - source :-) That's the thing lacking in most preprocessors, as often you would like to modify the processed source and propagate these changes back to the initial source; this is usually a pain as it must be done manually. Not here! I am, usually, in favor of using branches, for development, yet, in the current case, I think that branches have their drawbacks: 1- I have yet to see a branch/merge system that deals well with major code reorganization in a single file. 2- There's always the infrequent, yet very annoying (if not dangerous when it happens) semantically invalid merge result. 3- Developers of the main branch will remain mostly unaware of the parallel development on other branches. In this case, this can be a problem, as having j2se developers aware that they are changing lines for which other branches maintain distinct versions ones might be a good communication tool. The idea, for line preservation, would be to either use revertible comments (easiest) or maybe play with special Java annotations, if they prove sufficient for the task. The advantage of intelligent Java-syntax-aware processing (instead of dumb preprocessing) is that the processor could work intelligently. Here's a tentative example to illustrate my idea. (This is not a fully thought-up example/syntax; it's just a way of trying to explain the concept): Java source (j2se) --- ... // @Processor(add to j2me // // float some_field; // // ) @Processor(Not in j2me!) int some_field = some + initializing() code; ... Processed source (target = j2me) -- ... @BeginRevert(add to j2me) float some_field; @EndRevert() // @Revert(@Processor(Not in j2me!) // int some_field = //some + // initializing() //code; ) ... Modified processed source ;-) - ... @BeginRevert(add to j2me) float some_field = missing initializer!; @EndRevert() // @Revert(@Processor(Not in j2me!) // int some_field = //some + // initializing() //code; ) ... Revert-processed source (= ready for svn ci) --- ... // @Processor(add to j2me) // // float some_field = missing initializer!; // // ) @Processor(Not in j2me!) int some_field = some + initializing() code; ... The intelligent part comes with the idea that the @Processor directive, in the j2se uncommented source above, knows that it applies to the whole field declaration, including any initializer, regardless of the number of lines used by the field declaration. Such directives could be limited to work on: 1- classes/interface declarations 2- method/field declarations 3- static statement blocks 4- statement blocks in general 5- other things? The idea is that it is not a character/lined based processor, but really a syntax-driven one. Using comments (as above) is optional, yet I recommend doing it so that text editors don't syntax color unapplied code in any current situation (source, processed-source). Both j2se j2me (as well as other target developers) will be able to work on the processed and the original source code. Typically, the revert processor will be used by j2me developers before svn ci of their modifications. It would be quite easy for me to develop such a tool using SableCC. FYI, the output of SableCC is not encumbered by any copyright other than whatever the grammar developer wants. Also, I will contribute the developed tool to Harmony if this approach is chosen (or develop it as a Harmony developer, if you prefer...). [Geir: Yes, I know, I have to send the paperwork...] Some advantages: 1- Even if you package processed source into a jar file (for j2me users), line numbers match exactly. Bug reports from users won't be about Mars or Jupiter. :-) 2- Debugging using both initial and processed source works flawlessly. 3- Major code reorganization can be done cleanly without the need for bug-prone, tricky, hand-made merges (as automated merge can be misleading in such case) 4- We can make the tool and its syntax as gentle as you want; we have total control on its syntax and semantics. :-) 5- We could add a syntax checking mode to the processor, so that in that mode, *all* code would be checked for syntax correctness (even commented processor directives) instead of modified for a specific target. This could be added to the ant script for all developers, so that at no point somebody svn ci syntactically invalid code in case they, by
Re: [drlvm] Class unloading support
Aleksey Ignatenko wrote: Am I wrong, or does this proposition imply collecting classes independently from their class loader? If this is the case, I have to ... Yes, you are wrong. This proposition implies collection of classloader and clasess loaded by it at once. You can see what is class registry in the first letter of the discussion - Excellent. That was one of my main worries. And what about gagnon-phd.pdf: ... Drlvm already has similar functionality: look at classloader.h, function void* Alloc(size_t size); You'll see that most of classloader's data (not 100% yet) is already allocated from pool of that classloader. Heh! Super. Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [drlvm][sablevm] Desing of Class Unloading Support
4- [Informative] A class loader instance is also most likely to maintain ... This is not true. Look for the thread [drlvm] Non-bug difference HARMONY-1688?, where Eugene Ostrovsky desribed initiating loaders in details with links to specification. OK. 7- Each class loader structure maintains a set of boolean flags, one ... Requires identification of object' class type during GC. Will most probably degrade GC performance. Not necessarily. It really depends whether you want to always care about class unloading, or if you only care about it when doing major collections. Maybe you only want to unload classes on full collections, when all generations are collected. In such a case, you would not do anything special (e.g. not maintain these bits) during any other collection than full ones. As for type identification, this is not necessarily required. You only need to add a pointer in the vtable header (That's a 1-liner in SableVM) that points to: 1- NULL for any class of an unloadable class loader (e.g. bootstrap, system?) 2- ClassLoader structure, for ones that we wish to unload (user class loader). Maybe that's the big change to the vtable that was argued about in this thread? If yes, the bigness of it was quite misleading to me; such a change is a trivial one, to me. In SableVM, it's really just the following change: 1- Add a field in the vtable struct in file type.h (1 line) 2- Initialize the field to non-zero for classes of non-bootstrap loader (1 line). No big deal... As an additional optimization(???), one could set a bit in the object header when the pointer (in the vtable) is not NULL, yet parsing the bits might cost more that dereferencing the vtable pointer and checking the field against NULL. [I know, this is most probably a very bad idea!] You could even go further and only do class unloading when a special request is made for it. This way, you don't do anything special during normal collection. When the special request is done, you do a full GC and unload any class (and loader) you can... I guess that some of these ideas had already been somehow discussed on this thread; I likely did misunderstand some of the few messages I read. 8- Each method invocation frame maintains a hard reference to either its ... Not generally true for optimizing JITs. This (or class) can be omitted from enumeration if it is not used anywhere in the code. Generally, this technique reduces number of registers used in the code (register pressure they call it :)). OK. Yet, for correctness, you want to make sure that at any time you want to unload classes, you do make sure that you take into account classes of active methods. This can be achieved in various ways; I was proposing one that was natural to SableVM. :-) 9- A little magic is required to prevent premature collection of a class ... This requires more involvment of a GC in unloading process and affects GC code more. In DRLVM, GC is designed to be a replaceable component. Moreover, we already have 3 different working GCs and MMTk on the way. So, including GC into the design is not a good idea for DRLVM. There is a dependency between GC and class unloading. Somehow, you must be aware if there are still instances, around, of needed classes. You don't need to always care for class unloading, while doing GC; as I said above, you could reduce the overhead to well defined moments. [You could have rules such that: at full collections, only, and no more than once per 1hour | 10 minutes | ... -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
[admin] ICLA / ACQ (Was: [drlvm][sablevm] Desing of Class Unloading Support)
Geir Magnusson Jr. wrote: However, it would be great if you had an ICLA and ACQ on file to save you the trouble of typing this in the future :) Better living through paperwork! OK; I should have made this a while ago... Can they be submitted by email (where?) as scanned documents in PDF format? This is usually accepted here as much as Fax documents. [Much better quality, actually!] Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [drlvm][sablevm] Desing of Class Unloading Support
Actually, I think that Java vtables would be more expensive than my proposed approach (when you take my proposed improvements in my reply to Pavel Pervov), as you add overhead to all GC cycles! [Unless you don't trace from every visited object to its vtable?] I really don't like much the idea of an object vtable. It requires things such as pinning, etc. Looks more expensive than my solution. Etienne Rana Dasgupta wrote: Etienne, This is a good design, thanks. Conceptually, reference counting in the VM is somewhat similar to Aleksey's proposal 1, if I understand correctly. This design also requires quite a few hand-offs between the VM and GC. In DRLVM, the problem is that we have quite a few GC's, not all within our control. However, it seems to me that we can either desire to make unloading automatic, in which case, we will need things like java vtables etc and leave most things to the GC. Or we can do refcounting or tracing in the VM, and work lock step with the GC(s). I am not sure which is the better way. -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [drlvm][sablevm] Desing of Class Unloading Support
Yet: 1- You do need pinning, so you rule out some of the simplest GCs (e.g. simple, non-generational copying without pinning.) [Apparently, for some very large heaps, simple copying a can be quite difficult to beat, efficiency wise, if you believe some relatively recent JikesRVM related paper...] 2- You do have overhead even on minor collections. With my approach, you could limit the (quite similar to yours, if you put a class-loader/NULL pointer in the vtable) overhead only to selected GC cycles. Of course, I am sure that all of the proposed approaches have their benefits/drawbacks. I was simply contributing to the ongoing discussion. I have no special reason to try very hard to convince you that my idea is better than yours! I'm only joining the debate for trying find the most suitable solution. I've already gained knowledge, from the discussion so far, that I'll be able to apply eventually in SableVM. :-) Maybe the best solution lies in mixing some of the various ideas proposed so far... Etienne Ivan Volosyuk wrote: Actually, no need to add the overhead to _all_ cycles. We don't need to trace the vtables everytime. On minor collections all the pinned vtables can be linearly scanned, thus most expensive tracing from object to vtable can be avoided in this case. -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [classlib] Preprocessor - CHECKPOINT
Tim Ellison wrote: IMO it's not ideal that the preprocessed source still contains all the streams, albeit in comments. It wouldn't make the source very 'consumable' to the Mrs. SE or ME developer. Hmmm... It's always possible to have a special output mode that puts empty (or advertizing, hehe) comments, instead of other stream code (thus, preserving line numbers). To continue on my earlier example: Java source = j2se end-developer --- ... // Download Harmony[tm] from // // http://the.nice.harmony.url/download // // :-) @Processor(Not in j2me!) int some_field = some + initializing() code; ... Or, more likely: Java source = j2se end-developer --- ... // Please ignore this comment. It has been // intentionally left here to preserve line numbers // for bug reporting purpose. // // Please report bugs to http://bugs.of.harmony.url/... @Processor(Not in j2me!) int some_field = some + initializing() code; ... So, J2ME J2SE end-developers are kept happy. As a bonu$, you can al$o $tart a nice busine$$ $elling advertizing $pace in $ource code. ;-P Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [drlvm] Class unloading support
If I get it right, in case of automagic unloading, GC does all the job without a knowledge whether it collects a class, a classloader or whatever else. Perhaps I'm missing something, but to provide a callback on class unloading, the GC must know the semantic of the object being collected. Am I wrong, or does this proposition imply collecting classes independently from their class loader? If this is the case, I have to say that I disagree with the proposed approach. The JVM spec says quite clearly: 2.17.8 Unloading of Classes and Interfaces A class or interface may be unloaded if and only if its class loader is unreachable. The bootstrap class loader is always reachable; as a result, system classes may never be unloaded. Just think about it. One could take an instance o of a class C loaded by L, call it (C,L), and call o.getClass().hashcode(). Store this integer some where. Then, o could die, and maybe (C,L) unloaded while L is still reachable. As L is still reachable, some code could do a L.findClass(C).hashcode(). This will likely result in a different hashcode, in full breach of the both the VM and API specifications. In a related note, for memory management I highly encourage Drlvm to look at Chapter 3 of http://sablevm.org/people/egagnon/gagnon-phd.pdf that exposes a simple, yet very effective approach for managing class-loader related memory (i.e. memory used to store internal class data, vtables, jitted code) so that it can all be freed efficiently at class-loader unloading time. Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
[drlvm][sablevm] Desing of Class Unloading Support
Hi all, Here's a more structured proposal for a simple and effective implementation of class unloading support. In accordance with Section 2.17.8 of the JVM spec, class unloading (and its related native resource cleanup) can only happen when the class loader instance becomes unreachable. For this to happen, we put in place the following things: 1- Each class loader is represented by some VM internal structure. [We'll call it the class loader structure]. 2- Each class loader internal structure, except (optionally) the bootstrap class loader, maintains a weak reference to an object instance of class ClassLoader (or some subclass). The Java instance has some opaque pointer back to the internal VM structure. The Java instance is usually created before the internal VM structure. The instance constructor is usually in charge of creating the internal VM structure. [We'll call it the class loader instance] 3- Each class loader instance maintains a collection of loaded classes. A class/interface is never removed from this collection. This collection maintains hard (i.e. not weak) references to classes/interfaces. 4- [Informative] A class loader instance is also most likely to maintain a collection of classes for which it has initiated class loading. This collection should use hard references (as weak references won't lead to earlier class loading). 5- Each class loader instance maintains a hard reference to its parent class loader. This reference is (optionally) null if the parent is the bootstrap class loader. 6- Each j.l.Class instance maintains a hard reference to the class loader instance of the class loader that has loaded it. [This is not the initiating loaders, but really the loading loader]. 7- Each class loader structure maintains a set of boolean flags, one flag per non-nursery garbage collected area (even when thread-local heaps are used). The flag is set when an instance of a class loaded by this class leader is moved into the related GC-area. The flag is unset when the GC-area is emptied, or (optionally) when it can be determined that no instance of a class loaded by this class loader remains in the GC-area. This is best implemented as follows: a) use an unconditional write of true in the flag every time an object is moved into the GC-area by the garbage collector, b) unset the related flag in all class loader structures just before collecting a GC-area, then setting the flag back when an object survives in the area. 8- Each method invocation frame maintains a hard reference to either its surrounding instance (in case of instance methods, i.e. (invokevirtual, invokeinterface, and invokespecial) or its surrounding class (invokestatic). This is already required for synchronized methods (it's not a good idea to allow the instance to be collected before the end of a synchronized instance method call; yep, learned the hard way in SableVM...) So, the overhead is quite minimal. The importance of this is in the correctness of not letting a class loader to die while a static/instance method of a class loaded by it is still active, leading to premature release of native resources (such as jitted code, etc.). 9- A little magic is required to prevent premature collection of a class loader instance and its loaded j.l.Class instances (see [3-] above), as object instances do not maintain a hard reference to their j.l.Class instance, yet we want to preserve the correctness of Object.getClass(). So, the simplest approach is to maintain a hard reference in a class loader structure to its class loader instance (in addition to the weak reference in [2-] above). This reference is kept always set (thus preventing collection of the class loader instance), except when *all* the following conditions are met: a) All nurseries are empty. b) All GC-area flags are unset. Actually, for making this practical and preserving correctness, it's a little trickier. It requires a 2-step process, much like the object-finalization dance. Here's a typical example: On a major collection, where all nurseries are collected, and some (but not necessary all) other GC-areas are collected, we do the following sequence of actions: a) All class loader structures are visited. All flags related to non-nursery GC-areas that we intend to collect are unset. If this leads to *all* flags to be unset, the hard reference to the class loader instance is set to NULL (thus enabling, possibly, the collection of the class loader instance). b) The garbage collection cycle is started and proceeds as usual. Note that the work mandated in [7-] above is also done, which might lead to setting back some flags in class loader structures that had all their flags unset in [a)]. c) After the initial garbage collection is applied, and just before the usual treatment of weak references (where they are set to NULL when pointing to a collected object), all
Re: [drlvm] Class unloading support
Hi Weldon, Weldon Washburn wrote: I read section 3.2.3 (Class-Loader-Specific Memory) of gagnon-phd.pdf. Please tell me if the following is a correct interpretation. You create a new memory manager that is uniquely associated with each new class loader. Right. All the C data structures associated with a class loader (classes, vtables, etc) are malloc()ed out of the associated memory manager. [For those who have not read it...] malloc()ed is a big word... It is rather simpleAlloc()ed, i.e., once allocated, you cannot free it (...or if you do, the free-list manager is very minimal, performs no checks [you have to tell it how much you are freeing] and no aggregation). I do discuss this in the Chapter, of course, and you can look at the implementation in SableVM. [The SableVM trunk is under AL2.0 (unlike released versions)]. When the class loader becomes unreachable, then its associated memory manager is deallocated which automatically frees all the associated C structs (classes, vtables, etc.) Yep. Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [drlvm] Class unloading support
I have missed some messages of this thread, yet I do not remember seeing a discussion of what seems to me the obvious solution to the problem. So, here it is. Why don't you simply add a reference count on classes which is incremented on object allocation and decremented on object reclamation? [In case you use a copying collector, you could keep a separate count (in the class) for the collected area, so that you only have to count copied objects]. You would also use reference counting for the class loader (therefore eliminating any cyclic problem that you could have with normal garbage collection). This would work very well as unloading only happens when the class loader can be unloaded along all of its classes. No need for any supportive information in object header, or anything complex... Am I really missing something? Just an idea... Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [drlvm] Class unloading support
Ivan Volosyuk wrote: If I understand you correctly, you suggest to increment per-classloader object counter on allocation... It can be much overhead with the solution, as most of the objects die young. Do I miss something? No, I was thinking about a per-class counter. Actually, a counter is not needed. A simple boolean is suffucient (one boolean per gc generation in each class), so the cost would be a single inconditional memory write per object allocation. I would think that this would be lost in the noize of object field zero initialization. No? Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [drlvm] Class unloading support
Wait a minute! I missed something. Actually, there is no need to track allocations in the young generation! Only survivals. So, you apply the boolean trick only for objects that survive the nursery collection. So, there would be no bit overhead in objects, nor work overhead on allocations. Just a little overhead on moving objects across generations. As for the class loader, there are many solutions; one is to maintain a list of loaded classes with surviving objects. Every time a class sets one of its gc-area booleans to false (at the end of a gc cycle), it checks whether all its other gc-area booleans are also false. If yes, it removes itself from the class-loader loaded class with surviving objects list. When this list becomes empty, the class loader can be unloaded (as soon as it is not referenced elsewhere). Making any sense? Etienne Etienne Gagnon wrote: Ivan Volosyuk wrote: If I understand you correctly, you suggest to increment per-classloader object counter on allocation... It can be much overhead with the solution, as most of the objects die young. Do I miss something? No, I was thinking about a per-class counter. Actually, a counter is not needed. A simple boolean is suffucient (one boolean per gc generation in each class), so the cost would be a single inconditional memory write per object allocation. I would think that this would be lost in the noize of object field zero initialization. No? Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [drlvm] Class unloading support
I don't know about drlvm, but SableVM does keep a reference (i.e. a local native reference) in each method activation frame to either the instance (in case of an instance method) or to the class object (in case of a static method). This is required for correctly (and efficiently) implementing the synchronized method modifier. Of course, one could have distinct method activation records for synchronized vs unsynchronized methods, yet, if you think about it, keeping identical activation records give you, for almost free, the class/instance survival you seek to prevent class unloading. The class loader should be not be unloaded before 2 conditions are met: 1- No instances of any loaded class. 2- The ClassLoader instance has been garbage collected. This means that, internally, there is a class loader structure which maintains a weak global native reference to its ClassLoader instance. Hoping this helps... Any volunteer to try it in SableVM? Much easier than coding it in drlvm for doing some initial experimentation. ;-) Etienne Ivan Volosyuk wrote: I like your idea. We can skip counting on young generation. Good, this approach doesn't force us to convert VTables to java objects. There is one more thing to clarify. Having no objects in heap we can have running method in stack which holds classloader from unloading. How can we deal with that? Should we examine root-set when going to trigger deallocation? -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [classlib] Preprocessor (was Re: [classlib][rmi] Code smell - Thread.sleep() in ActivationGroup method)
Chris Gray wrote: For JavaME I think it's the only way we'd be able to maintain a single source tree. We need to be able to #ifdef out references to classes we don't have, methods we don't implement, etc.. That much being said, I don't have a recommendation for a tool to use. Java syntax is sufficiently C-like that cpp is probably do-able, but we'd probably stumble over a whole series of gotcha's, and cpp isn't exactly [deity]'s gift to preprocessing anyway. Maybe one of the aspect-oriented tools (with which I am not at all familiar) could be a better bet? You could always do clean source-to-source processing using SableCC...:-) Java is a nice language to parse, so you could do some clean parsing, instead of the dumb unstructured text replacement of preprocessors. Actually, if all you need if ifdef'ing out undesirable references, it could be done by hiding modification directives in structured comments, so that these comment remain javac invisible. This way you could make it such that: 1- Plain source compilation - j2se . 2- Structured processed source compilation - j2me . If you need it, there are 2 or 3 Java 1.5 grammars available for SableCC (different parsing approaches, not different syntax!). As I said, Java is a pleasure to parse when compared to C C++. It's just an idea, of course... [I know that people can start religious wars about pre-processing; that's why I am suggesting a clean approach, so that j2se people don't have to pre-process]. Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [announcement] Apache Board approved Apache Harmony as a Top Level Project of the ASF
Congratulations! Etienne Geir Magnusson Jr. wrote: I am happy to report that the Apache Board was willing to consider our proposal and voted to accept it at today's board meeting. As stated in the Incubation vote, this is a necessary condition for graduation from the Incubator. Therefore, upon a successful outcome of the Incubator PMC vote, we are Apache Harmony, project of the Apache Software Foundation! Congratulations to everyone! When the vote is complete, we'll get to work on the transition activities, but until then, just give yourself a well-deserved pat on the back. -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/
Re: [vote] Graduate Apache Harmony podling from the Incubator
+1 Good luck! Etienne Geir Magnusson Jr. wrote: [...] So, without any further ado : [ ] +1 Graduate Apache Harmony from incubation, and let it petition the board for Top Level Project status [ ] 0 No opinion [ ] -1 No, don't graduate Harmony. Here's why : This vote will end 72 hours from now + time of Apache mail outage. It will therefore end on Monday, October 23, at 3:30 PM eastern, + delta for mail outage. [...] -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ - Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [SableVM] please tell us the status of the Apache License 2.0
Hi all, Yes, I'm still alive... I've been overloaded by work, so I took a leave from reading/answering mailing-lists, over the last month. OK. Here's the status: SableVM's trunk is licensed under the AL2, yet, as the Harmony porting is not yet functional, there is no release yet of that code. The last release was under the LGPL, that's why the web page still says LGPL. To be sincere, I should admit that we need more people to help with the Harmony transition. Given the DLRVM contribution, it should be even easier to achieve the core classes re-implementation. Any volunteer should express himself on the SableVM mailing list. Thanks, Etienne Weldon Washburn wrote: Nektarios, Thanks. I looked at a few files in the SableVM source tree. They all point to AL2 license. It looks like the only clean up is the home page, http://sablevm.org/ Geir, Does someone at Apache need to officially say SableVM license is now OK for Harmony work? On 6/23/06, Nektarios K. Papadopoulos [EMAIL PROTECTED] wrote: Weldon Washburn wrote: Etienne, My apologies if its already been disclosed on harmony-dev. I searched for 20 minutes and could not find anything more recent than: http://sablevm.org/lists/sablevm-devel/2006-March/000620.html It is not more recent, but a similar message to harmony-dev is: http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200603.mbox/[EMAIL PROTECTED] I guess the SableVM home page[1] needs an update (it still says LGPL on the bottom), but the AL2 license is in the svn [2],[3] [1]http://sablevm.org/ [2]http://sablevm.org/svn/repository/sablevm/trunk/LICENSE [3]http://sablevm.org/svn/repository/sablevm/trunk/AL20 -- __ Nektarios K. Papadopoulos Senior Engineer Software Engineering Group inAccess Networks 95A Pentelis Avenue.Tel: +30-210-6837640 152 34 Halandri Athens Fax: +30-210-6899504 __ - Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Etienne M. Gagnon, Ph.D. [EMAIL PROTECTED] President http://www.sablevm.org/ J-Meg inc. http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [announce] New Apache Harmony Committer : Mark Hindess
Congratulations! Geir Magnusson Jr wrote: Please join the Apache Harmony PPMC in welcoming the project's newest committer, Mark Hindess. -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [VOTE] Acceptance of HARMONY-438 : DRL Virtual Machine Contribution
+1 Etienne Geir Magnusson Jr wrote: I have received the ACQs and the BCC for Harmony-438, so I can assert that the critical provenance paperwork is in order and in SVN. Please vote to accept or reject this codebase into the Apache Harmony class library : [ ] + 1 Accept [ ] -1 Reject (provide reason below) Lets let this run a minimum of 3 days unless a) someone states they need more time or b) we get all committer votes before then. I think that getting this into SVN and letting people supply patches against SVN will be productive... geir - Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: OPEN Specification
Hi Anton, Are you proposing that all Harmony JVMs must abide by the OPEN proposal? If yes, I think that some process has to be put in place to present and discuss each of this proposal's part, and dedicate time to do so. IMO, I don't think that everyone (in the JVM sub-communityof Harmony) can simply read through this proposal and be able to make an enlightened decision about it. I think that each point would gain much from being presented along the motivation behind it. For example, would your OPEN proposal work with a bidirectional object layout, without incurring prohivitive performance costs? [Just asking, I didn't have time to read through all of it...] Of course, this is only an opinion. :-) Etienne Anton Luht wrote: Hello, I would like to try to draw attention to the OPEN proposal again. It was published about two weeks ago and produced a very small response in the community. This interface is very important, because if it is accepted, it will become a base of (many?) Harmony VMs. For example, one of the current limitations of OPEN interfaces is that Component Manager loads all components at startup and there's no possibility to change a component (for example, Garbage Collector) later. Is it OK for everyone? Maybe someone foresees problems with such approach? -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: So today Sun announced...
Stefano Mazzocchi wrote: Now, let's go back to let the code speak. +1 ;-) -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: Supporting working on a single module?
Hi Mark, Mark Hindess wrote: Can you not work on a single module now? Yes. But only by checking out everything. I am developing a API stubs project, which is a full stubs implementation of the Java 1.5 API. My objective was actually to allow for not needing an API implementation to compile code against the API. I was planning to use this, among other uses, for compiling SableVM's luni-kernel implementation. Maybe I could contribute the stubs to Harmony, and you could use them to achieve your goal? Note: By stubs, I mean, Java sources that, when compiled, produce .class files which export the specified API. Of course, they don't implement the functionality. You can have a look at the preliminary work: svn co svn://svn.sablevm.org/classlib-stubs/trunk/ Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [admin] Split the list (was RE: Split the list ?)
I agree with Geir. It's a good idea to keep classlib and vm people aware of what the others are doing. Sometimes, issues that might seem unrelated by classlib or vm people might actually affect the other group. I much prefer the suggested [topic] approach. Etienne Magnusson, Geir wrote: I've thought about it, but I think that it's a bit premature. Why? Because I think that once we fracture the list traffic, we lose a lot of the intermixing that's important. I think once we have a lot more VM traffic, it makes perfect sense. However, I think that as an intermediate step though we should all work very hard to tag our subject lines with a [topic] :) -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [classlib] String is special
Hi Tim, I have an objection. Locking has nothing to do with commit control. Allowing for svn locking functionality is opening a can of worms. What if somebody aquires a lock and loses network connectivity for a week (because of a Hurricane, because he forgot and went on vacation, etc.)? The whole svn philosophy is to allow for parrallel development, instead of the serialized development imposed by locking based repositories (Visual Source Safe, RCS, etc). I would sugget, instead, to put a BIG WARNING at the top of String.java, indicating that any change must first be approved on [EMAIL PROTECTED] I think that this would accomplish your goal in a more appropriate manner. Etienne Tim Ellison wrote: To ensure that all committers can continue to update String, but that they do so 'knowingly' (i.e. after considering the consequences) I'd like to impose a 'positive action' pre-commit step by setting the svn:needs-lock property on String.java. ... If there are no objections I'll go ahead and do this. -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [classlib] String is special
Tim Ellison wrote: Yes it does, your commit will fail if I have the lock; but I am abusing it to achieve a somewhat different goal of 'inadvertent modification'. OK, we both agree on the abuse part. ;-P If we cannot contact the lock holder we simply break/steal the lock. Right. But, locks should only be used for files which require serialized modifications, such as binary blobs. Other commit control should be made through pre-commit scripts. Not really. I can add the warning, but I was looking for a way to ensure people did not mistakenly change String or did not read the doc/dev list. By failing people's commit and making them explicitly acquire the token first they have to know what they are doing. So, you should investigate a pre-commit script based approach. For example, you could require a harmony:string-revision property on the String.java file. The pre-commit script would check that any commit increases this revision by one; if not, then an explicit (custom) error message can be given. Quite easy to achiev with svnlook, etc. What do you think? Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [classlib] String is special
Geir Magnusson Jr wrote: 3) Lock down function and behavior tightly with tests - that if modified, tests will break, and that should raise an alarm to the committer. One easy way would be to add a test that compares the String.java source code with a saved version, and that fails if unequal. This way, any change will trigger a test failure. To get past this failure, one will have to change both String.java and the saved test version. Just an idea... Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [jira] Commented: (HARMONY-290) Incorrect SerialVersionUID in java.lang.*
Hi Mikhail, I guess I'll have to open a new report for the unit tests, right? Etienne Mikhail Loenko (JIRA) wrote: [ http://issues.apache.org/jira/browse/HARMONY-290?page=comments#action_12374906 ] Mikhail Loenko commented on HARMONY-290: fixed in revision 394917 -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [jira] Commented: (HARMONY-290) Incorrect SerialVersionUID in java.lang.*
Of course. Etienne Geir Magnusson Jr wrote: to contribute them? -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [jira] Commented: (HARMONY-290) Incorrect SerialVersionUID in java.lang.*
Ah! OK. I should have looked more carefully. Thanks, Etienne Mikhail Loenko wrote: I did not close the issue, as you asked. -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: svn commit: r394890 - in /incubator/harmony/enhanced/classlib/trunk/modules: applet/make/common/ archive/make/common/ auth/make/common/ awt/make/common/ beans/make/common/ crypto/make/common/ jndi
+1 too. Mikhail Loenko wrote: +1 2006/4/18, Geir Magnusson Jr [EMAIL PROTECTED]: Wait - no tabs! no tabs! PLEASE! You can have your 1,743 character test case names if you want, but NO TABS! :) -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [classlib] unmodified generic GNU Classpath JVM can now run Classlib hello world
Hi Weldon, Weldon Washburn wrote: Question for SableVM/JCHEVM guys: Did I miss the documentation on lining up native method args? Can you point me to the correct place to figure out how to do this? There is nothing specific, as far as I remember. SableVM's native calling code takes care of everything, normally. Have you tried applying your layer to SableVM and see if it works, if it's really a generic layer? Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [vmi] JNI spec interpretation?
Geir Magnusson Jr wrote: Tim Ellison wrote: The IBM VME comes with a check utility that complains about bad practices detected in JNI code: Usage: -Xcheck:jni:[option[,option[,...]]] ... pedantic perform more thorough, but slower checks What a cool option. :-) This is very cool. I wonder if after some time when we get this under control, we should add to the standard test bed... Actually, I'm surprised that it's not already turned on while testing... Getting JNI usage right is not just a simple option; it's a requirement according to the JNI spec. Bad JNI usage leads to undefined behavior. SableVM has no (or minimal) protection against badly behaving JNI code. Of course, with time, we'll add robustness checks, but this takes time to implement. And then remains the question of how to deal with misbehaving code... I know: match the RI will be suggested. :-) Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [classlib] update on running Harmony Classlib on GNU Classpath VMs
Like Tim, I object to have GNU Classpath specific code in the Harmony kernel code. Etienne Tim Ellison wrote: Weldon Washburn wrote: I added a bunch of VMx.java files to kernel/src/main/java/java/lang/ These files are only needed for GNU Classpath compatible JVMs. Should we keep these files in this directory or should we create a separate directory? It might make sense to create kernel/src/gnuclasspathadapter/java/java/lang/VMx.java I would object to making the Harmony kernel GNU Classpath specific, so why not make the adapter a separate 'project', i.e. just gnuclasspathadapter/src/java/java/lang/VMx.java -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [sablevm] SIGSEGV signal received from Cygwin
Enrico Migliore wrote: Yet, I remember, that in JCHEVM I had to remove the POPT_AUTOHELP string in a struct in order not to receive that signal, therefore, I'm pretty much sure that somewhere in the Pthread library there's an access to an area used to initialize constants: something that maybe GCC likes but Windows doesn't. That's just a guess. I have to look more carfully to the port library, and see what thread supports it provides. I've had lots of professorish things to do over the last days, so I didn't get enough time to look deeply. I'll get back to you about it. The documentation included in SableVM trunk is more than good. Therefore, I would like to try do this job. I'm not sure I'll succeed but I want to As far as the port is concerned, there are 3 things to talk about: build system in MSVC --- to my knowledge, MSVC doesn't have a build system like UNIX (autotool + configure) therefore, I think that the config.h file will have to be built manually. This shouldn't be too much of a problem. SableVM has no intrinsict dependence on auto* tools. It does, however, require m4. This could be eliminated if we decided to allow for a Java dependency for building (i.e. ant sablecc-based code generator). Yet, on shorter term, some simpler make file is feasible. All you actually need to build is libsablevm.dll. There's no need for the sablevm launcher; we already know that the harmony one works, and it is much more complete... how to modify the source files? -- Let's say that removing a POSIX dependancy in a SableVM file means: replacing some lines of codes and replacing some include files. How am I suppost to deal with this kind of thing? Should I modify the trunk or my sandbox? Ideally, initial work should happen in your sandbox. This way you can use it to do backups of your code, even when it doesn't compile, without disturbing anybody. I am playing with the idea of making a copy of current trunk in sablevm/branches/last-stable, and simply allowing for a broken trunk during the Harmony transition (not as broken as a typical sandbox; should at least compile). We have a strict policy on using the svn merge operation. We can discuss it the day you need it. how to track source modifications -- Let's say that one source file gets modified by one of SableVM developers, and another is added to the trunk. Should I rely on SVN to know where the modifications were done? Not sure I understand the question... Developers may only checkin code they wrote themselves. This way, svn keeps a clear history of who wrote what. I think this discussion should probably continue on sablevm-devel@, as not to add too much noise into [EMAIL PROTECTED] We could, there, discuss of work distribution among you, hadrien and me, as well as any other interested contributor, to get a working sablevm+harmony duo as soon and as cleanly as possible. We should come back to this list for any topic related to the class-lib and port library side of things. Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [sablevm] SIGSEGV signal received from Cygwin
Enrico Migliore wrote: I'm doing some testing on SableVM, and noticed that is receives the very same segmentation fault signal that JCHEVM does, from pthread_key_create() which is embedded in: /usr/bin/cygwin1.dll I read around that this problem could be fixed, but the error means that the Cygwin platform can only be used for developing, studying and testing and not for real world use. Can you expand a little on this. I am not sure what you mean. problem 1: POSIX dependancy - The code contains a certain number of POSIX calls (dependancies). Much of them can be easily replaced, but some might be hard to replace. I think that moving to Harmony's port library should solve this. I agree that it shouldn't be hard to fix. problem 2: GCC extensions - The code contains some GCC extensions which are not ANSI. When compiling a switch interpreter (--with-treading=switch), there shouldn't be GCC extensions. The only exception is atomic operations which cannot be expressed in C. Yet, I've found an elegant solution to this: use Hans Boehm's atomic_ops library to get this code out of SableVM. See: http://sablevm.org/bugs/179 problem 3: Which compiler? - GCC seems to be the best candidate but MSVC is more popular. Ideally, I'd like both to work. Getting the faster direct/inlined interpreters to work with MSVC might be tricky (require inline assembly or linking to an asm library), but the switch interpreter shouldn't be a problem. problem 4: Native code dependancies --- The Harmony class library depends on the port layer: http://svn.apache.org/viewcvs.cgi/*checkout*/incubator/harmony/enhanced/classlib/trunk/doc/vm_doc/html/index.html Yep, using this layer and atomic_ops should hopefully be sufficient to remove all system-specific dependencies. If anything is missing, we should probably abstract it into the port layer. Now, the interesting thing would be for Harmony native code to compile and work on something other than ia32. SableVM already works on pretty much anything (using gcc, so far), as long as libffi and GNU classpath compiles on the target. The only other limitation is 2 operations that cannot be expressed in C: compare_and_swap and clear_cache. Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [sablevm] SIGSEGV signal received from Cygwin
Enrico Migliore wrote:
I debugged the classical HelloWorld class with DDD and found the problem
in the following function:
_svmf_init(void)
{
pthread_once(...); SEGSEGV signal
That's definitely a cygwin bug. I see.
problem 1: POSIX dependancy
-
...
I think that moving to Harmony's port library should solve this. I
agree that it shouldn't be hard to fix.
Are you referring to the work-in-progress that is adapting SableVM to
the VMI?
Yep.
problem 2: GCC extensions
-
...
this: use Hans Boehm's atomic_ops library to get this code out of
SableVM. See:
http://sablevm.org/bugs/179
Just FYI, fixing this is a short term objective. (i.e. work in progress).
In any case, before starting the port, I think that I and the people who
would like to help, will have analyze the code file by file.
Actually, you should really start looking at:
src/libsablevm/include/jni_system_specific.h
src/libsablevm/system.c
src/libsablevm/system.h
These are the files which contain system-specific code. Outside of
these files, the only real dependencies are POSIX calls. [I think
there's some exception in System.getCurrentMillis() implementation that
shouldn't even be in the VM to start with, as there's no VM-specific
functionality in it... I had to live with Classpath's decisions on
their VM interface.]
If you really want to read every single source file (!), then you should
definitely:
0.1) [prerequisite] read the JVMS fully, a few times over
0.2) [prerequisite] read the JNI spec fully, a few times over
1) read my Ph.D. thesis
2) read the documents in doc/
3) ask questions on sablevm-devel@ for clarifications
Going that deep shouldn't be necessary, though. Identifying POSIX
dependencies and replacing them with VMI-port calls should be sufficient
to start with.
Etienne
--
Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/
SableVM: http://www.sablevm.org/
SableCC: http://www.sablecc.org/
signature.asc
Description: OpenPGP digital signature
Re: New Harmony Launcher Plug-in Available For Eclipse
The link gives me: An Exception Has Occurred Python Traceback Traceback (most recent call last): File /usr/local/viewcvs-1.0-dev/lib/viewcvs.py, line 3351, in main request.run_viewcvs() ... George Harley wrote: split. Please point your Eclipse update manager at the following site and install version 1.0.1 ... http://svn.apache.org/viewcvs.cgi/*checkout*/incubator/harmony/enhanced/tools/trunk/eclipse/org.apache.harmony.eclipse.site -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [vmi] JNI spec interpretation?
By specification, the general contract for misbehavior of JNI code is undefined VM behavior, which can mean anything from blow up to corrupt internal data structures: 11.8.1 No Checking for Programming Errors The JNI functions do not check for programming errors. Passing illegal arguments to JNI functions results in undefined behavior. [...] Etienne Tim Ellison wrote: So what should a VM do if somebody calls ExceptionDescribe() and there is no pending exception? -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [classlib] Switching to a 5.0 compiler
I think that using Sun's javac and Eclipse with their undocumented and unsupported options during the transition period seems the cleanest strategy for both class library and VM maintenance. I see no trouble having such a dependency for a limited period of time (a few months). Etienne Geir Magnusson Jr wrote: Tim Ellison wrote: Using the Eclipse compiler: In the Eclipse JDT UI you *can* specify source=5.0 target=1.4 and I successfully ran code written using generics on a 1.4 VM. However, the Eclipse batch compiler does not allow this source=5.0 target=1.4 combination; and when I asked why the answer was that the UI usage is wrong and will be fixed to prevent that combination. LOL! How about suggesting that users find it useful right now? ... It seems like we have two options - Sun and Eclipse, and because the eclipse compiler is CDDL, and can be integrated into our build system, we seem to be safe. -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: SableVM on Cygwin
Hi Enrico, This is great! My hope would be, eventually, to have SableVM to also build using Microsoft's compiler. The switch threaded interpreter (./configure --with-threading=switch) avoids using any GCC extension in order to remain pure ISO C. My main problem is that I do not have a Windows machine, nor any Microsoft tool, and I do not have much time to dedicate to such a port. Harmony provides some platform insulation libraries for both the VM and the classlib. It would be interesting to see how far we can go by getting SableVM to use these libraries instead of POSIX calls... Volunteers are welcome. :-) Etienne Enrico Migliore wrote: I've just built SableVM and Classpath-0.20 on Cygwin without problems and the HelloWorld console application runs perfectly :-) While waiting for the VMI interface to be ready, I'm gonna do the following steps: 1. Try to enable the -no-cygwin GCC flag 2. Try to setup the mingw environment for building JCHEVM and SableVM without Cygwin 3. Make some experimentations with the help of DDD. -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
[sablevm] Signals under windows (Was: Starting my next round on BootJVM)
Enrico Migliore wrote: As far as I can say, the main problem of porting a JVM, designed for UNIX, to the Windows environment are the ANSI signals: Windows, in fact, doesn't honor not even a fourth of all ANSI signals, therefore, the JVM signals handler WILL NOT be called by Windows. You should try compiling SableVM with: ./configure --with-treading=switch --disable-signals-for-exceptions this will turn on ISO C compatibility and reduce (eliminate?) the use of signals. I think that the only remaining signal handling will be one provided for debugging: dumping a stack trace on SIGQUIT signals. Other POSIX dependencies might persist... See my message on a different thread. Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [classlib] Switching to a 5.0 compiler
Tim Ellison wrote: my point is that we could hack the Eclipse batch compiler to make it do something it is not meant to do (source=1.5 target=1.4). If we get burnt then we'll have nobody to complain to ;-) ... As it stands we have no option to use the Eclipse batch compiler (without tweaking it and therefore I assume bringing a modified JAR into our SVN for people to use etc. -- yuk!) Wouldn't the following be sufficient: 1- Put the (probably very small) patch in our repository. 2- Update make/depends.xml to fetch eclipse sources. 3- Update the build procedures to apply the patch and build/rebuild eclipse when necessary. = * Very small increase in repository * Longer *initial* build, but no change to next ones. * We have 2 compilers: Eclipse Sun = Geir's condition is satisfied :-) Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [vmi] JNI spec interpretation?
Oliver Deakin wrote: What I mean to say is that the behaviour when the function is called without a pending exception is unspecified, and in that case I think it makes most sense to match the RI. Let's say that I disagree to *impose* matching the RI's behavior for undefined JNI code on Harmony VMs. In many cases, matching the RI's behavior on undefined JNI would require reverse engineering way beyond a point I feel comfortable with. I definitely think that Harmony's native code should NEVER rely on undefined JNI behavior. JIRA reports should be raised for faulty JNI code. On the other hand, I think that it would be a nice thing to keep an explicit list of expected behavior for some widely used (by 3rd parties) undefined JNI, so that VM implementors are *encouraged* (not *forced*) to provide such workarounds. Some workarounds can be expensive to implement; we had we had to implement a more expensive approach for badly written JNI code that does not explicitly reserve enough local native references (only 16 are guaranteed by default in the spec). So, I'll add a the ExceptionDecribe workaround to SableVM permanently, but I do not wish feel obligated to do so. :-) Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [classlib/legal?] Re: Downloading dependencies
Leo Simons wrote: To have a dependency on external packages, we extend some trust to their authors/vendors to keep them available as open source... OK, I selected the wrong example. I am sure that you do understand that my worry is not a trust issue, but motly a convenience one and only a safety one in catastrophic cases [plane crash of development team, etc.]. The other one is about voting/democracy. I just wanted to mention it, I know, I know... :-) Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
[vmi] JNI spec interpretation?
Hi, I have started playing with the launcher and SableVM. I have a little fun trying to figure out how to debug the thing using DDD. The execv trick was a nasty one to get over... Anyway, now that it works, I am facing a JNI specification interpretation problem. It seems that the launcher assumes that it is OK to call (*env)-ExceptionDescribe() even when there is no pending exception. My reading of the JNI spec is different. The spec says: ... This function has the side effect of clearing *the* pending exception... This contrasts with ExceptionClear which says: ...Clears *any* pending exception... If no exception... Therefore, I always assumed that the necessarily was a pending exception. One cannot talk about *the* pending exception, if there is not necessarily one. Otherwise, the spec would have said: ...clearing any pending exception... What do others think? Should I file a bug against the launcher, i.e: native-src/shared/launcher/main.c line 1006 ? Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
[sablevm] Launching harmony/sablevm and debugging it with DDD
Hi all, So, I've temporarily modified SableVM's trunk to ignore the Harmony launcher JNI bug. You can now launch SableVM using the Harmony launcher. More interestingly, I have written down the precise instructions to get it all running in debug mode, in DDD, (a nice graphical front-end to gdb), under Linux/x86. This should help making SableVM's transition to Harmony much easier. Don't hesitate to try it, and maybe eventually to help with the VMI implementation task. :-) It's in a wiki. Feel free to improve the instructions. Oh, yes, THE LINK: http://sablevm.org/wiki/Harmony_Porting_Effort Have fun! Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Downloading dependencies
Hi, I've noticed that the build system tends to download external files. While I see some merit to this, I think that it is not a good approach in general, as it creates a very fragile build system in the long term. Imagine that some bug is discovered in Harmony 25.4. Regressions are applied against it and historical versions. Versions 24.x, 23.x, 22.x all exhibit the same bug. Then, ooops!, version 21.7 does not build as some library dependency can't be satisfied, the remote host has died/changed URL/etc. I can understand that we might not want to store dependency libraries in the repository (yet, I am not that sure I really understand...). What prevents us from storing them on the web site? (I guess the web site is already all in the repository anyway, so storing them in one branch or another doesn't make much difference other than the difficulty to checkout the system, then building w/o a network connection...) Unless we don't consider building historical versions as being important, or that a network connection is a hard dependency for building the system (a source CD wouldn't be sufficient to build)? Etienne PS: I know, I know... I've got a few other messages to reply to... Just give me some time. :-) -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: Downloading dependencies
Geir Magnusson Jr wrote: I don't consider being able to do an online build of any historical version that important. Because our releases will be archived, any dependencies included in the build will be archived on the mirror system, so any build should be reproducable. Mainly, I would be in favor of also storing the dependencies of the *head* trunk revision somewhere on the web site or ftp site, for practical reasons (simplifying the life of contributors to the trunk). Yet, as this is an old and settled subject, I won't push the discussion forward. I'll simply cast a modest retroactive -1 vote on this issue, for what it's worth. [Or a +1 vote on providing the dependencies of head trunk revision in a single location, somewhere outside of the repository] ;-) Cheers, Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
[classlib/legal?] Re: Downloading dependencies
Geir Magnusson Jr wrote: What are you voting against, exactly? I don't grok why this is a problem. You'll download the jars *once* (until a version changes), and with the exception of the MSVC runtime dll, it's automated for you. [Just explaining my position, not trying to change your mind]. I'm mainly voting against having our trunk dependent on remote (non-Apache) sites availability. For example, what if libXYZ-1.02.jar became unavailable tomorrow (assuming the trunk depended on it)? Maybe Harmony is the only project using it, and the author went bankrupt, or more likely, lost his site because he/she graduated from school, and that person wasn't aware Harmony used his/her open-source stuff. I'll let it to your imagination to come up with other likely scenarios. In such case, our trunk would break. I don't see how temporarily re-(?) distributing open-source stuff is a problem, but I'm not a lawyer, so I'll trust Apache's lawyers on this. Yet, I'll admit that the build system is dealing with this issue in a rather clean way, in that a separate ant file is used for the download, and the main build file does not depend on downloading from remote sites. And just as a note: I have no problem living in democracy, and not always being on the majority side. A few discussions here and there are key to coming up with better ideas, as long as discussions remain civilized. :-) Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: Patches to fix java.lang [Was: Fixing java.lang]
Thanks for the pointers and the guidance. Working on it. :-) Etienne Paulex Yang wrote: There are some helper classes on serialization test, which may make serialization testcases easier to create. hmmm..., in fact, *some* here means two, which come with contributions of different components from different ones, and people here haven't decided how to merge them:-(. One[1] is a abstract test case, which can be extended to create customized serialization test, the other[2] provides some util methods. IMO, both are handy, and you can choose any of them, and maybe provide some good suggestions on the merge issue. [1] http://svn.apache.org/viewcvs.cgi/incubator/harmony/enhanced/classlib/trunk/modules/security/test/common/unit/org/apache/harmony/security/test/SerializationTest.java?rev=378448view=markup [2] http://svn.apache.org/viewcvs.cgi/incubator/harmony/enhanced/classlib/trunk/support/src/test/java/tests/util/SerializationTester.java?rev=386058view=markup Tim Ellison wrote: It would be helpful if you could also include a regression test case to ensure that the serialized form is the same as the reference implementation. We do that by storing the serialized form of an object as written using the reference implementation. -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: java.lang.reflect
Tim Ellison wrote: Gary Clark wrote: 3) Do any of the current VMs support the 1.5 features? Off the top of my head I don't recall if anything changed within the VM to support them. I'll let Etienne, Archie, Dalibor, et al describe their status wrt Java 1.5 support. The IBM VM can switch to 1.5 support trivially but we are limiting it to 1.4 until there is reasonable coverage in the open source VMs. One of my former students succeeded to run SableVM with the generic branch of GNU Classpath over the holidays, by only implementing one or two additional native methods in SableVM (the code should be in his sandbox somewhere). As far as I know, the type erasure stuff was chosen by Sun as to minimize the impact of generics on the virtual machine. So, I don't expect the switch to 1.5 to be something big. But, first, it would be nice to get SableVM to work with current Harmony... So, I'm working on that for the time being. Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [classlib] splitting kernel in two
Tim Ellison wrote: Please ensure your student feels welcome to send comments and questions to the dev list. There are lots of people here who can help. I invited a student to present himself and send some question to this list. He'll probably do it soon. We have started playing around with the code and we obviously need a little additional explanations that those in the VMI docs. :-) Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [classlib/legal?] Re: Downloading dependencies
Geir, Actually, I didn't reply to your question. See below. Geir Magnusson Jr wrote: Ok. So are you against having the external dependency, or having an external dependency for which we don't have a jar in svn or the website? I'm against having an external dependency, in the HEAD trunk, for which we don't have a SOURCE jar in svn or the website. I assume that official releases do include all dependencies (most is probably more appropriate; we probably won't include the OS and base C libraries...). Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: SableVM or JCHEVM?
Hi Enrico, SableVM's trunk ( svn co svn://svn.sablevm.org/sablevm/trunk ) is now licensed under the Apache License 2.0. As SableVM is maintained by a number of developers, but is also used by many of my students to develop new VM components and do research, it was deemed more appropriate not to put SableVM in Harmony's repository for the following reasons: 1- Harmony's repository is not the appropriate place for creating many, many branches to do academic research on VMs. 2- Keeping a clear Intellectual Property trail would be a nightmare if SableVM's development and maintenance was spread over two distinct repositories. 3- According to Geir, integration of the VM into Harmony's repository is not a preriquisite for J2SE certification. The SableVM project has chosen to integrate our VM with Harmony's class library. In particular, I have started with a student to implement Harmony's VMI. Our objective is to fully adapt SableVM to work with Harmony's VMI as is. If I understand JCHEVM's approach, they are going in a different direction; they are developing a GNU Classpath compatibility layer so that Classpath-based VMs could migrate to Harmony with little effort. SableVM's goal is to become entirely dependent on Harmony's LUNI packages, as we do like Harmony's VMI. This will have the consequence of breaking SableVM's compatibility to Classpath's LUNI packages. Yet we do not see this as a problem; you will still be able to use Classpath other packages with SableVM (awt, swing, etc.). So, if you wish to contribute to SableVM for helping with the Harmony integration, please join us at http://sablevm.org. If you are interested, we'll use the SableVM developer mailing-list to coordonate the Harmony adaptation effort. See below. Enrico Migliore wrote: SableVM or JCHEVM? At the moment there two virutual machines: which of the two should one choose to work on? I would say: - If you want to work on a Classpath layer = JCHEVM - If you want to fully implement VMI = SableVM SableVM svn access - I can't find the SableVM repository in the Harmony site, can somebody address me where to find it? Contributing to SableVM is done in SableVM's repository at http://sablevm.org. I am in the process of updating our contribution policy to better reflect the recent license change. Yet, it retain much of the current policy: http://sablevm.org/svn/repository/sablevm/trunk/doc/contribution_policy.txt You're welcome to help! :-) Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Fixing java.lang
Hi, I was looking at the Japi output at: http://www.kaffe.org/~stuart/japi/htmlout/h-jdk12-harmony.html Is there any reason not to fix the 4 SerialVersionUIDs?: * class java.lang.IllegalArgumentException * class java.lang.IllegalStateException * class java.lang.SecurityException * class java.lang.UnsupportedOperationException I've got the feeling that I missed some discussion about this, and somebody will tell me that I'm clueless... Anyway, I can provide a patch for it. Would be my first little concrete contribution. :-) Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: SableVM or JCHEVM?
Hi Enrico, Enrico Migliore wrote: My ultimate goal, is contributing to the development of a reliable JVM which will run on the ARM platform, because I work in the embedded systems area. I don't care if such a goal will be achieved in 2 or 3 years, the only thing I care is not to waste my time in volunteering on a thing that may be abandoned. 2 things: 1- SableVM has, from the beginning, put portability as one of its main objectives. This is why we went all the way to write an inline-threaded interpreter engine, as to get as much speed without sacrificing portability and requiring knowledge of intimate details of the target platform (assembly language, ABI, etc.). We also kept easier to port interpreters (switch and direct threaded). The switch-threaded interpreter is a pure ISO C interpreter, does not emit a single gcc warning with -pedantic on. The other two interpreters require gcc extensions (and/or a little inline-assembly code). 2- The SableVM project will not be abandoned any time soon. It is one of my main research vehicles, and I am a relatively young tenured professor at UQAM. Unlike many other open source JVMs which are primarily the project of students, SableVM is a project driven by a paid professor. You should chose to work on the project that you feel most comfortable to contribute to. I don't know of any open-source VM that would reject contributions (at least, without a good reason). :-) Have fun! Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: Patches to fix java.lang [Was: Fixing java.lang]
Thanks, Nathan. I've opened HARMONY-290 in the bug database. :-) Etienne Nathan Beyer wrote: Here's what I do for patch submissions: ... -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [announce] New Apache Harmony Committers : Mikhail Loenko, George Harley, Stepan Mishura
Congrats! Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: EOL differences in svn diffs
Neat page! Could a link to it be added to the Harmony web page, so that new contributors, which are already overwhelmed by information, don't forget to read it? ;-) Etienne Leo Simons wrote: Please see http://www.apache.org/dev/, in particular -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [classlib] splitting kernel in two
Hi Nathan, Not sure what others think... Personally, I would highly recommend being very careful before (or when?) changing kernel classes, as changing them could lead to a very unstable VM interface if care isn't put into preserving stability. An unstable VMI is not something I would like to live with, as a VM developer. Unless I am wrong, IBM's VM works with the current class library as is (can somebody confirm/infirm this?). It would be sad if Harmony stopped working with IBM's VM, at least until it also worked with an open source VM. I am currently working with a student to get SableVM working with Harmony's VMI and kernel classes. We're just starting, and I have a busy schedule, so it would be difficult if the VMI kept changing under our feet over the next weeks (couple of months?)... Of course, we could try to get SableVM to work with a fixed historic Harmony version, but it would be nicer to get it working with the head revision in svn. Now, this being said, if all you want is simply to add additional constructors/methods, or make changes that do not impact the VM, such as adding erasable parametric types which result in fully binary compatible class files, then I have nothing to say about it. ;-) Just an opinion, of course. Have fun! Etienne Nathan Beyer wrote: No concerns here, but I do have a somewhat related question. What's the prescribed development model around the Java code in the kernel? For example, there are some additionally methods and constructors that were added to String in Java 5 (e.g. code point methods, StringBuilder constructor, etc) that I was thinking about addressing. Would I just make updates against the kernel module, just like any other module? What about the test cases? Assuming that's somewhat correct, if my understanding of the vm-specific concepts is correct, then there's no guarantee that any of the kernel-stub code is used, right? Put another way, can a VM just completely implement the kernel classes itself? One of the reasons I ask is because of some of the kernel classes, like String, have package-private (default) scoped methods that are used in LUNI by other java.lang classes. If this is a RTFM question, then feel free to point me back to the VMI documents with a scolding. -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: [classlib] ant platform property definitions
Isn't there some standard naming strategy, used by other Apache projects? I know that autoconf has a platform naming strategy. I recommend to study other platform naming strategies before making a decision. This will help reduce the number of mistakes. Etienne bootjvm wrote: Bottom line:Mixed case just adds one more level of complexity to the whole situation. Other comments below OS names: linux cygwin windows solaris CPU names: intel sparc ppc Word widths: 64 32 -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: EOL differences in svn diffs
The property works. I has been in use in the SableVM repository for a long time now. Actually, you need to set 2 properties: svn:mime-type : text/plain svn:eol-style : native Note that svn will give you trouble if you try to set these properties on a file which contains mixed EOL types. Etienne Nikolay Kuznetsov wrote: However, I tried to play with this switch on local machine (svn client/local svn server) and this had no effect. -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: SableVM and Harmony Class library
Up to now, SableVM has managed not to implement any class in C. Of course, one could question the efficiency of doing so, but hardly question the maintainability of it. :-) So, I do not think that step 2 is a hard requirement. (Methods, of course, can be native). Etienne Enrico Migliore wrote: step 2 The SableVM has to implement a Java interface, which means that is /has to/ implement in C a small set of Java classes, documented in the following link: -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
VMI Questions
Hi Weldon, I've started reading about the VMI. While my initial goal would be to get SableVM to work with Harmony as a drop-in replacement for IBM's VM, I have some questions about some assumptions of the VMI. In particular, I would like to understand the motivation for the implicit direct link from the hyluni library to the VM library. In other words, it seems logical to me that the VM library would cause the dynamic loading of the hyluni library. On the other hand, I don't understand why the hyluni library assumes that it will be necessarily loaded by a specific VM library sitting in a particular directory. This assumption has an important side effect. The hyluni library assumes that it can simply call VMI_* global functions which are resolved by the dynamic linker. Personally, I would have done things some other way; I would either have had the VM library call a VMI_SetVMI() function exported by the hyluni library and requested the the VM library call it, or, better, I would have done like the debug interface and requested the VMI function table through the GetEnv function of the invocation interface : (*jvm)-GetEnv(jvm, vmi, VMI_VERSION_1); This would seem the cleanest approach to me. The only problem is to avoid clashing with Sun's constants. The problem I see, with the current approach, is that it makes it impossible to provide distinct VM libraries for different purposes (debugging, faster execution, profiling, ...), as they can't be all in a single file. Etienne Weldon Washburn wrote: ... In any case, feel free to borrow parts of kernel_path that make sense. Also, feel free to ask questions about it. -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Re: VMI Questions
On the other hand, I don't
understand why the hyluni library assumes that it will be necessarily
loaded by a specific VM library sitting in a particular directory.
It doesn't. The hyluni library code uses macros that expand out to a
function table maintained by the VMStruct as you elude to below.
...
The only functions that are exported by the VM are [3]
VMInterface* JNICALL VMI_GetVMIFromJNIEnv(JNIEnv* env)
VMInterface* JNICALL VMI_GetVMIFromJavaVM(JavaVM* vm)
Exactly. Now, these 2 functions are global. How does the hyluni
library resolves these two global references? As far as I can see, in
the build/install process, the hyluni library simply assumes that the
system's dynamic linker will resolve them assuming they reside in a
specific file. See build.xml:
!-- Don't copy the VMI library. the one we build is a stub that is
replaced by a concrete implementation by the VM-implementor --
patternset excludes=*vmi*${shlib.suffix}* /
Unless I am missing something?
The problem I see, with the current approach, is that it makes it
impossible to provide distinct VM libraries for different purposes
(debugging, faster execution, profiling, ...), as they can't be all in a
single file.
We want that too, and I believe that it is already possible.
I am surely missing something...
Etienne
--
Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/
SableVM: http://www.sablevm.org/
SableCC: http://www.sablecc.org/
signature.asc
Description: OpenPGP digital signature
Re: VMI Questions
Ah! I think I see now. Thanks! :-) Etienne Tim Ellison wrote: Unless I am missing something? The actual implementation are provided by the VM in a DLL loaded by the Harmony launcher[1]. At launch time the DLL is loaded (and the VMI functions globally exported). -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
Portability of class libraries?
Hi all, What's the portability status of the native code of Harmony's class libraries? My primary SableVM development environment is a Debian/PowerPC machine. Of course I have a Debian/Pentium desktop somewhere, but I prefer working locally on my PowerBook. :-) Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
VM Interface Description?
I've found the following: http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200507.mbox/[EMAIL PROTECTED] Is there any other documentation on the VMI? The above email describes VM hooks; is there any other document describing the classlib hooks side of VMI? Thanks, Etienne -- Etienne M. Gagnon, Ph.D.http://www.info2.uqam.ca/~egagnon/ SableVM: http://www.sablevm.org/ SableCC: http://www.sablecc.org/ signature.asc Description: OpenPGP digital signature
