RE: [arch] How much of java.* and friends does Harmony need to write. Was: VM/Classlibrary interface
You are correct, but why take the performance and complexity hit to solve a non-existing problem? Regards, Jeroen -Original Message- From: Aaron Hamid [mailto:[EMAIL PROTECTED] Sent: Sunday, June 05, 2005 20:44 To: harmony-dev@incubator.apache.org Subject: Re: [arch] How much of java.* and friends does Harmony need to write. Was: VM/Classlibrary interface I actually had not considered this issue which would seem to warrant these classes living in the same package. But can not an equivalent solution be constructed such that the implementations of these public classes can hand the VM* classes references to internal structures (and vice versa) though well defined interfaces, instead of relying on visibility modifiers to allow the VM* objects to access private state of java.lang classes, thereby allowing the VM* objects to live in a separate package? Or am I misunderstanding your statement (or maybe that is just too cumbersome)? Aaron Jeroen Frijters wrote: You're missing the fact that moving these classes into another packages creates another problem that is much worse. Namely that you often need to access private state in the public classes, you can do that by living in the same package, that's why the VM* classes live in the same package as their public counterpart. Regards, Jeroen
Re: [arch] How much of java.* and friends does Harmony need to write. Was: VM/Classlibrary interface
Jeroen Frijters wrote: You are correct, but why take the performance and complexity hit to solve a non-existing problem? The simpler solution is to just use class loaders as a modularization mechanism. It is possible (since I have done it in my OSGi framework) to create a class loader that handles libraries packaged as JAR files such that some packages are externally visible and some are not. Thus, the VM objects could be in another package and all have public access modifiers, but they would only be accessible to classes inside the JAR file, not outside of it. Sure, it is not impossible to get access to the VM classes in this approach, but none of the proposed solutions offer this guarantee either. - richard Regards, Jeroen -Original Message- From: Aaron Hamid [mailto:[EMAIL PROTECTED] Sent: Sunday, June 05, 2005 20:44 To: harmony-dev@incubator.apache.org Subject: Re: [arch] How much of java.* and friends does Harmony need to write. Was: VM/Classlibrary interface I actually had not considered this issue which would seem to warrant these classes living in the same package. But can not an equivalent solution be constructed such that the implementations of these public classes can hand the VM* classes references to internal structures (and vice versa) though well defined interfaces, instead of relying on visibility modifiers to allow the VM* objects to access private state of java.lang classes, thereby allowing the VM* objects to live in a separate package? Or am I misunderstanding your statement (or maybe that is just too cumbersome)? Aaron Jeroen Frijters wrote: You're missing the fact that moving these classes into another packages creates another problem that is much worse. Namely that you often need to access private state in the public classes, you can do that by living in the same package, that's why the VM* classes live in the same package as their public counterpart. Regards, Jeroen
RE: [arch] How much of java.* and friends does Harmony need to write. Was: VM/Classlibrary interface
Richard S. Hall wrote: Jeroen Frijters wrote: You are correct, but why take the performance and complexity hit to solve a non-existing problem? The simpler solution is to just use class loaders as a modularization mechanism. No, that does not solve the problem of getting access to the internals of public classes. For example, java.lang.VMThread needs to access package private methods in java.lang.Thread. Regards, Jeroen
Re: [arch] How much of java.* and friends does Harmony need to write. Was: VM/Classlibrary interface
Jeroen Frijters wrote: No, that does not solve the problem of getting access to the internals of public classes. For example, java.lang.VMThread needs to access package private methods in java.lang.Thread. You are correct, if your design is such that you need private access both ways, then you are pretty much screwed. I would have thought that java.lang.Thread would have only needed access to java.lang.VMThread, not the other way around. C'est la vie. - richard
Re: [arch] How much of java.* and friends does Harmony need towrite. Was: VM/Classlibrary interface
From a user perspective, Jikes RVM uses classpath out of the box. (It builds with unmodified GNU classpath releases and classpath cvs head; we don't distribute GNU classpath with Jikes RVM). We do provide our own versions of a few classes in java.* that the build process selects instead of using the classpath ones, but if that's the definition of not using classpath out of the box, then I'm not sure anyone actually uses classpath out of the box. --dave You're a bit confused here. Of course the Classpath VM interface requires the VM to provide certain classes. How else would it work? That does not mean to say that classpath itself needs modification - jamvm Uses Classpath out of the box. - jikesRVM Does not. It did not use Classpath originally, and hasn't migrated fully to the Classpath VM interface. - sableVM (1.11.3) Uses Classpath out of the box. SableVM bundles it's own Classpath for practicality, and because they want some things slightly different. It can still use Classpath out of the box if you want that. Out of your three examples, two can use Classpath straight out of the box. Only JikesRVM can't, and it was, after all, developed for an entirely different class library. This is inconvenient for VM writers, as they must bundle or track the development of those classes in both libraries. Yes. Changes to the class library-VM interface are inconvenient. I think that this is why there was a suggestion to isolate all references in a separate deliverable (be it package, jar, bundle, whatever), That's what the VM interface is for. /Sven
Re: Other interesting papers and research
[EMAIL PROTECTED] wrote on 06/05/2005 10:48:29 PM: - The machine code concatinating technique consumes much memory. In my experience, generated machine code is about 10 times larger than the original instructions in Java bytecode. In the paper, the authors have not mentioned memory consumption of the technique. We cannot guess how much it is precisely, but it is possible to be a big drawback. Yes, we can say the same for the approach taking a baseline compiler instead of an interpreter (like Jikes RVM). Memory consumption of the baseline compiler of Jike RVM is very interesting. It's platform dependent of course, but on IA32 isn't too horrible. For example, running SPECjvm98 we see a 6.23x expansion from the Jikes RVM baseline compiler machine code bytes over bytecode bytes. One thing to note is that a threaded interpreter would see something like a 2-4x expansion over normal bytecodes when it converts from bytecodes to its internal form (arrays of function pointers). So, a 6x expansion is probably only roughly 2x worse than some interpreted systems would actually see in practice. You can get this data out of Jikes RVM for your platform/program with -X:vm:measureCompilation=true. --dave
Re: Other interesting papers and research
Hi Dave, From: David P Grove [EMAIL PROTECTED] [EMAIL PROTECTED] wrote on 06/05/2005 10:48:29 PM: - The machine code concatinating technique consumes much memory. In my experience, generated machine code is about 10 times larger than the original instructions in Java bytecode. In the paper, the authors have not mentioned memory consumption of the technique. We cannot guess how much it is precisely, but it is possible to be a big drawback. Yes, we can say the same for the approach taking a baseline compiler instead of an interpreter (like Jikes RVM). Memory consumption of the baseline compiler of Jike RVM is very interesting. It's platform dependent of course, but on IA32 isn't too horrible. For example, running SPECjvm98 we see a 6.23x expansion from the Jikes RVM baseline compiler machine code bytes over bytecode bytes. Thanks for giving us such an useful number. It looks reasonable. One thing to note is that a threaded interpreter would see something like a 2-4x expansion over normal bytecodes when it converts from bytecodes to its internal form (arrays of function pointers). Direct threading interpreters like JDK's one work on plain Java bytecode and they do not need to expand normal bytecode instructions. Such expansion may have been required if Java bytecode is not linear and rather a tree or other complicated form. Then, So, a 6x expansion is probably only roughly 2x worse than some interpreted systems would actually see in practice. We have to just say the baseline compiler of Jikes RVM generates 6x larger native code than the original bytecode instructions. For Java-written JVM, it seems to be natural to have a baseline compiler instead of an interpreter. It looks complicated to have an interpreter for a Java-written JVM. We hope that the architecture of a JVM (e.g. interpreter or baseline compiler) is independent of the language for implementing a certain part of JVM. But there seems to be an implication between them. Any comment? Kazuyuki Shudo[EMAIL PROTECTED] http://www.shudo.net/
Re: [arch] How much of java.* and friends does Harmony need to write. Was: VM/Classlibrary interface
Jeroen I think there are at least two perceived problems being voiced: * aversion to the idea of polluting a standard library distribution (e.g. Classpath) with VM-specific interface implementations. Although there have been assurances that such work is trivial and the interface is always-changing, it makes it harder to decouple the VM from the class library (although another opinion is that this is not a goal of Harmony anyway).The fact that all (?) VMs using the Classpath library must make modifications to Classpath itself is an indication that perhaps standardization on interfaces (not implementations of course) would be beneficial. * uncomfortableness with relying on language visibility modifiers alone to prevent user code access to these internal APIs (this I am personally not that concerned about) Not to be pedantic - but for purposes of clarification can we quantify the performance and complexity hit for using well-defined interfaces as opposed to carnal knowledge between the class library and VM interface implementations? From what I have understood, for example, of the JikesRVM, these sorts of things are candidates for some heavy optimizations - especially given that these interfaces should not be used by user code at all, and should probably not be reloaded/unloaded/subject to traditional class loading issues (perhaps use some modularization, annotation, or magic keyword syntax that allows aggressive optimization of otherwise public interfaces?). Aaron Jeroen Frijters wrote: You are correct, but why take the performance and complexity hit to solve a non-existing problem? Regards, Jeroen -Original Message- From: Aaron Hamid [mailto:[EMAIL PROTECTED] Sent: Sunday, June 05, 2005 20:44 To: harmony-dev@incubator.apache.org Subject: Re: [arch] How much of java.* and friends does Harmony need to write. Was: VM/Classlibrary interface I actually had not considered this issue which would seem to warrant these classes living in the same package. But can not an equivalent solution be constructed such that the implementations of these public classes can hand the VM* classes references to internal structures (and vice versa) though well defined interfaces, instead of relying on visibility modifiers to allow the VM* objects to access private state of java.lang classes, thereby allowing the VM* objects to live in a separate package? Or am I misunderstanding your statement (or maybe that is just too cumbersome)? Aaron Jeroen Frijters wrote: You're missing the fact that moving these classes into another packages creates another problem that is much worse. Namely that you often need to access private state in the public classes, you can do that by living in the same package, that's why the VM* classes live in the same package as their public counterpart. Regards, Jeroen
Re: [arch] How much of java.* and friends does Harmony need towrite. Was: VM/Classlibrary interface
Gah. :( So if I am to understand this correctly: Classpath java.lang.* implementation does not rely on specifics of any given VM* interface implementation, but said VM* interface implementations MAY rely on internals of existing Classpath java.lang.* classes? (so there is a one-way dependency from the VM* implementation to Classpath but not vice-versa, hence the out of the box claims?) And through deduction, standardizing this VM* interface would also entail standardizing reverse access to the class lib java.lang.* implementations (either through some informal agreement on package visibility members *yuck*, or through a more sophisticated, but more cumbersome, API)? Aaron Jeroen Frijters wrote: You're missing the fact that moving these classes into another packages creates another problem that is much worse. Namely that you often need to access private state in the public classes, you can do that by living in the same package, that's why the VM* classes live in the same package as their public counterpart. Sven de Marothy wrote: On Mon, 2005-06-06 at 00:26 +0200, Santiago Gala wrote: A few classes need to be modified: You're a bit confused here. Of course the Classpath VM interface requires the VM to provide certain classes. How else would it work? That does not mean to say that classpath itself needs modification
Re: [arch] VM Interface
On 6/6/05, Peter Donald [EMAIL PROTECTED] wrote: The reason being that their customers do not want to be exposed to differences between rt.jar and GNU Classpath. oh well aren't both implemented according to a well-designed exported API. So how there would be differences that would hurt the clients of this API (assuming that the GUN Classpath got completed and they are both might be retrofitted to be easily installed in this modular architecture even if they depend on some native libraries). regards, ahmed
Re: Other interesting papers and research
Hi, On 6/6/05, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: Hi Dave, From: David P Grove [EMAIL PROTECTED] One thing to note is that a threaded interpreter would see something like a 2-4x expansion over normal bytecodes when it converts from bytecodes to its internal form (arrays of function pointers). Direct threading interpreters like JDK's one work on plain Java bytecode and they do not need to expand normal bytecode instructions. Such expansion may have been required if Java bytecode is not linear and rather a tree or other complicated form. According to my understanding, an indirect threaded interpreter uses the original bytecode stream. It's indirect because the handler address must be looked up via the bytecode. A direct threaded interpreter removes this step by placing the actual handler addresses in the rewritten instruction stream. For what it's worth, JamVM supports both direct and indirect threading, with static and dynamic stack-caching respectively. I seem to recall an average code size increase of ~4x for JamVM's internal instruction format but I'll need to recheck my figures to be sure. Note, this is assuming a 32-bit architecture. Handler addresses will be double on a 64-bit machine, and the code increase over bytecodes therefore larger. Rob.
Re: Other interesting papers and research
Hi Rob, From: Robert Lougher [EMAIL PROTECTED] Date: Mon, 6 Jun 2005 14:58:45 +0100 One thing to note is that a threaded interpreter would see something like a 2-4x expansion over normal bytecodes when it converts from bytecodes to its internal form (arrays of function pointers). Direct threading interpreters like JDK's one work on plain Java bytecode and they do not need to expand normal bytecode instructions. Such expansion may have been required if Java bytecode is not linear and rather a tree or other complicated form. According to my understanding, an indirect threaded interpreter uses the original bytecode stream. It's indirect because the handler address must be looked up via the bytecode. Ah, thanks for the indication. My wording 'direct threading' was not correct. Threading (interpreting) techniques I referred as implemented in JDKs should be called 'token threading', neither direct nor indirect threading because they work directly on bytecode instructions withought any expansion. Note that the interpreter provides NEXT routines to for all native code fragments corresponding to VM instructions. For JVM, this wording like something threading is not very informative because direct interpretation of portable bytecode is naturally 'token threading'. Dave's last posting was based on direct threading technique and his saying was correct about direct threading but my posting was incorrect in advance. Kazuyuki Shudo[EMAIL PROTECTED] http://www.shudo.net/
Re: [arch] VM Interface
Peter Donald wrote: I assume that if the Harmony JVM gets half as good as is hoped there will be companys who want to adopt the JVM but continue to use Suns class library so that differences in libraries don't hurt their customers. If that is a goal of Harmony then we've just made things a lot harder. First of all, Sun's class library - VM interface is proprietary and unpublished. How would people become experts in it without studying the Sun source code, with all the potential legal problems that entails? Secondly, you can no longer use Classpath as is, so Harmony will have to create a new fork of the Classpath code. Lots of work, zero forward progress. Thirdly, what's to stop Sun from changing things around every release? Their API is not standardized in any way. It involves sun.* classes, etc. On the other hand, if down the road the various interested parties got together and said, Let's all agree on a common class library/JVM API then certainly Harmony should be involved and supportive. However somehow to me that seems about as likely as Toyota, Ford, and GM all agreeing to standardize the connection between engines and gearboxes. -Archie __ Archie Cobbs *CTO, Awarix* http://www.awarix.com
Re: [arch] How much of java.* and friends does Harmony need towrite. Was: VM/Classlibrary interface
On Mon, 2005-06-06 at 10:01 -0400, Aaron Hamid wrote: Gah. :( So if I am to understand this correctly: Classpath java.lang.* implementation does not rely on specifics of any given VM* interface implementation, but said VM* interface implementations MAY rely on internals of existing Classpath java.lang.* classes? (so there is a one-way dependency from the VM* implementation to Classpath but not vice-versa, hence the out of the box claims?) Please read: Classpath VM integration guide. http://www.gnu.org/software/classpath/docs/vmintegration.html#SEC3 See also the reference implementation of the VM classes Classpath requires: http://savannah.gnu.org/cgi-bin/viewcvs/classpath/classpath/vm/reference/java/lang/ /Sven
[arch] Modules and interfaces (was: How much of java.* ...)
Geir Magnusson Jr. wrote: Doesn't this imply that the GNU Classpath interface should add a second API that *it* should comply with for symmetry? That way you don't get dependencies on GNU Classpath internals? I've been a bystander in this discussion as I know very little about the class library issues. There were obviously a lot of concerns being discussed in this thread, but I'd just like to respond briefly to the above... It brings to mind some of the portability/modularity issues we've been wrestling with (at great length) with MMTk and the interdependence between the memory manager and the VM. Our solution is not earth shattering, but it evolved out of a very long struggle with issues like the one Geir is alluding to above. In the end, our interface is not in the form of a simple API, but of reciprocal packages implemented on the MM and VM sides of the fence. So we have org.mmtk.vm, which captures all VM-specific dependencies, leaving the rest of MMTk's packages (the bulk of the code) strictly VM-neutral. We have a generic template implementation of the org.mmtk.vm package which serves to define the interface, and against which a VM-neutral mmtk.jar is built. Each VM then provides a VM-specific implementation of this package which binds into that VM's services (such as the way that VM identifies roots for collection, supports mmap(), defines the size of an address, or whatever...). At this stage we only have one example in cvs, but there are two others in various states of development (jnode, which is actively being developed Rotor which is a little out of date right now). http://cvs.sourceforge.net/viewcvs.py/jikesrvm/MMTk/ext/vm/ (the stub directory defines the package abstractly, the JikesRVM directory has the Jikes RVM - specific implementation) We want this to be symmetric, so that the VM has a similar arrangement whereby it can support various memory managers by having each of them implement some package. We have not yet cleaned up this aspect of Jikes RVM, but it is on our short list of planned cleanups. MOre generally, we want to use this model in a complete componentization of Jikes RVM. --Steve
Standard Bibliography for Java
A useful standard bibligraphy for many Java topics: http://www.topcoder.com/pl/?module=Staticd1=javaone04d2=book_titles including all your favorites. Dan Lydick
Re: [arch] VM Interface
Hi, Ahmed Saad wrote: oh well aren't both implemented according to a well-designed exported API. So how there would be differences that would hurt the clients of this API (assuming that the GUN Classpath got completed and they are both might be retrofitted to be easily installed in this modular architecture even if they depend on some native libraries). You are right but I am not sure the classlibrary has been specified well enough for that level of compatibility to occur. Even if it was specified well enough every implementation has bugs and sometimes application code ends up relying on these bugs or working around these bugs (with or without the developers awareness). Consider something like xsl and its support in java. The specification has always been realtively complete but in many cases certain xsl sheets will only work with one xsl processor or may only work with acceptable performance in one xsl processor. The classlibrary is even more complex than this. Cheers, Peter Donald
Re: The Classpath VM interface. (Please read)
Hi, Consider the following class libraries: [A] Suns rt.jar and derivatives (such as IBMs) class libraries [B] GNU Classpaths class libraries [A] has a proprietary VM interface that is not publicly documented and some people fear looking at the implementation because of possible taint (IMHO with the new license this is less of an issue). [B] has an interface that is relatively well documented but incomplete. AFAIK no one has done any investigation to see if [A] could be retrofitted to use [B]'s VM interface with minimal changes. Both [A] and [B] * can arbitrarily change between versions. * have been integrated into multiple JVMs In an ideal world Harmony VM would be able to use either [A] or [B] with a small adapter layer. Much like MMTk can be used in multiple VMs with a small adapter layer. It seems unlikely that either [A] or [B] is going to invest the time in trying to develop a common VM interface because they are not interested in facilitating reuse of the alternative. So it is probably going to be up to a third party like Harmony to investigate a common VM layer. Personally I suspect that it is going to be a LOT of work and even then I am not sure it is going to be possible either technically or politically but it is an interesting idea and worth trying. -- Cheers, Peter Donald Nothing changes your opinion of a friend so surely as success - yours or his. - Franklin P. Jones
Re: The Classpath VM interface. (Please read)
On Tue, 2005-06-07 at 12:36 +1000, Peter Donald wrote: [A] Suns rt.jar and derivatives (such as IBMs) class libraries [B] GNU Classpaths class libraries [..] In an ideal world Harmony VM would be able to use either [A] or [B] with a small adapter layer. Much like MMTk can be used in multiple VMs with a small adapter layer. If you have downloaded Harmony, which intends to be a full JDK including a VM and class library, why would you want to be able to use that with the class library from a different JDK? Disregarding the illegality of distributing such a combo, there is no good practical reason for wanting that either. It seems unlikely that either [A] or [B] is going to invest the time in trying to develop a common VM interface because they are not interested in facilitating reuse of the alternative. Java specifications are created by the JCP and not Sun. This issue should be raised there before jumping to conclusions like that. And as a Classpath developer (but speaking for myself), I feel that we would be quite happy to use a common VM interface, if there was a such a specification. Unless of course if it was so extremely bad, that all the Classpath-using VMs refused to use it. Which I think is unlikely. So it is probably going to be up to a third party like Harmony to investigate a common VM layer. Harmony is not a third party. We're all in this together. However, Harmony can help solve this. Apache has a good standing in the JCP, and thus better chances than most for getting this specified. I am not sure it is going to be possible either technically or politically but it is an interesting idea and worth trying. The politics are easier when you work from an open mind. /Sven
