Re: New native layer
On Feb 4, 2006, at 12:53 PM, Guilhem Lavaux wrote: Hi Casey, Following my previous mail, please try this small performance test. It illustrates what I want to do and the performance "loss" that is expected. Simply unpack and run make. I wasn't saying that what you were doing would incur any performance penalty; I was just speculating about ways to write a native lib that supported user threads out of the box.
Re: New native layer
Hi Casey, Following my previous mail, please try this small performance test. It illustrates what I want to do and the performance "loss" that is expected. Simply unpack and run make. Cheers, Guilhem. Casey Marshall wrote: On Feb 2, 2006, at 10:27 AM, Guilhem Lavaux wrote: Casey Marshall wrote: On Jan 31, 2006, at 6:10 PM, David P Grove wrote: Jikes RVM also does m-to-n threading, so it's there's more than 1 VM that's whacky in this regard. The things we need to do are most likely different than what Kaffe needs to do, but having a chance to inject a VM callback before the thread dives off into a blocking system call is something we would like to be able to do. We have some linux specific hacks (evil with dlopen to intercept poll, select, etc), but it's fragile and doesn't work on other platforms like AIX and OS X that Jikes RVM runs on. Would my suggestion for Enter/Exit callbacks help this? Though enter/exit callbacks may help for a few syscalls but it won't for some (like blocking read/write). For these new calls you need handle blocking queues in the VM thread scheduler. That needs to put the fd into non-blocking mode and when you do a read you first check if there is any data (in non-blocking mode) and if there is not data then the thread system queues the current thread and tells the scheduler to jump somewhere else until some datas are available on the specified fd. So we must really stick to rerouting a few IOs. I am sure that handling m-to-n threading system is as difficult as what we are doing (maybe more). Aha, I grok the issue now. Would it make sense to do conditional compilation based on a "green threads" model, so that you can compile all blocking I/O calls to support such a model, if configured thusly? Or would file descriptor callbacks help -- where the FD is put into non-blocking mode before the call, then reset after, with support to catch the `EAGAIN' if it arrives? Also, I'm sure you could write a kernel-threads compatible library that also supported user-mode thread models at the same time (the restrictions on the former seem to be a subset of the latter), but don't know about what performance drain that would incur on kernel threaded systems. Because, "one POSIX library to rule them all," that supported both thread models, would still be an awesome thing. perf.tar.gz Description: Unix tar archive
Re: New native layer
Casey Marshall wrote: On Feb 2, 2006, at 10:27 AM, Guilhem Lavaux wrote: Casey Marshall wrote: On Jan 31, 2006, at 6:10 PM, David P Grove wrote: Jikes RVM also does m-to-n threading, so it's there's more than 1 VM that's whacky in this regard. The things we need to do are most likely different than what Kaffe needs to do, but having a chance to inject a VM callback before the thread dives off into a blocking system call is something we would like to be able to do. We have some linux specific hacks (evil with dlopen to intercept poll, select, etc), but it's fragile and doesn't work on other platforms like AIX and OS X that Jikes RVM runs on. Would my suggestion for Enter/Exit callbacks help this? Though enter/exit callbacks may help for a few syscalls but it won't for some (like blocking read/write). For these new calls you need handle blocking queues in the VM thread scheduler. That needs to put the fd into non-blocking mode and when you do a read you first check if there is any data (in non-blocking mode) and if there is not data then the thread system queues the current thread and tells the scheduler to jump somewhere else until some datas are available on the specified fd. So we must really stick to rerouting a few IOs. I am sure that handling m-to-n threading system is as difficult as what we are doing (maybe more). Aha, I grok the issue now. Would it make sense to do conditional compilation based on a "green threads" model, so that you can compile all blocking I/O calls to support such a model, if configured thusly? Or would file descriptor callbacks help -- where the FD is put into non-blocking mode before the call, then reset after, with support to catch the `EAGAIN' if it arrives? Hi Casey, You would continue to need some VM callbacks. Putting the FD in non-blocking mode is not sufficient though I could maybe imagine only rerouting select/poll in such occasions. You will also need the Enter/LeaveSyscall callback to ensure the thread safety of errors. To sum up, you can always invent a bunch of interfaces which can look more compact but they will give you a more obscure code. #if defined(GREEN_THREADS) #ifdef THIS_VARIANT_DOES_NOT_LIKE_THIS_SYSCALL #else #endif #ifdef... #endif #else /* ! GREEN_THREADS */ /* Do some other macro checking */ #endif Put in the perspective of what we already has around the code it will be a mess... Also, I'm sure you could write a kernel-threads compatible library that also supported user-mode thread models at the same time (the restrictions on the former seem to be a subset of the latter), but don't know about what performance drain that would incur on kernel threaded systems. Because, "one POSIX library to rule them all," that supported both thread models, would still be an awesome thing. Performance drain would a be one more indirection by syscall. I am not sure if it really matters as anyway syscalls are expected to be long compared to a usual function call. I can run some performance test on my computer though. The testcase would be: syscall.c: write some function to alias a syscall. I compile syscall.c with PIC to ensure the worst case. Then: main.c: Call this syscall an huge amount of time. Then compare to calling the real syscall. Regards, Guilhem.
Re: New native layer
On Feb 2, 2006, at 10:27 AM, Guilhem Lavaux wrote: Casey Marshall wrote: On Jan 31, 2006, at 6:10 PM, David P Grove wrote: Jikes RVM also does m-to-n threading, so it's there's more than 1 VM that's whacky in this regard. The things we need to do are most likely different than what Kaffe needs to do, but having a chance to inject a VM callback before the thread dives off into a blocking system call is something we would like to be able to do. We have some linux specific hacks (evil with dlopen to intercept poll, select, etc), but it's fragile and doesn't work on other platforms like AIX and OS X that Jikes RVM runs on. Would my suggestion for Enter/Exit callbacks help this? Though enter/exit callbacks may help for a few syscalls but it won't for some (like blocking read/write). For these new calls you need handle blocking queues in the VM thread scheduler. That needs to put the fd into non-blocking mode and when you do a read you first check if there is any data (in non-blocking mode) and if there is not data then the thread system queues the current thread and tells the scheduler to jump somewhere else until some datas are available on the specified fd. So we must really stick to rerouting a few IOs. I am sure that handling m-to-n threading system is as difficult as what we are doing (maybe more). Aha, I grok the issue now. Would it make sense to do conditional compilation based on a "green threads" model, so that you can compile all blocking I/O calls to support such a model, if configured thusly? Or would file descriptor callbacks help -- where the FD is put into non-blocking mode before the call, then reset after, with support to catch the `EAGAIN' if it arrives? Also, I'm sure you could write a kernel-threads compatible library that also supported user-mode thread models at the same time (the restrictions on the former seem to be a subset of the latter), but don't know about what performance drain that would incur on kernel threaded systems. Because, "one POSIX library to rule them all," that supported both thread models, would still be an awesome thing.
Re: New native layer
yep, that's pretty much the situation with m-n threading in Jikes RVM. For the blocking operations, we need to replace the entire call with something else. There might be some cases where the callbacks could be useful, but they wouldn't be a complete solution for us. --dave Guilhem Lavaux <[EMAIL PROTECTED]> wrote on 02/02/2006 01:27:28 PM: > > Though enter/exit callbacks may help for a few syscalls but it won't for > some (like blocking read/write). For these new calls you need handle > blocking queues in the VM thread scheduler. That needs to put the fd > into non-blocking mode and when you do a read you first check if there > is any data (in non-blocking mode) and if there is not data then the > thread system queues the current thread and tells the scheduler to jump > somewhere else until some datas are available on the specified fd. So we > must really stick to rerouting a few IOs. I am sure that handling m-to-n > threading system is as difficult as what we are doing (maybe more).
Re: New native layer
Casey Marshall wrote: On Jan 31, 2006, at 6:10 PM, David P Grove wrote: Jikes RVM also does m-to-n threading, so it's there's more than 1 VM that's whacky in this regard. The things we need to do are most likely different than what Kaffe needs to do, but having a chance to inject a VM callback before the thread dives off into a blocking system call is something we would like to be able to do. We have some linux specific hacks (evil with dlopen to intercept poll, select, etc), but it's fragile and doesn't work on other platforms like AIX and OS X that Jikes RVM runs on. Would my suggestion for Enter/Exit callbacks help this? Though enter/exit callbacks may help for a few syscalls but it won't for some (like blocking read/write). For these new calls you need handle blocking queues in the VM thread scheduler. That needs to put the fd into non-blocking mode and when you do a read you first check if there is any data (in non-blocking mode) and if there is not data then the thread system queues the current thread and tells the scheduler to jump somewhere else until some datas are available on the specified fd. So we must really stick to rerouting a few IOs. I am sure that handling m-to-n threading system is as difficult as what we are doing (maybe more). Regards, Guilhem.
Re: New native layer
On Jan 31, 2006, at 6:10 PM, David P Grove wrote: Jikes RVM also does m-to-n threading, so it's there's more than 1 VM that's whacky in this regard. The things we need to do are most likely different than what Kaffe needs to do, but having a chance to inject a VM callback before the thread dives off into a blocking system call is something we would like to be able to do. We have some linux specific hacks (evil with dlopen to intercept poll, select, etc), but it's fragile and doesn't work on other platforms like AIX and OS X that Jikes RVM runs on. Would my suggestion for Enter/Exit callbacks help this?
Re: New native layer
Jikes RVM also does m-to-n threading, so it's there's more than 1 VM that's whacky in this regard. The things we need to do are most likely different than what Kaffe needs to do, but having a chance to inject a VM callback before the thread dives off into a blocking system call is something we would like to be able to do. We have some linux specific hacks (evil with dlopen to intercept poll, select, etc), but it's fragile and doesn't work on other platforms like AIX and OS X that Jikes RVM runs on. --dave > A major issue is that for green threads (i.e. jthreads) in Kaffe, we'd > have to make sure that we can disable interrupts before invoking a > system method, and enable them before we exit again. If we don't do > that, than bad things can (and do ;) happen. > > That's a pretty Kaffe-specific need, though, so it may not make sense to > make room for it in GNU Classpath, unless it is accompanied by some > other, instant gratification, like looking very much like plain old > POSIX. In theory, if we're lucky, Kaffe may also be able to get by using > AspectC++[1] or something equivalently weird to weave in the stuff we > need for jthreads. In practice, I don't think anyone has done something > like that yet, though. ;)
Re: New native layer
On Jan 31, 2006, at 3:51 PM, Dalibor Topic wrote: On Tue, 2006-01-31 at 22:00 +0100, Roman Kennke wrote: Granted, Aicas is a commercial party and cannot publish every port. There is also Kaffe, which is 100% GPL and now also suggest something similar like the target layer, only somewhat nicer, and again it seems to turn out to be a fight against windmills. A major issue is that for green threads (i.e. jthreads) in Kaffe, we'd have to make sure that we can disable interrupts before invoking a system method, and enable them before we exit again. If we don't do that, than bad things can (and do ;) happen. That's a pretty Kaffe-specific need, though, so it may not make sense to make room for it in GNU Classpath, unless it is accompanied by some other, instant gratification, like looking very much like plain old POSIX. In theory, if we're lucky, Kaffe may also be able to get by using AspectC++[1] or something equivalently weird to weave in the stuff we need for jthreads. In practice, I don't think anyone has done something like that yet, though. ;) Would it help to have a utility function, say as a private function in the JNIEnv, that told the VM that the code was entering or just exited a possibly-blocking system call? Which, in the default implementation for other VMs or threading systems, would be a no-op? E.g.: void EnterSystemCall (JNIEnv *env, void *address); void ExitSystemCall (JNIEnv *env, void *address); This would (I think) be a good place for Kaffe to disable whatever interrupts it had to, and would still be a generic-enough mechanism for a VM to keep track of where the native code is. We would need to find all the right places to call this, but I don't think that's much harder than rewriting everything with wrapper functions, and it would be extensible -- anything added that Kaffe needed to tweak on before calling would be made safe, as long as the usage was consistent.
Re: New native layer
On Tue, 2006-01-31 at 22:00 +0100, Roman Kennke wrote: > I get the impression that there > is a general opposition against real portability concerns within the GNU > (Classpath?) project. Correct me if I am wrong. There certainly are > parties that have portability demands that go beyong posix. Yes. Many parties probably do, or will do eventually. It's in the nature of free software to get ported, somehow, to anything people want to run it on. ;) But I doubt there are many developers inside GNU Classpath as a whole who'd be interested in maintaining non-POSIX code, since most VMs either support only POSIX, POSIX + something special or only something special, so that the set of stuff that's got a fair chance of being communally maintained since it is useful to more than a single VM boils down to POSIX alone. That's pretty much the stuff that has a fair chance of getting love by diffeent VMs in the common tree. So if, theoretically, Kaffe was ported to run on top of Ruby[0] it would make most sense to keep the maintenance & the associated busywork for the Ruby port of GNU Classpath's native libraries in KaffeOnRuby, rather than trying to maintain it inside GNU Classpath. Same if JamVM was ported to 16-bit Minix 2 on Amiga: the necessary changes to the native layer would be too invasive to bother the POSIX-only VMs with, as they'd add some burden without necessary immediate gratification for posix-only VMs. Immediate gratification good, maintenance burden bad. :) I think the hard question is how to make the native layer stuff such that a) most vms can profit from it, in particular when starting from ground up and b) it is easy enough to embrace and extend for those cases where it does not fit well, without forcing one to wholesome fork the native layer, or to maintain the odd, single-VM cases inside GNU Classpath. The answer to a) seems to be POSIX for the reasons given above, and to b) seems to be something like Guilhem's work on the native layer. > Granted, > Aicas is a commercial party and cannot publish every port. There is also > Kaffe, which is 100% GPL and now also suggest something similar like the > target layer, only somewhat nicer, and again it seems to turn out to be > a fight against windmills. A major issue is that for green threads (i.e. jthreads) in Kaffe, we'd have to make sure that we can disable interrupts before invoking a system method, and enable them before we exit again. If we don't do that, than bad things can (and do ;) happen. That's a pretty Kaffe-specific need, though, so it may not make sense to make room for it in GNU Classpath, unless it is accompanied by some other, instant gratification, like looking very much like plain old POSIX. In theory, if we're lucky, Kaffe may also be able to get by using AspectC++[1] or something equivalently weird to weave in the stuff we need for jthreads. In practice, I don't think anyone has done something like that yet, though. ;) cheers, dalibor topic [0] Just imagine the potential for book sales! ;) [1] http://www.aspectc.org/Publications.6.0.html
Re: New native layer
Per Bothner wrote: Casey Marshall wrote: We have the responsibility, as contributors to a GNU project, to maintain the project for the GNU system. GNU is sorta-POSIX, as are a lot of other interesting platforms, and targeting them earns us, as free software contributors -- not necessarily other groups or companies that want to use Classpath -- a lot. I see these "native portability layers" as being counter to that goal, and they especially don't make sense given that there's no other free implementations for platforms other than what we are targeting. An alternative take with similar conclusion: There is a standard "native portability layer". It is called Posix. There are multiple implementations of this layer available for MS-Windows. Other platforms we are likely to support (such as OS-X) already support Posix. I.e. there is no need for an extra portability layer. Exactly my sentiments (as a bystander in this thread so far) The parts of POSIX that Classpath uses are pretty much exactly those parts that Classpath needs. So why not just implement that subset of POSIX on your obscure platform? As a side-benefit, lots of other stuff might compile and work on your platform then too. Another thing that bugs me is that it's extra work to create a new 'layer' for each platform. Instead, the autoconf system is much more efficient: once you implement a test for whatever non-POSIX thing, that test can work for all platforms that need it, now and in the future. The "layer" idea is too big a level of granularity to be porting at. We should be porting more like at the function call and macro level. -Archie __ Archie Cobbs *CTO, Awarix* http://www.awarix.com
Re: New native layer
On Jan 31, 2006, at 1:00 PM, Roman Kennke wrote: Hi Casey, Am Dienstag, den 31.01.2006, 12:37 -0800 schrieb Casey Marshall: On Jan 31, 2006, at 11:32 AM, Roman Kennke wrote: Hi Brian, hi list, Yea, I think the point for me would be to keep Classpath's java hackers out of the business of writing native code, and especially out of the business of porting native code for such common idioms as generic file operations, network operations, etc. BTW, Torsten, the man who first wrote the target native layer, mostly works on native code and porting of this to platforms you would not dream of. That is arrogant. I'm sure everyone here who hacks on Classpath has worked with interesting technologies, and are all great engineers, each in his own right. So let's stop measuring cocks here, OK? Sorry, I didn't want to sound arrogant or whatnot. I am only a bit upset about this braindead discussion again. I'm sure some of us do appreciate the discussion, and don't find it brain-dead in the least. You have your own goals, and have made up your mind about POSIX and autoconf, that is clear, so you can feel free to ignore this discussion. I get the impression that there is a general opposition against real portability concerns within the GNU (Classpath?) project. Correct me if I am wrong. I think you're mistaken, if you think it is mere opposition. We have, like Per said in another reply, a portability layer already, and it is POSIX. It's an established standard, and it is available on many free (and non-free) platforms today, all of which we desire our code to run on with the highest priority. To put it a little more bluntly, and to paraphrase David Wallace, we can really only love what we value. Portability for the sake of platforms we don't use, or even know the names of, is not something we find immediately valuable. There certainly are parties that have portability demands that go beyong posix. Granted, Aicas is a commercial party and cannot publish every port. There is also Kaffe, which is 100% GPL and now also suggest something similar like the target layer, only somewhat nicer, and again it seems to turn out to be a fight against windmills. I don't get the windmills reference, but I think I get your point. With Kaffe, it isn't completely clear to me what the issues are, and how a native portability layer solves them. But after all, that is what we have the VM interface for and I for myself don't want to discuss C issues anymore and instead concentrate on getting the VM interface into a good shape (it already is IMO, needs some tweaks though) and let people with different opinions on native code fork/implement their own stuff below the VM interface. Yes, that is the best idea, I think. I still want to see a nice POSIXy reference implementation, though, even if it is only for the selfish reason so I can run jamvm on my OS X and GNU/Linux machines to test out code.
Re: New native layer
Hi Casey, Am Dienstag, den 31.01.2006, 12:37 -0800 schrieb Casey Marshall: > On Jan 31, 2006, at 11:32 AM, Roman Kennke wrote: > > > Hi Brian, hi list, > > > >> Yea, I think the point for me would be to keep Classpath's java > >> hackers > >> out of the business of writing native code, and especially out of the > >> business of porting native code for such common idioms as generic > >> file > >> operations, network operations, etc. > > > > BTW, Torsten, the man who first wrote the target native layer, mostly > > works on native code and porting of this to platforms you would not > > dream of. > > That is arrogant. I'm sure everyone here who hacks on Classpath has > worked with interesting technologies, and are all great engineers, > each in his own right. > > So let's stop measuring cocks here, OK? Sorry, I didn't want to sound arrogant or whatnot. I am only a bit upset about this braindead discussion again. I get the impression that there is a general opposition against real portability concerns within the GNU (Classpath?) project. Correct me if I am wrong. There certainly are parties that have portability demands that go beyong posix. Granted, Aicas is a commercial party and cannot publish every port. There is also Kaffe, which is 100% GPL and now also suggest something similar like the target layer, only somewhat nicer, and again it seems to turn out to be a fight against windmills. But after all, that is what we have the VM interface for and I for myself don't want to discuss C issues anymore and instead concentrate on getting the VM interface into a good shape (it already is IMO, needs some tweaks though) and let people with different opinions on native code fork/implement their own stuff below the VM interface. Cheers, Roman signature.asc Description: Dies ist ein digital signierter Nachrichtenteil
Re: New native layer
Casey Marshall wrote: We have the responsibility, as contributors to a GNU project, to maintain the project for the GNU system. GNU is sorta-POSIX, as are a lot of other interesting platforms, and targeting them earns us, as free software contributors -- not necessarily other groups or companies that want to use Classpath -- a lot. I see these "native portability layers" as being counter to that goal, and they especially don't make sense given that there's no other free implementations for platforms other than what we are targeting. An alternative take with similar conclusion: There is a standard "native portability layer". It is called Posix. There are multiple implementations of this layer available for MS-Windows. Other platforms we are likely to support (such as OS-X) already support Posix. I.e. there is no need for an extra portability layer. That is not to say we cannot add hooks and conditional compilation in modest doses to support Windows and other non-Posix platforms. But any extra indirection that hurts performance on Posix (even trivially), or anything that makes the code harder to maintain and understand is generally inappropriate. -- --Per Bothner [EMAIL PROTECTED] http://per.bothner.com/
Re: New native layer
On Jan 31, 2006, at 11:32 AM, Roman Kennke wrote: Hi Brian, hi list, Yea, I think the point for me would be to keep Classpath's java hackers out of the business of writing native code, and especially out of the business of porting native code for such common idioms as generic file operations, network operations, etc. BTW, Torsten, the man who first wrote the target native layer, mostly works on native code and porting of this to platforms you would not dream of. That is arrogant. I'm sure everyone here who hacks on Classpath has worked with interesting technologies, and are all great engineers, each in his own right. So let's stop measuring cocks here, OK? He is hardly a 'Classpath java hacker'. If the world was so nice that posix and autoconf is the solution to everything, then such work would hardly be necessary. But this seems to be a little behind the horizon of the brave GNU world. Man, all this narrow-mindedness sets me up, I think I better go back to java hacking ;-) GNU Classpath is STILL a GNU project, and above all, we are trying to support free platforms, and are first of all concerned with supporting GNU (in it's GNU/Linux form). There is reams of evidence that targeting mostly POSIX systems, and handling the divergences with autoconf, produces a lot of software that is usable in a lot of places (witness, all the programs of the GNU project). Please, recognize that: - Macro-based portability layers like this are ugly and extremely hard to maintain and debug. - The benefits of such a thing never materialized anyway, because the only implementation available is for POSIX-like platforms. So it STILL relied on POSIX and autoconf. We have the responsibility, as contributors to a GNU project, to maintain the project for the GNU system. GNU is sorta-POSIX, as are a lot of other interesting platforms, and targeting them earns us, as free software contributors -- not necessarily other groups or companies that want to use Classpath -- a lot. I see these "native portability layers" as being counter to that goal, and they especially don't make sense given that there's no other free implementations for platforms other than what we are targeting. You can call that narrow-minded if you want, but we have to have goals and rules for the project, and they should be mandated by the larger project of which we are a part, which is GNU.
Re: New native layer
Hi Brian, hi list, > Yea, I think the point for me would be to keep Classpath's java hackers > out of the business of writing native code, and especially out of the > business of porting native code for such common idioms as generic file > operations, network operations, etc. BTW, Torsten, the man who first wrote the target native layer, mostly works on native code and porting of this to platforms you would not dream of. He is hardly a 'Classpath java hacker'. If the world was so nice that posix and autoconf is the solution to everything, then such work would hardly be necessary. But this seems to be a little behind the horizon of the brave GNU world. Man, all this narrow-mindedness sets me up, I think I better go back to java hacking ;-) Roman signature.asc Description: Dies ist ein digital signierter Nachrichtenteil
Re: New native layer
Brian Jones wrote: Dalibor Topic wrote: On Mon, 2006-01-30 at 12:20 -0500, Brian Jones wrote: It would be nice, I believe, to re-use libraries that have handled most of the "porting" and "wrapping" for you such as APR (http://apr.apache.org/), or NPR (http://www.mozilla.org/projects/nspr/) to platforms GNU Classpath might care to support. You might also find glib useful, http://developer.gnome.org/arch/gtk/glib.html. I believe that's the reason why neither APR nor NSPR (nor other similar projects) have seen wide spread acceptance, even though they have been around for many years now: they don't really pay off enough for people to rewrite their existing code bases to them. Okay, but I brought it up since the work Guilhem Lavaux is doing sounds an awful lot like a rewrite. Although technically I think you can just run the macro processor by hand to get the current real source code for say Linux and autoconfiscate it from there. Or you could go back a couple of years and look in native/ for the original autoconf/posix version of the libraries and move forward from there. See also what the OpenBSD developers did as one of the first things, once they 'forked' Apache web server 1.3.x after the Apache license change: rip out APR and replace its use by plain old posix functions, since those were more maintainable for them. Yea, I think the point for me would be to keep Classpath's java hackers out of the business of writing native code, and especially out of the business of porting native code for such common idioms as generic file operations, network operations, etc. Hi Brian, The code is not really a rewrite. Actually I am just moving out TARGET layer and replacing it with native function call with an API not so different than posix. My idea is that these function calls should be purely POSIX (so that classpath's hackers does not have deal with subtleties). Anyway, as I have already said it, seeing the amount of code to do pure administrations in the JNI world, it is easier to read autoconfiscated code which is completely outside the JNI code (again look at javanet). The advantage of this method is that some VMs may want to reroute these syscalls and it is easy to do so using this semi-abstract interface without rewriting anything. Regards, Guilhem.
Re: New native layer
Dalibor Topic wrote: On Mon, 2006-01-30 at 12:20 -0500, Brian Jones wrote: It would be nice, I believe, to re-use libraries that have handled most of the "porting" and "wrapping" for you such as APR (http://apr.apache.org/), or NPR (http://www.mozilla.org/projects/nspr/) to platforms GNU Classpath might care to support. You might also find glib useful, http://developer.gnome.org/arch/gtk/glib.html. I believe that's the reason why neither APR nor NSPR (nor other similar projects) have seen wide spread acceptance, even though they have been around for many years now: they don't really pay off enough for people to rewrite their existing code bases to them. Okay, but I brought it up since the work Guilhem Lavaux is doing sounds an awful lot like a rewrite. Although technically I think you can just run the macro processor by hand to get the current real source code for say Linux and autoconfiscate it from there. Or you could go back a couple of years and look in native/ for the original autoconf/posix version of the libraries and move forward from there. See also what the OpenBSD developers did as one of the first things, once they 'forked' Apache web server 1.3.x after the Apache license change: rip out APR and replace its use by plain old posix functions, since those were more maintainable for them. Yea, I think the point for me would be to keep Classpath's java hackers out of the business of writing native code, and especially out of the business of porting native code for such common idioms as generic file operations, network operations, etc. Anyway, that was my $0.02, Brian
Re: New native layer
On Mon, 2006-01-30 at 12:20 -0500, Brian Jones wrote: > It would be nice, I believe, to re-use libraries that have handled most > of the "porting" and "wrapping" for you such as APR > (http://apr.apache.org/), or NPR (http://www.mozilla.org/projects/nspr/) > to platforms GNU Classpath might care to support. You might also find > glib useful, http://developer.gnome.org/arch/gtk/glib.html. I think that's a nice idea in theory, but not so nice in practice. APR is not GPL compatible atm, though it may be next year. NSPR looks like it is GPL compatible right now, all files I peeked at are dual licensed with the GPL or tri-licensed with GPL and LGPL. For added fun, NSPR actually comes with its own tiny mark and sweep gc, a Java-like Monitor data type, and a bunch of other goodies. APR is being used by subversion and the Apache web server. NSPR is being used by evolution, openoffice & various mozilla projects. They seem to cover a similar range of platforms, thought NSPR also now covers RISC OS ;) Otoh, there is no lack of GPL-compatible utility libraries, from Glib, and GNU Common C++, to Qt's non-GUI classes & Boost. I'd have a hard time picking any of them, since they all seem to address problems larger than those we'd need addressing at the moment (we're expecting a largely POSIXy/glibc environment anyway, and don't have any ports to very weird environments yet in the CVS tree), while at the same time 'obfuscating' the code in ways similar to the current target layer. I believe that's the reason why neither APR nor NSPR (nor other similar projects) have seen wide spread acceptance, even though they have been around for many years now: they don't really pay off enough for people to rewrite their existing code bases to them. See also what the OpenBSD developers did as one of the first things, once they 'forked' Apache web server 1.3.x after the Apache license change: rip out APR and replace its use by plain old posix functions, since those were more maintainable for them. cheers, dalibor topic
Re: New native layer
Guilhem Lavaux wrote: I thought to have been already clear about that in the past (and with no answers !). Let's try to summarise my goal: * if we want something which is quite portable (but maybe not as portable as aicas portability layer) we must ensure some level of abstraction to hide how syscalls are really used. That way if we have to call different things in the OS (e.g. for more exotic platforms like windows) or to use the syscalls differently we will not be completely stuck by tons of autoconf code in the common VM layer. * some VMs (like us in kaffe) do like to be able to intercept most syscalls to ensure that syscall operations are atomic from the point of view of the threading system. Sometimes you even need to know when a syscall may block to acknowledge the VM to take wise decisions for the GC (I don't remember if it was about wait4 at that time). * autoconf code is a lot clearer if we can isolate the portions of code to be replaced. The native layer I am writing would like to give an answer these problems. The default code will just be a rough encapsulation of the native syscalls (though maybe not for recv/send/... which needs some polling operations). I think that in future we'll need another configure option for the VM to be able to specify a new adaptative native layer. At the moment, I have separated this code in another directory of the classpath tree (native/jni/native-lib). It would be nice, I believe, to re-use libraries that have handled most of the "porting" and "wrapping" for you such as APR (http://apr.apache.org/), or NPR (http://www.mozilla.org/projects/nspr/) to platforms GNU Classpath might care to support. You might also find glib useful, http://developer.gnome.org/arch/gtk/glib.html. If Apache's license and GPLv3 will be compatible then this is probably the way to go. People that care about platform specific optimizations, or code size, etc. could then go beat up the external library provider or write a replacement themselves. Brian
Re: New native layer
On Sun, 2006-01-29 at 15:17 +0100, Mark Wielaard wrote: > Hi, > >- Writing concise, portable (in the realm of POSIXy systems, at > > least) native implementations of VM* classes that require native > > support. > > Through autoconf and replacement functions where needed. That is how > libgcj (and kaffe) do it. I'd like to praise gnulib there a bit, since it makes some of that stuff really nice and convenient. It's a fairly new thing, though. cheers, dalibor topic
Re: New native layer
Hi, On Sat, 2006-01-28 at 17:02 -0800, Casey Marshall wrote: > On Jan 28, 2006, at 7:21 AM, Guilhem Lavaux wrote: > > I thought to have been already clear about that in the past (and > > with no answers !). > > Sorry, I meant to reply about what you were proposing, but forgot :-) Yeah, sorry. We all need little reminders from time to time. But showing up with real code does help getting replies :) > What you're working on seems a bit better than the target layer > stuff, in that you have real functions that the compiler/debugger can > tell you about, if there's an error. Agreed, it is definitely better then what we have now with the macros. > I don't, however, entirely see > the point of another funcall to wrap the syscalls in. It seems to me > as though efforts like this are trying to re-use the bits of JNI code > that wrap syscalls in the native parts of VM* classes, and not much > else. I do see some value because it makes it clear where the JNI part ends and where the platform specific part starts. But I am not sure putting all platform specific parts in separate files is always an improvement since most of it is actually pretty small and most of it seems not to be that different between platforms. Adding separate functions for things that might be different or that need VM specific hooks/callbacks is good though. I would just hate to see it overdone for small things that could just as well be in the same file or even function. > In my opinion, time would be better spent: > >- Writing utility functions that make JNI easier to deal with. That is what JCL and NSA were for (native/jni/classpath). They were supposed to be "safe wrappers" for some common things like throwing exceptions, conversion of jstrings and cstrings and generic RawData support that hides whether the platform is 32 or 64 bits. But those functions were never really worked out much, nor are they used very consistently in all our JNI code at the moment. >- Writing concise, portable (in the realm of POSIXy systems, at > least) native implementations of VM* classes that require native > support. Through autoconf and replacement functions where needed. That is how libgcj (and kaffe) do it. > > * if we want something which is quite portable (but maybe not as > > portable as aicas portability layer) we must ensure some level of > > abstraction to hide how syscalls are really used. That way if we > > have to call different things in the OS (e.g. for more exotic > > platforms like windows) or to use the syscalls differently we will > > not be completely stuck by tons of autoconf code in the common VM > > layer. > > It seems like a lot of the argument for this kind of native layer is > "we can port to Windows/some exotic platform," but frankly, I don't > see it. Windows (in my uninformed opinion ;-) is probably wildly > different enough such that writing a win32 `cpio_read' isn't much > less work than writing a win32 > `Java_gnu_java_nio_channels_FileChannelImpl_read__.' And since any > port like this is (AFAIK) merely hypothetical, I don't see the value. libgcj has a somewhat separate Windows implementation of some platform native code (see the nat*Win32.cc files in gnu/java/nio, gnu/java/net, java/lang and java/net in gcc/libjava). And they also support cygwin and mingw to a point (although I don't know whether the Posix or Win32 native implementation is used by either the cygwin or mingw target). All this can/should of course be wrapped behind our VMClass interface layer since that also helps making our library JNI/CNI/.NET/Oberon/etc agnostic. > I don't mind this proposal, and I think you should go ahead with it. > I still have my own opinions about how to write Classpath's native > library, so I may try my own experiment in another branch (unless, of > course, no-one really agrees with me, in which case I will just stop > critiquing other people's code). I think I agree with Casey. Please go ahead with your work Guilhem. We will try to be constructive in our criticism. Actually doing the work and providing patches is a very good way to proof your point that this is needed and makes things cleaner. Thanks for working on this. Cheers, Mark signature.asc Description: This is a digitally signed message part
Re: New native layer
On Jan 28, 2006, at 7:21 AM, Guilhem Lavaux wrote: Hi Mark, Mark Wielaard wrote: Hi Guilhem, On Sat, 2006-01-28 at 14:43 +0100, Guilhem Lavaux wrote: I would like to mention that I am developping/fine tuning the new native layer for classpath in a separate branch called "NATIVE- LAYER". If you have some time to give your impressions then fetch it and look into native/jni/native-lib, native/jni/java-io, ... the new layer itself is developped in native-lib. The other libraries are being adjusted currently... Thanks for doing this on a branch. That makes it less disruptive and easier for people to choose to test it out before we decide whether to incorporate the work on the trunk. But even for CVS branches you need to send patches to the patches mailinglist. That way people can follow your work and already read up and comment on the things they see that they like or dislike. And it prevents you from having worked on something very hard and then "suddenly" dropping a big patch. You also need to update the ChangeLog file on the branch to show the steps taken in creating the new code. Please use [native] in the subject when sending patches to the list. (Sorry to be pedantic about this, but we want CVS to be our shared development space, not someone personal playground.) Ok. There was no patch at the moment so I could get to a real point. Now is nearly the case. Let's begin the patch flood... And please do send a proper patch for the things you already checked in If you didn't create a branch-point cvs tag then the following should give you the initial diff: cvs diff -Nr classpath-0_20-release native Ok. I'll send three sets of patches to classpath-patches: one to remove target layer (obviously), one to add the foundations of the new layer, and another to show the adaptations to the already existing classpath's code. What are the objectives for this branch? How will it be similar/different from the work that Roman was doing? I thought to have been already clear about that in the past (and with no answers !). Sorry, I meant to reply about what you were proposing, but forgot :-) What you're working on seems a bit better than the target layer stuff, in that you have real functions that the compiler/debugger can tell you about, if there's an error. I don't, however, entirely see the point of another funcall to wrap the syscalls in. It seems to me as though efforts like this are trying to re-use the bits of JNI code that wrap syscalls in the native parts of VM* classes, and not much else. In my opinion, time would be better spent: - Writing utility functions that make JNI easier to deal with. - Writing concise, portable (in the realm of POSIXy systems, at least) native implementations of VM* classes that require native support. - Optimizing some functions for target systems (e.g., using epoll on Linux, and kqueue on FreeBSD, etc.). But on the whole, making library that is concise, easy-to-understand, and which works efficiently on free platforms. The point is, we want (at least I want) Classpath not just a *complete* class library, but also an extremely good one. I think trying to make system details generic like this accomplishes that. Let's try to summarise my goal: * if we want something which is quite portable (but maybe not as portable as aicas portability layer) we must ensure some level of abstraction to hide how syscalls are really used. That way if we have to call different things in the OS (e.g. for more exotic platforms like windows) or to use the syscalls differently we will not be completely stuck by tons of autoconf code in the common VM layer. It seems like a lot of the argument for this kind of native layer is "we can port to Windows/some exotic platform," but frankly, I don't see it. Windows (in my uninformed opinion ;-) is probably wildly different enough such that writing a win32 `cpio_read' isn't much less work than writing a win32 `Java_gnu_java_nio_channels_FileChannelImpl_read__.' And since any port like this is (AFAIK) merely hypothetical, I don't see the value. * some VMs (like us in kaffe) do like to be able to intercept most syscalls to ensure that syscall operations are atomic from the point of view of the threading system. Sometimes you even need to know when a syscall may block to acknowledge the VM to take wise decisions for the GC (I don't remember if it was about wait4 at that time). That's kind of interesting. Can you explain how this works a little more, or point me to the code where this happens? And why having the native-lib shim between JNI and the system helps this? * autoconf code is a lot clearer if we can isolate the portions of code to be replaced. This is true, to a degree. But in many places where APIs hardly diverge at all, having this separation makes less sense. That is, having a good system-level event notification mechan
Re: New native layer
Hi Mark, Mark Wielaard wrote: Hi Guilhem, On Sat, 2006-01-28 at 14:43 +0100, Guilhem Lavaux wrote: I would like to mention that I am developping/fine tuning the new native layer for classpath in a separate branch called "NATIVE-LAYER". If you have some time to give your impressions then fetch it and look into native/jni/native-lib, native/jni/java-io, ... the new layer itself is developped in native-lib. The other libraries are being adjusted currently... Thanks for doing this on a branch. That makes it less disruptive and easier for people to choose to test it out before we decide whether to incorporate the work on the trunk. But even for CVS branches you need to send patches to the patches mailinglist. That way people can follow your work and already read up and comment on the things they see that they like or dislike. And it prevents you from having worked on something very hard and then "suddenly" dropping a big patch. You also need to update the ChangeLog file on the branch to show the steps taken in creating the new code. Please use [native] in the subject when sending patches to the list. (Sorry to be pedantic about this, but we want CVS to be our shared development space, not someone personal playground.) Ok. There was no patch at the moment so I could get to a real point. Now is nearly the case. Let's begin the patch flood... And please do send a proper patch for the things you already checked in If you didn't create a branch-point cvs tag then the following should give you the initial diff: cvs diff -Nr classpath-0_20-release native Ok. I'll send three sets of patches to classpath-patches: one to remove target layer (obviously), one to add the foundations of the new layer, and another to show the adaptations to the already existing classpath's code. What are the objectives for this branch? How will it be similar/different from the work that Roman was doing? I thought to have been already clear about that in the past (and with no answers !). Let's try to summarise my goal: * if we want something which is quite portable (but maybe not as portable as aicas portability layer) we must ensure some level of abstraction to hide how syscalls are really used. That way if we have to call different things in the OS (e.g. for more exotic platforms like windows) or to use the syscalls differently we will not be completely stuck by tons of autoconf code in the common VM layer. * some VMs (like us in kaffe) do like to be able to intercept most syscalls to ensure that syscall operations are atomic from the point of view of the threading system. Sometimes you even need to know when a syscall may block to acknowledge the VM to take wise decisions for the GC (I don't remember if it was about wait4 at that time). * autoconf code is a lot clearer if we can isolate the portions of code to be replaced. The native layer I am writing would like to give an answer these problems. The default code will just be a rough encapsulation of the native syscalls (though maybe not for recv/send/... which needs some polling operations). I think that in future we'll need another configure option for the VM to be able to specify a new adaptative native layer. At the moment, I have separated this code in another directory of the classpath tree (native/jni/native-lib). Cheers, Guilhem.
Re: New native layer
Hi Guilhem, On Sat, 2006-01-28 at 14:43 +0100, Guilhem Lavaux wrote: > I would like to mention that I am developping/fine tuning the new native > layer for classpath in a separate branch called "NATIVE-LAYER". If you > have some time to give your impressions then fetch it and look into > native/jni/native-lib, native/jni/java-io, ... > > the new layer itself is developped in native-lib. The other libraries > are being adjusted currently... Thanks for doing this on a branch. That makes it less disruptive and easier for people to choose to test it out before we decide whether to incorporate the work on the trunk. But even for CVS branches you need to send patches to the patches mailinglist. That way people can follow your work and already read up and comment on the things they see that they like or dislike. And it prevents you from having worked on something very hard and then "suddenly" dropping a big patch. You also need to update the ChangeLog file on the branch to show the steps taken in creating the new code. Please use [native] in the subject when sending patches to the list. (Sorry to be pedantic about this, but we want CVS to be our shared development space, not someone personal playground.) And please do send a proper patch for the things you already checked in If you didn't create a branch-point cvs tag then the following should give you the initial diff: cvs diff -Nr classpath-0_20-release native What are the objectives for this branch? How will it be similar/different from the work that Roman was doing? Cheers, Mark signature.asc Description: This is a digitally signed message part
