Re: Finish moving java/* to modules/java

[Waldek Kozaczuk]

Nadav,

I have just sent 2 new patches - for osv source tree and another for apps -
that replace all my previous related patches and combine those change.

Regards,
Waldek

On Wed, Dec 7, 2016 at 1:11 AM, Waldek Kozaczuk 
wrote:

> Based on my further investigation java/cloudius produces a cloudius.jar
> that is uses by https://github.com/cloudius-systems/mgmt project (per
> module.py). Also all java/jni/*.cc files except monitor.cc (used by
> balloning) is only used to implement native method in  java/cloudius
> classes.
>
> Ideally both java/cloudius folder and most java/jni/*.cc should also be
> moved to mgmt project. On other hand  java/jni/monitor.* and
> java/jvm/java.cc can be moved to modules/java-base.
>
> Remaining java/jvm/* should stay where they are as they are necessary to
> link with core OSv image.
>
> Does it seam reasonable?
>
> Waldek
>
> PS. BTW do my latest patches I sent look correct to you - ([PATCH2] Moved java
> files from ./java folder to new modules/java-base and modules/java-tests]
> and [PATCH2] Added openjdk8-zulu-compact3-with-java-beans app and
> modified jdk apps to remove any duplication] ?
>
>
> On Monday, November 28, 2016 at 7:28:08 AM UTC-5, Waldek Kozaczuk wrote:
>>
>> Hi,
>>
>> I just submitted 2 patches that are addressing moving java stuff from
>> java/ to modules/java-*. There is still some work to do as far as moving
>> jolokia-agent and cloudius modules from java-base and however I am not sure
>> what is the role of each module listed below (apart from jolokia) to know
>> if it is optional or essential and what it really is:
>>
>> cat modules/java-base/module.py
>> from osv.modules import api
>>
>> api.require('fonts')
>> api.require('ca-certificates')
>> api.require('libz')
>> api.require('josvsym')
>> api.require('httpserver-jolokia-plugin')
>> api.require('httpserver-jvm-plugin')
>>
>> Also locally I have tried to remove all java c++ stuff from Makefile and
>> move it to modules/java-. However after I adjusted these 3 cc files
>> (core/mempool.cc, core/mmu.cc, libc/mman.cc) to point to the new location
>> of modules/java-base/jvm/jvm_balloon.hh I came across the linker errors:
>>
>> Building into build/release.x64
>>   GEN gen/include/osv/version.h
>>   LINK loader.elf
>> build/release.x64/bsd/porting/mmu.o: In function `vm_throttling_needed':
>> /home/wkozaczuk/projects/osv/bsd/porting/mmu.cc:41: undefined reference
>> to `memory::throttling_needed()'
>> build/release.x64/core/mmu.o: In function 
>> `mmu::jvm_balloon_vma::fault(unsigned
>> long, exception_frame*)':
>> /home/wkozaczuk/projects/osv/core/mmu.cc:1555: undefined reference to
>> `jvm_balloon_fault(std::shared_ptr, exception_frame*,
>> mmu::jvm_balloon_vma*)'
>> build/release.x64/core/mempool.o: In function `memory::on_alloc(unsigned
>> long)':
>> /home/wkozaczuk/projects/osv/core/mempool.cc:445: undefined reference to
>> `memory::jvm_balloon_adjust_memory(unsigned long)'
>> build/release.x64/core/mempool.o: In function
>> `memory::reclaimer::_do_reclaim()':
>> /home/wkozaczuk/projects/osv/core/mempool.cc:1031: undefined reference
>> to `jvm_balloon_voluntary_return()'
>> /home/wkozaczuk/projects/osv/core/mempool.cc:1011: undefined reference
>> to `memory::throttling_needed()'
>> build/release.x64/libc/mman.o: In function `mmap':
>> /home/wkozaczuk/projects/osv/libc/mman.cc:145: undefined reference to
>> `memory::return_jvm_heap(unsigned long)'
>> Makefile:1796: recipe for target 'build/release.x64/loader.elf' failed
>> make: *** [build/release.x64/loader.elf] Error 1
>>
>> It seems to be that building loader.elf relies on stuff in
>> java/jvm/jvm_balloon.o, java/jvm/java_api.o and java/jvm/jni_helpers.o.
>> Based on my very shallow understanding of what is going on it seems that
>> JVM ballooning logic is pretty entangled with the core of OSv memory
>> management logic. How easy is it to untangle that and make it somehow more
>> pluggable?
>>
>> It seems that moving compilation of java.cc to modules/java-base should
>> be easy but apart from that the rest does not seem trivial.
>>
>> Any further advice?
>>
>> Waldek
>>
>>
>> On Tue, Sep 20, 2016 at 8:43 AM, Nadav Har'El  wrote:
>>
>>>
>>> On Sun, Sep 4, 2016 at 7:55 AM, Waldek Kozaczuk 
>>> wrote:
>>>
 I believe at some point somebody started reorganizing java folder
 content to move it to modules/java and modules/java-tests but it was never
 completed.

>>>
>>> Indeed. The comment in commit df0fd92cc3a713f11874af369eb5f54c560d8b1f
>>> and "TODO" in Makefile make this intention clear.
>>>
>>>

 I am willing to finish this process but I would like a little bit of
 guidance as far as:
 - possible major challenges to such restructuring (somebody stated that
 changing Makefile might be most challenging)
 - elimination of possible dead code (for example so called ballooning
 was disabled in java.cc and there seems to be a lot of relevant C++ and

Re: Finish moving java/* to modules/java

[Waldek Kozaczuk]

Hi,

I just submitted 2 patches that are addressing moving java stuff from java/
to modules/java-*. There is still some work to do as far as moving
jolokia-agent and cloudius modules from java-base and however I am not sure
what is the role of each module listed below (apart from jolokia) to know
if it is optional or essential and what it really is:

cat modules/java-base/module.py
from osv.modules import api

api.require('fonts')
api.require('ca-certificates')
api.require('libz')
api.require('josvsym')
api.require('httpserver-jolokia-plugin')
api.require('httpserver-jvm-plugin')

Also locally I have tried to remove all java c++ stuff from Makefile and
move it to modules/java-. However after I adjusted these 3 cc files
(core/mempool.cc, core/mmu.cc, libc/mman.cc) to point to the new location
of modules/java-base/jvm/jvm_balloon.hh I came across the linker errors:

Building into build/release.x64
  GEN gen/include/osv/version.h
  LINK loader.elf
build/release.x64/bsd/porting/mmu.o: In function `vm_throttling_needed':
/home/wkozaczuk/projects/osv/bsd/porting/mmu.cc:41: undefined reference to
`memory::throttling_needed()'
build/release.x64/core/mmu.o: In function
`mmu::jvm_balloon_vma::fault(unsigned long, exception_frame*)':
/home/wkozaczuk/projects/osv/core/mmu.cc:1555: undefined reference to
`jvm_balloon_fault(std::shared_ptr, exception_frame*,
mmu::jvm_balloon_vma*)'
build/release.x64/core/mempool.o: In function `memory::on_alloc(unsigned
long)':
/home/wkozaczuk/projects/osv/core/mempool.cc:445: undefined reference to
`memory::jvm_balloon_adjust_memory(unsigned long)'
build/release.x64/core/mempool.o: In function
`memory::reclaimer::_do_reclaim()':
/home/wkozaczuk/projects/osv/core/mempool.cc:1031: undefined reference to
`jvm_balloon_voluntary_return()'
/home/wkozaczuk/projects/osv/core/mempool.cc:1011: undefined reference to
`memory::throttling_needed()'
build/release.x64/libc/mman.o: In function `mmap':
/home/wkozaczuk/projects/osv/libc/mman.cc:145: undefined reference to
`memory::return_jvm_heap(unsigned long)'
Makefile:1796: recipe for target 'build/release.x64/loader.elf' failed
make: *** [build/release.x64/loader.elf] Error 1

It seems to be that building loader.elf relies on stuff in
java/jvm/jvm_balloon.o, java/jvm/java_api.o and java/jvm/jni_helpers.o.
Based on my very shallow understanding of what is going on it seems that
JVM ballooning logic is pretty entangled with the core of OSv memory
management logic. How easy is it to untangle that and make it somehow more
pluggable?

It seems that moving compilation of java.cc to modules/java-base should be
easy but apart from that the rest does not seem trivial.

Any further advice?

Waldek


On Tue, Sep 20, 2016 at 8:43 AM, Nadav Har'El  wrote:

>
> On Sun, Sep 4, 2016 at 7:55 AM, Waldek Kozaczuk 
> wrote:
>
>> I believe at some point somebody started reorganizing java folder content
>> to move it to modules/java and modules/java-tests but it was never
>> completed.
>>
>
> Indeed. The comment in commit df0fd92cc3a713f11874af369eb5f54c560d8b1f
> and "TODO" in Makefile make this intention clear.
>
>
>>
>> I am willing to finish this process but I would like a little bit of
>> guidance as far as:
>> - possible major challenges to such restructuring (somebody stated that
>> changing Makefile might be most challenging)
>> - elimination of possible dead code (for example so called ballooning was
>> disabled in java.cc and there seems to be a lot of relevant C++ and Java
>> code to support it)
>> - what to avoid
>> - general guidelines
>>
>
> One thing we need to do is to move the java-targets, java, jdk, etc.,
> stuff out of the main Makefile and into modules/java/Makefile.
> The challenge here is to get the right compilation line with the OSv
> headers. You can take examples from other modules like modules/tests, which
> also uses OSv headers.
>
> Second thing it would be nice to figure out is if we can get rid of the
> "jdkbase" setting in Makefile and build/script. For example, perhaps the
> "java" module can set up a link in its own directory, and the modules
> requiring the java module can just use that link.
>
> After both changes, grepping for "java" or "jdk" should come up empty.
>
> Finally it would be nice to move the remaining stuff in java/ to
> modules/java.
>
> Commit d9a7eba6e3f0a92074b579c0c181663e6c1d5e46 also points out that the
> modules/java/Makefile currently runs "maven" to compile a lot of Java code
> which isn't actually needed by all Java applications but just with some
> specific modules (e.g., jolokia-agent); We could move some of this
> compilation to the modules that needed them. I was always too afraid of
> Maven to actually go ahead and do this :-)
>
> Finally, please do not remove the Balloon stuff. I just added a
> bug-tracker issue that one day we need to bring it back - it was a nice
> feature, and unique to OSv (and we even explained it in the OSv paper). So
> I'd like to continue