Hello! I'm putting together an application that has many large 3D model files included as bundle resources. Most 3D file formats (and the one I'm using is no exception) are designed such that the data can be easily extracted and loaded into the GPU via simple mmap() and memcpy() calls. In other words, you mmap() the 3D model data, allocate a block of memory on the GPU and map it into the host computer's address space (via Vulkan or OpenGL or something similar), and then simply copy bytes from one map to the other. This is nice because it's efficient in terms of CPU use, and it's also efficient in terms of memory - you don't use up any precious JVM heap space to load resources. The average size of each 3D model is ~100mb, and there are quite a lot of them.
The problem: Files included as Java resources can't be mmap()d nicely because they're living inside jar/zip archives. My options seem to be: 1. Extract bundle resources to the filesystem. The resources can then be mapped via the FileChannel API. This works, but it's not great because it then means I'm consuming twice the storage space on the user's computer. With people moving to smaller SSD disks, disk space has become somewhat more of a precious resource than it had been previously. It also means that I have to check that resources still exist and have been extracted every time the application runs. 2. Try to load resources via the ModuleReader API: https://docs.oracle.com/javase/9/docs/api/java/lang/module/ModuleReader.html#read-java.lang.String- This *might* work, although I'm not sure. All of my JPMS modules are OSGi bundles, but not all of my OSGi bundles are JPMS modules. It's not clear if they would have to be or not. The spec for the read method also doesn't actually outright say that the implementation memory-maps resources, although the OpenJDK implementation does do this. Given that I specifically want the characteristics and resource usage profile of memory mapping, I'm not sure I want to rely on something that isn't specified. 3. Produce a new non-jar format for delivering classes and resources. It would be trivial to design a new container format that isn't based on the zip format and that could be memory-mapped efficiently, but it would be brutally difficult to get all of the tools to support it. Naturally, IDEs would not know how to read classes from the container. 4. Come up with a way to associate extra separate data files with a given OSGi bundle. This seems like it would be sort of doable, but I'm not sure how this would interact with things like the repository API and resolver. One thing I like about OSGi bundles is that a single file contains everything you need. Moving to a model where you suddenly need multiple files for a bundle instead of just a single bundle seems like it'd be going against the grain of the system somewhat. 5. Come up with a hack to append data to a jar file. The zip format has a lot of leeway when it comes to trailing or leading garbage. I believe people use this to produce "self-extracting exe archives" on Windows, for example: An exe file has a zip file appended, and can then behave as an executable on Windows, or can be unzipped correctly by tools on other platforms (ignoring the exe content). This seems unpleasant though, and it's probably not something I'd want to do. 6. Download extra resources at runtime. This has all the same benefits and drawbacks as option 1, with the extra drawback that it makes the application require network access and external infrastructure to host files. I feel like I can't be the first person to run into this. Has anyone else come up with a nice solution to this? -- Mark Raynsford | http://www.io7m.com
pgpwUReg1TkbL.pgp
Description: OpenPGP digital signature
_______________________________________________ OSGi Developer Mail List osgi-dev@mail.osgi.org https://mail.osgi.org/mailman/listinfo/osgi-dev
