I also think there are two audiences in this context we need to take into account:
- OSv developers - people that build apps just to test stuff; typically experienced with OSv build system and its limitations -> can find their way around - OSv users - people that build apps to deploy apps to run on OSv - should not need to understand OSv build system to deploy their apps The suggestions I have listed mostly pertain to first group. And I agree we should just tune what already works and document it. The second group's need should be addressed by Capstan. So we should avoid duplication between what Capstan does well (and hopefully will do even better in future) and OSv build system. Now capstan packages are often generated using osv-apps and OSv build system. I must say I love Nadav's idea of having a way to generate CMakefile.txt. I will try to experiment with it. Waldek On Sunday, March 31, 2019 at 12:48:38 PM UTC-4, Geraldo Netto wrote: > > Hi! > > I might be saying some nonsense, but... > > On Sun, 31 Mar 2019 at 17:25, Nadav Har'El <[email protected] > <javascript:>> wrote: > >> n Sun, Mar 31, 2019 at 5:58 PM Waldek Kozaczuk <[email protected] >> <javascript:>> wrote: >> >>> Here is a list of things we should try to do: >>> >> >> I think there are two very different issues involved here, that don't >> really need the same solution: >> The OSv kernel's makefile, and the "apps" build system. >> >> >>> - Propose and define model app layout (Makefile, GET, ...) under >>> osv-apps >>> - Typically each app has GET shell script, Makefile and module.py: >>> - GET typically downloads ad puts source files into upstream >>> directory (sometimes using latest script) >>> - Makefile typically does what needs to be done (sometime >>> compiles) and puts source files into install subdirectory >>> (sometimes >>> usr.manifest is generated, sometime usr.manifest is pre-defined). >>> - module.py is called to execute extra logic - sometimes to >>> generate usr.manifest, define cmdline and possibly multiple >>> flavors >>> - This is somewhat documented in the README.md of osv-apps >>> project but is not clear enough and the apps are all over the place >>> following this loose convention and I am myself guilty of not >>> following it >>> (see python2x/3x apps that do too much in GET). I doubt it is worth >>> fixing >>> existing apps but I think we could improve the documentation in this >>> area >>> and and possibly point to a good model app. >>> >>> I agree. Also, note that both the "GET" and "makefile" there are rather >> arbitrary conventions - >> all we need from the Makefile is to have the "make" and "make clean" >> target. That's it. Because people >> were too lazy to write a proper makefile around this requirement, "make" >> just calls "GET" which is a >> shell script that just does everything. >> We could easily have any other convention to build and clean a package. >> > > In the past, I did start writing a python script to handle OSv apps in a > uniform way the code is lost somewhere here > > But the idea was to have a JSON file with 2 sections, one for > configuration, another for stages > and based on some reserved names, we do operations like downloading files, > checksum, etc > pretty much like and dependency/build manager like > maven/gradle/pip/conan/... > Of course, any suggestion is welcomed :) > > Also, because we are limited in terms of developers, I would suggest > leaving documentation until the new app builder is ready, otherwise, we > will take a lot of time to write documentation for current build that will > be lost and people will certainly ask for help if they are really > interested anyway > > >> I think the more difficult part is understand what the heck is module.py >> and what is usr.manifest, etc. >> > > if I understood, the script that handles OSv modules uses module.py to > expose OSv apps as OSv modules, otherwise, we would need to move OSv apps > inside <osv-dir>/modules (or change the python script that handles modules > in other directories accordingly) > > Well, in fact, it might be a future refactoring, moving non-essential > modules to OSv apps, for example, I think java, go, openssl, httpserver-*, > monitoring-agent could be moved to OSv apps > > As for usr.manifest, personally, I would prefer to not have user.manifest > and instead have a default directory inside app directory and everything > that you puts there will be added to the final osv file system image > I think most of the apps already generates a ROOTFS directory that creates > such a structure > not sure > > also, we could create tests to build each app automatically and see if > they're still working > I guess we have a number of apps that are not working due to various > reasons > > > I think you're right that just documenting the conventions we used will go >> a long way in helping people to use this system, even if we don't change >> anything. >> >> >>> >>> - >>> - I know our build system (scripts/build and related python scripts) >>> is complicated enough. But it would be super nice to allow passing >>> parameters from scripts/build to app Makefile. The best use case for >>> that >>> is building java9 (10,11,12 as well) JREs using jlink where it would be >>> nice to pass a list of JRE modules to incorporate - see >>> >>> https://github.com/cloudius-systems/osv-apps/blob/master/openjdk9-zulu-java-base/Makefile#L26; >>> >>> I think there are many other apps that would benefit >>> >>> Possibly. Indeed if properly documented, it shouldn't matter that it's >> even more complicated. >> >>> >>> - >>> - Eliminate external - >>> https://github.com/cloudius-systems/osv/issues/743 >>> - It would be nice to detail what to do to each subdirectory (I am >>> not positive we want to eliminate all - see x64/acpica) >>> >>> I agree. We should do that. "external" is an idea that really outlived >> its usefulness, I think. >> Indeed we don't need to remove everything at once, we can do it piece by >> piece (and stop using pieces before actually removing them). >> > > It might also be interesting updating acpica to the latest version > Of course, I understand that talk is cheap (speaking for myself...) > > >> >> >>> >>> - Convert to CMakefile >>> - I am not sure how difficult it would be, but one benefit of it >>> would be the ability to load a project in IDEs like CLion that are >>> able to >>> work with CMakefile, >>> >>> I would hate for this to happen. I really think OSv's makefile is well >> tuned and works well. >> Switching to cmake just to make an editor happy would be very sad. >> I don't know anything about clion, but is it possible to tell it OSv's >> include paths, etc., *without* switching the Makefile language? >> >> E.g., could we have "make clion" which automatically generates a file >> CMakeLists.txt >> which somehow includes the right incantations to tell CLIon what are OSv's >> include path and predefined macros needed to have the source files properly >> formatted? >> The file doesn't need to contain real instructions on how to compile >> anything, I hope... >> >> >>> >>> - Some apps (like python2x/python3x) pull most of the binaries from >>> host instead of re-compiling from scratch. It would be nice to have a >>> shared app/tool to build apps from host like that. >>> >>> I agree. It would really be nice to have scripts or python code for >> module.py or something, of how to copy a bunch of files from the host and >> possibly the shared libraries they need, etc., and then just use this same >> script from various apps instead of repeating the same code (with >> variations nobody understands) in a dozen apps. >> >> > While I'm always in favor of "newer"/"experimental" work, I would also > stick to the things that we can help and support by now because once again > we have limited resources to develop OSv > > >> >> >>> Feel free to add any other things you would like to be improved in the >>> build system. >>> >>> Waldek >>> >> >> > Let me know if I can help with anything :) > > > -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
