Please see my specific responses inlined below. In essence in short term I am simply arguing for adding an extra capability of exporting files in addition of appending them to an image as it happens now during build process. And as I understand you do not seem to be opposed to this idea (however others that have not responded yet might be).
Also if we went ahead and added export logic to upload_manifest.py it would break capstan-packages as it patches upload_manifest.py. So we either modify OSv upload_manifest.py and update the patch or simply add new script called export_manifest.py that would be called by scripts/build. One caveat to the latter approach is that there would be no way to share any common code between export_manifest.py and upload_manifest.py (because again we would not want to modify upload_manifest.py). On Tuesday, July 4, 2017 at 5:54:33 AM UTC-4, Nadav Har'El wrote: > > > On Fri, Jun 30, 2017 at 10:43 PM, Waldek Kozaczuk <jwkoz...@gmail.com > <javascript:>> wrote: > >> I am not sure if this is the best forum as it relates to the MikelAngelo >> capstan. >> > > I think it is, because all the relevant Mikelangelo developers are on this > list, and anyway the "original" Capstan is more-or-less dead, and it's good > we have a live alernative. > > >> Assuming it is let me pose couple of questions regarding the tool as I >> am in process of learning new capstan and creating my own repository in S3. >> >> As I understand the main difference between the new capstan and the >> original one is the concept of packages which is really very neat idea I >> think. I was reading the documentation (under >> https://github.com/mikelangelo-project/capstan) and I found it quite >> easy to understand how to build "application" packages which would then get >> composed with the dependant "base" (aka runtime) packages. On other hand is >> not that easy or obvious to understand how to create your own base packages >> which is what https://github.com/mikelangelo-project/capstan-packages >> seems to be doing for MikelAngelo. >> >> I can see that for each package (per >> https://github.com/mikelangelo-project/capstan-packages/blob/master/docker_files/recipes/osv.httpserver/build.sh) >> >> the command ($OSV_BUILD_PATH)/scripts/build is executed which then creates >> manifest files for each module and the final usr.manifest and creates QCOW2 >> image. Then build.sh from capstan-packages exports the files based on the >> usr.manifest generated by previous step minus the skeleton. It is achieved >> by the login from the pach >> https://github.com/mikelangelo-project/capstan-packages/blob/master/docker_files/common/upload_manifest.py.patch >> >> which adds export_package(manifest, dest): to upload_manifest.py. >> >> So first of I was wondering if it would be a good idea add this exporting >> logic to upload_manifest.py in the main OSv source tree and also modify the >> scripts/build to allow exporting which would the also skip creating QCOW2 >> image in this mode. Is this a good idea? >> > > So you mean that scripts/build would, with a certain option, build *only* > the package, not kernel (and not combine the kernel and the package > together)? > > Makes sense, I guess, although in the OSv build system, there will be > nothing which *uses* the resulting packages, so it won't be very useful. > But maybe it can't hurt to have it as another feature. > It would not be useful to typical OSv developer in strictly speaking however it would be useful to a Capstan user to be able to create their own version of the packages that I call "base" ones. Currently there is no "easy" automated way to build a list of files that go in those example packages: openjdk7 OpenJDK 1.7.0 0.1 0001-01-01T00:00:00Z openjdk8-zulu-compact1 OpenJDK 1.8.0_112 0.1 0001-01-01T00:00:00Z openjdk8-zulu-compact3-with-java-beans OpenJDK 1.8.0_112 zulu-compact3-with-java-beans 0.1 0001-01-01T00:00:00Z osv.bootstrap OSv Bootstrap 0.1 0001-01-01T00:00:00Z osv.cli OSv Command Line Interface 0.1 0001-01-01T00:00:00Z osv.cloud-init cloud-init 0.1 0001-01-01T00:00:00Z osv.httpserver OSv HTTP REST Server 0.1 0001-01-01T00:00:00Z > > >> >> Currently if built and exported module X that depended on other modules - >> say A and B - the resulting image or package would have files from A, B and >> Y. Also I was wondering if we could change it allow creating packages for >> deltas only. That way the build process using capstan would be even more >> modular or additive if you will. >> > > Unfortunately, I am really not familiar with what Capstan provide (the > old, or the new), so I can only comment on what I think *should* happen, > not what it does today. > > I think that people should be able to compile packages A, B, and X > *separatey*, and the result is 3 separate tarballs (or whatever). These > tarballs should contain some sort of spec file which specifies dependencies > - e.g., X specifies it depends on A and B. Then when someone asks to add > tarball X to the image, A and B are also picked up. This is more-or-less > what happens in package systems of Linux distributions (e.g., RPM). > Agreed. I think this feature is most beneficial when building Java app packages as they depend on specific major version of Java (7,8 or 9), type (full, compact, etc), specific mode (isolated vs non-isolated) and specific minor version of JDK (121, etc). Each of these 4 Java runtime characteristics could be provided by specific version of specific type of package and thus minimize proliferation of base packages to run Java apps. And I think it is all possible with current Mikelangelo capstan as long as there is an easy way to know what files go into each package which new exporting capability of OSv would provide. > >> >> Lastly I noticed that the base repository assumes the packages are stored >> in AWS S3 bucket. I was wondering what it would take to introduce >> repository plugin concept to new capstan ( >> https://github.com/mikelangelo-project/capstan/blob/master/util/s3_repository.go) >> >> where it would let one store and retrieve from artifactory (( >> https://www.jfrog.com/artifactory/)) or maven repository. Any thoughts? >> > > Sounds reasonable. Would be nice to have a URL-like specification of > source, so the user can specify any one of these options - or even normal > http://. > >> >> Regards, >> Waldek >> >> -- >> You received this message because you are subscribed to the Google Groups >> "OSv Development" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to osv-dev+u...@googlegroups.com <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
