Sounds great.. What you are describing is basically source and binary packages.
The only thing I would do differently is that you wouldn't want bundles (at least not as the only feature) but individual source and binary packages. For example bro-pkg install sethhall/domain-tld would still install the package from git like normal, but one could do bro-pkg dist sethhall/domain-tld or just bro-pkg dist # inside the project and this would generate a versioned sethhall_domain-tld-1.0.0.tar.gz source package Then, that file could be copied over to the target machine and bro-pkg install sethhall_domain-tld-1.0.0.tar.gz or variations of bro-pkg install https://packages.internal/bro/sethhall_domain-tld-1.0.0.tar.gz would install it. The hard part is how to handle compiled architecture specific packages. For something like a myricom plugin, bro-pkg dist bro-plugins/myricom would generate a bro-plugins_myricom-1.0.0.tar.gz source package as before If copied over to the target system bro-pkg install bro-plugins_myricom-1.0.0.tar.gz would build and install it as long as build tools were installed. If 'no build tools on servers' is a requirement, then bro-pkg build bro-plugins/myricom on a build server would compile and generate something like bro-plugins_myricom-1.0.0-bro2.5-linux-amd64.tar.gz And with some more glue, could easily build .deb and .rpm packages. This is basically how python/pip/wheel works now (after ~10 years of being pretty bad). You could still have a bundle command as a very small layer on top of packages, but the building block should be an individual package distribution. Also, > That command will completely replace the system’s current set of packages > with the bundle’s content should probably not be the only behavior. Installing should be the default unless you do something like bro-pkg install --replace my.bro.bundle otherwise it would be impossible to compose bundles. -- - Justin Azoff > On Sep 19, 2016, at 7:27 PM, Robin Sommer <[email protected]> wrote: > > At BroCon several people pointed out a need to install Bro packages on > machines that do not have a direct external connection. One idea would > be some kind of proxy scheme where an intermediary git repository > mirrors packages locally; bro-pkg would then pull from there. However, > I don't think any of us really liked that idea much. After a few > rounds of discussions, Seth and I came up with a different idea that > seems easier to manage: extending bro-pkg to bundle packages into > deployment files that can be easily pushed to Bro systems simply by > copying them over. > > I’ve tried to flesh this out a bit more, and would be interested to > hear what you all think about this approach. And @Jon: Do you think > this would be doable that way? > > Here’s the idea: > > 1. Generally, one first uses bro-pkg as usual to install packages onto > a local Bro system that does have external Internet connectivity > (this could be just a dummy Bro installation). One installs new > packages there, updates existing ones, etc., until reaching a state > that one wants to push out to the actual Bro system. > > 2. We add a new “bundle” command to bro-pkg that serializes the > current state of packages into a single file on disk, a “package > bundle”. The bundle contains the complete content of all currently > installed packages, using some kind of suitable container format > (could be just a ZIP file, or whatever works; the internal > representation doesn’t really matter). > > 3. Users create such a bundle on the local system and then simply copy > that bundle file over to all target Bro machines that do not have a > external connectivity themselves, using whatever mechanism they > have available (e.g., just scp; or maybe through some configuration > management system like Ansible etc.). > > 4. On the target machine, one runs a corresponding “bro-pkg unbundle” > command on that bundle file. That command will completely replace > the system’s current set of packages with the bundle’s content. As > a result, that machine will now have exactly the same packages > installed as the original system. > > This would be the general scheme. A couple of people I talked to at > BroCon confirmed that this would offer a viable solution for them, and > that they would indeed much prefer copying files around over > maintaining local git mirrors. > > Some additional thoughts on variations/extensions of this basic scheme: > > - I’m not quite sure if the bundle should contain just the packages > themselves or further bro-pkg state as well, such as which packages > are currently loaded. Right now I’m learning towards saying “just > the packages”; that would basically treat bundles just as a > transport mechanism to get packages over to another box. The actual > Bro machines would still keep control over which packages to > actually load, etc. > > - As it is described above, Step 1 would require having a local Bro > installation into which packages get installed before they can be > bundled up. It would be nice to have a mode where bro-pkg can > operate without having a Bro around at all, just downloading > packages locally somewhere for bundling them up. I could also see > offering an even simpler mode where one simply lists packages to > bundle on the command-line: “bro-pkg bundle <pkg1> <pkg2> <pkg3>”. > That would be particularly useful with configuration management > systems I think. > > - It would be neat if bro-pkg's Python library exposed operations to > inspect & retrieve the content of a bundle, such as iterating over > the packages inside a bundle and iterating over the files inside a > package. That way one could easily build target-side scripts that > process and validate bundles before going ahead and installing them > (e.g., imposing custom restrictions on what kind of packages one > allows to put in place; or ensuring that a bundle always contains a > set of packages the site deems mandatory, to avoid configuration > mistakes; or even just logging what gets pushed out). > > What do you guys think about this? > > Robin _______________________________________________ bro-dev mailing list [email protected] http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
