Sounds great.. What you are describing is basically source and binary packages.

The only thing I would do differently is that you wouldn't want bundles (at 
least not as the only feature) but individual source and binary packages.

For example

bro-pkg install sethhall/domain-tld

would still install the package from git like normal, but one could do

bro-pkg dist sethhall/domain-tld

or just

bro-pkg dist  # inside the project

and this would generate a versioned sethhall_domain-tld-1.0.0.tar.gz source 
package

Then, that file could be copied over to the target machine and

bro-pkg install sethhall_domain-tld-1.0.0.tar.gz

or variations of

bro-pkg install https://packages.internal/bro/sethhall_domain-tld-1.0.0.tar.gz

would install it.



The hard part is how to handle compiled architecture specific packages. For 
something like a myricom plugin,

bro-pkg dist bro-plugins/myricom

would generate a bro-plugins_myricom-1.0.0.tar.gz source package as before

If copied over to the target system

bro-pkg install bro-plugins_myricom-1.0.0.tar.gz

would build and install it as long as build tools were installed.  If 'no build 
tools on servers' is a requirement, then

bro-pkg build bro-plugins/myricom 

on a build server would compile and generate something like 
bro-plugins_myricom-1.0.0-bro2.5-linux-amd64.tar.gz

And with some more glue, could easily build .deb and .rpm packages.

This is basically how python/pip/wheel works now (after ~10 years of being 
pretty bad).

You could still have a bundle command as a very small layer on top of packages, 
but the building block should be an individual package distribution.


Also,

> That command will completely replace the system’s current set of packages 
> with the bundle’s content

should probably not be the only behavior.  Installing should be the default 
unless you do something like 

bro-pkg install --replace my.bro.bundle

otherwise it would be impossible to compose bundles.

-- 
- Justin Azoff

> On Sep 19, 2016, at 7:27 PM, Robin Sommer <ro...@icir.org> wrote:
> 
> At BroCon several people pointed out a need to install Bro packages on
> machines that do not have a direct external connection. One idea would
> be some kind of proxy scheme where an intermediary git repository
> mirrors packages locally; bro-pkg would then pull from there. However,
> I don't think any of us really liked that idea much. After a few
> rounds of discussions, Seth and I came up with a different idea that
> seems easier to manage: extending bro-pkg to bundle packages into
> deployment files that can be easily pushed to Bro systems simply by
> copying them over. 
> 
> I’ve tried to flesh this out a bit more, and would be interested to
> hear what you all think about this approach. And @Jon: Do you think
> this would be doable that way?
> 
> Here’s the idea:
> 
> 1. Generally, one first uses bro-pkg as usual to install packages onto
>   a local Bro system that does have external Internet connectivity
>   (this could be just a dummy Bro installation). One installs new
>   packages there, updates existing ones, etc., until reaching a state
>   that one wants to push out to the actual Bro system.
> 
> 2. We add a new “bundle” command to bro-pkg that serializes the
>   current state of packages into a single file on disk, a “package
>   bundle”. The bundle contains the complete content of all currently
>   installed packages, using some kind of suitable container format
>   (could be just a ZIP file, or whatever works; the internal
>   representation doesn’t really matter).
> 
> 3. Users create such a bundle on the local system and then simply copy
>   that bundle file over to all target Bro machines that do not have a
>   external connectivity themselves, using whatever mechanism they
>   have available (e.g., just scp; or maybe through some configuration
>   management system like Ansible etc.).
> 
> 4. On the target machine, one runs a corresponding “bro-pkg unbundle”
>   command on that bundle file. That command will completely replace
>   the system’s current set of packages with the bundle’s content. As
>   a result, that machine will now have exactly the same packages
>   installed as the original system.
> 
> This would be the general scheme. A couple of people I talked to at
> BroCon confirmed that this would offer a viable solution for them, and
> that they would indeed much prefer copying files around over
> maintaining local git mirrors.
> 
> Some additional thoughts on variations/extensions of this basic scheme:
> 
> - I’m not quite sure if the bundle should contain just the packages
>  themselves or further bro-pkg state as well, such as which packages
>  are currently loaded. Right now I’m learning towards saying “just
>  the packages”; that would basically treat bundles just as a
>  transport mechanism to get packages over to another box. The actual
>  Bro machines would still keep control over which packages to
>  actually load, etc.
> 
> - As it is described above, Step 1 would require having a local Bro
>  installation into which packages get installed before they can be
>  bundled up. It would be nice to have a mode where bro-pkg can
>  operate without having a Bro around at all, just downloading
>  packages locally somewhere for bundling them up. I could also see
>  offering an even simpler mode where one simply lists packages to
>  bundle on the command-line: “bro-pkg bundle <pkg1> <pkg2> <pkg3>”.
>  That would be particularly useful with configuration management
>  systems I think.
> 
> - It would be neat if bro-pkg's Python library exposed operations to
>  inspect & retrieve the content of a bundle, such as iterating over
>  the packages inside a bundle and iterating over the files inside a
>  package. That way one could easily build target-side scripts that
>  process and validate bundles before going ahead and installing them
>  (e.g., imposing custom restrictions on what kind of packages one
>  allows to put in place; or ensuring that a bundle always contains a
>  set of packages the site deems mandatory, to avoid configuration
>  mistakes; or even just logging what gets pushed out).
> 
> What do you guys think about this?
> 
> Robin



_______________________________________________
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Reply via email to