Re: [Bro-Dev] package manager progress

[Slagell, Adam J]

> On Aug 5, 2016, at 6:37 PM, Siwek, Jon  wrote:
> 
> 
>> On Aug 5, 2016, at 1:29 PM, Slagell, Adam J  wrote:
>> 
>> If we are going to rely on metadata, which I agree can be better as you can 
>> tag a package with multiple categories, we should probably have some basic 
>> requirements for this type of metadata.
> 
> At a minimum, it’s useful to provide a list of “suggested keywords” that 
> people can choose from when tagging their packages so there’s at least a 
> common terminology to search within.
> 
> Do you mean to require something more than that?  E.g. “packages submitted to 
> the ‘bro’ package source MUST NOT use keywords outside the pre-approved list” 
> ?  Or “packages submitted to the ‘bro’ package source MUST contain at least 
> one keyword tag” ?
> 
I was thinking of requiring at least one keyword. 

> - Jon

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] package manager progress

[Siwek, Jon]

> On Aug 5, 2016, at 1:29 PM, Slagell, Adam J  wrote:
> 
> If we are going to rely on metadata, which I agree can be better as you can 
> tag a package with multiple categories, we should probably have some basic 
> requirements for this type of metadata. 

At a minimum, it’s useful to provide a list of “suggested keywords” that people 
can choose from when tagging their packages so there’s at least a common 
terminology to search within.

Do you mean to require something more than that?  E.g. “packages submitted to 
the ‘bro’ package source MUST NOT use keywords outside the pre-approved list” ? 
 Or “packages submitted to the ‘bro’ package source MUST contain at least one 
keyword tag” ?

- Jon

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] package manager progress

[Slagell, Adam J]

> On Aug 5, 2016, at 12:52 PM, Robin Sommer  wrote:
> 
>> 
>> Hhhm, is it naming conventions that people have a problem with or the 
>> implication of policing?
> 
> The problem is that the suggested naming convention wouldn't work:
> it's not clear how somebody would name their plugin if it provided
> more than one specific piece of functionality.

I would expect that any package repository has that same issue, there is no 
perfect taxonomy.

If we are going to rely on metadata, which I agree can be better as you can tag 
a package with multiple categories, we should probably have some basic 
requirements for this type of metadata. Do you agree?

:Adam

--

Adam J. Slagell
Chief Information Security Officer
Director, Cybersecurity Division
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
www.slagell.info

"Under the Illinois Freedom of Information Act (FOIA), any written 
communication to or from University employees regarding University business is 
a public record and may be subject to public disclosure." 









___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] package manager progress

[Robin Sommer]

On Fri, Aug 05, 2016 at 02:47 +, you wrote:

> Hhhm, is it naming conventions that people have a problem with or the 
> implication of policing?

The problem is that the suggested naming convention wouldn't work:
it's not clear how somebody would name their plugin if it provided
more than one specific piece of functionality.

Robin

-- 
Robin Sommer * ICSI/LBNL * ro...@icir.org * www.icir.org/robin
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] package manager progress

[Slagell, Adam J]

Hhhm, is it naming conventions that people have a problem with or the 
implication of policing? These can be separated. I don’t see a downside to 
promoting conventions. 

It also seems that some of the reason (e.g., that we have metadata is based on 
an assumption that we will have good metadata). But I recall a lot of 
resistance to requiring basic metadata.

I believe this merits a little more discussion and would like to nudge behavior 
if possible, though not compel it. We could do this by simply providing a 
skeleton taxonomy into which people could always just through things in “misc” 
or some equivalent.

> On Jul 27, 2016, at 12:57 PM, Robin Sommer  wrote:
> 
> Make it four. :) I'm with Seth, too, better not to enforce any naming
> scheme because the boundaries are unclear. Also, note that a single
> binary Bro plugin can provide multiple quite different things (say, a
> reader and an analyzer and a packet source all at the same time, if
> one so desires :).
> 
> Also agree with Johanna: the username is part of the package name if I
> follow correctly, so there's disambiguation there.
> 
> I have some more feeback on the package manager and Jon's questions
> starting this thread, will send soon.
> 
> Robin
> 
> On Wed, Jul 27, 2016 at 09:15 -0700, you wrote:
> 
>> And to add a me three to this - I am also with him on this one. On top 
>> of things - I might misremember this, but didn't we plan package names 
>> to include the github user name at one point of time? So a package name 
>> would be user/redis, for example, and there also could be user2/redis?
>> 
>> Johanna
>> 
>> On 27 Jul 2016, at 9:05, Matthias Vallentin wrote:
>> 
 I actually don't like this that much because some of these can cross
 boundaries and do all sorts of different things in a single plugin.
 It makes more sense to me to leave the naming open.
>>> 
>>> I'm with Seth on this one. The reason why I think we should keep the
>>> naming open is that it's the job of the meta data tags to take care of
>>> the grouping. If someone writes a redis package, then they should 
>>> apply
>>> the redis package. Encoding this meta data into the package name is
>>> quite limited, however.
>>> 
>>>Matthias
>>> ___
>>> bro-dev mailing list
>>> bro-dev@bro.org
>>> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
>> ___
>> bro-dev mailing list
>> bro-dev@bro.org
>> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
>> 
> 
> 
> -- 
> Robin Sommer * ICSI/LBNL * ro...@icir.org * www.icir.org/robin
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev


___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] package manager progress

[Siwek, Jon]

> On Jul 28, 2016, at 3:37 PM, Robin Sommer  wrote:
> 
>> in favor of auto-installing the python dependencies into Bro’s install
>> prefix.
> 
> I also prefer auto-installation, unless there's a reasonable risk that
> it could interfere with already installed versions of those packets,
> not sure?

Don’t think so.

> Ok, we probably need to write down our policy somewhere what we
> do/expect for the default source.

Expanding the README of https://github.com/bro/packages to include notes on the 
submission process and naming convention/policy seems like the place to me.

- Jon

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] package manager progress

[Robin Sommer]

On Thu, Jul 28, 2016 at 17:52 +, you wrote:

> I think still allowing package sources to be structured in a directory
> hierarchy is more intuitive to navigate and maybe less intimidating to
> modify than a single file as the source grows over time.  And I’m
> already using INI format in 2 other places, so seems fine to apply
> here, too.

Yep, agree with both. That also makes merging pull requests easy /
contained.

>  So a proposed structure of a package source:

Looks good to me.

> the CMake summary output.  Let me know which is preferred.  I’m a bit
> in favor of auto-installing the python dependencies into Bro’s install
> prefix.

I also prefer auto-installation, unless there's a reasonable risk that
it could interfere with already installed versions of those packets,
not sure?

> The hierarchy isn’t strictly required to use GitHub usernames. 
> Generally could be "$author_name/$package_name”, where the most common
> case is for $author to be a GitHub user name.  A domain name,
> company/organization name, or any string to help identify the author
> would work.

Ok, we probably need to write down our policy somewhere what we
do/expect for the default source.

> Agree about changing “package-manager” to “bro-package-manager”, but
> do you also mean to get rid of the “lib” subdir?

No, I didn't, sorry for the confusion. I was just too lazy to type the
full path again, should have inserted 3 dots to make that clear. 

> Tried to do this in the Overview/README’s “Installation” section.  I
> think reorganizing that in smaller sections with bullet points to
> follow and re-labeling it as “quick-start guide” may help.

Ack.

Robin

-- 
Robin Sommer * ICSI/LBNL * ro...@icir.org * www.icir.org/robin
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] package manager progress

[Siwek, Jon]

> On Jul 27, 2016, at 6:37 PM, Robin Sommer  wrote:
> 
> On the other hand, if, for example, somebody ends up
> browsing the package source repository on GitHub, I'm sure they'd be
> confused by all the packages pointing to very old versions.

Yeah, agree that is confusing and a problem of using submodules without ever 
updating them in the source repo.

> So I'm
> wondering if it would be worth switching to custom index instead of
> submodules; seems that wouldn't be difficult either if indeed all we
> need to do is track the external URLs somehow.

I’m on board with switching to a custom index format.

> Also, if you want to
> track discoverability metadata there already as well, seems that the
> URL could just become part of that, no?

Yes.

Any preference on the new index format?  Single index file?  Multiple files?  
INI, JSON, something else?

I think still allowing package sources to be structured in a directory 
hierarchy is more intuitive to navigate and maybe less intimidating to modify 
than a single file as the source grows over time.  And I’m already using INI 
format in 2 other places, so seems fine to apply here, too.  So a proposed 
structure of a package source:

https://github.com/bro/packages
alice/
bro-pkg.index
bob/
bro-pkg.index
…

alice’s bro-pkg.index:

[foo]
url = https://github.com/alice/foo
keywords = Mr.T pity

[bar]
url = https://github.com/alice/bar
keywords = club pub drinks

bob’s bro-pkg.index:

[baz]
url = https://github.com/bob/baz
keywords = lightning storm

Output of `bro-pkg list all`:

bro/alice/foo
bro/alice/bar
bro/bob/baz

> - Would suggest to rename “pkg.meta” to, say, “bro-pkg.meta”

Sure.

> - Does "upgrade" show the packages affected and ask for confirmation?
>  I would suggest either doing that or require an --all option for
>  upgrading everything, as that's a potentially dangerous operation.

It doesn’t ask for confirmation, but in favor requiring the explicit --all.

> - I suppose upgrading does (better: will do) dependency checking
>  again, including making sure the Bro version matches the one that
>  update now requires?

Yes, I imagine the dependency analysis for upgrading and installing being the 
same or similar process.

> - When installing the package manager as part of Bro, could we pull in
>  the Python dependencies automatically, for example by installing
>  them into the same prefix?

Yes, I can likely get that to work.

> Both GitHub and semantic_version are
>  pretty non-standard. Using them is ok I think but it would be nice
>  if "bro-pkg" wouldn't abort first thing because they aren't
>  installed yet.

Alternatively, I can have CMake detect whether they are installed, then, if 
not, don’t install bro-pkg and put a warning/explanation in the CMake summary 
output.  Let me know which is preferred.  I’m a bit in favor of auto-installing 
the python dependencies into Bro’s install prefix.

> - How about adding a note to either packages.bro or the whole
>  packages/ directory that's it's automatically maintained and not
>  supposed to be manually messed with?

Ok.

> - In bro-pkg.conf, has "default" in "[sources]" a special meaning, or
>  could it be any tag? Assuming the latter, I would just call it
>  "bro"

It’s arbitrary, will change ‘default' to ‘bro’.

> - For our default package source, do we want to support non-GitHub
>  repositories? If so, a naming scheme by GitHub user won't work.

The hierarchy isn’t strictly required to use GitHub usernames.  Generally could 
be "$author_name/$package_name”, where the most common case is for $author to 
be a GitHub user name.  A domain name, company/organization name, or any string 
to help identify the author would work.

> - Suggest to rename "/opt/bro/var/lib/package-manager" to
>  "../bro-package-manager" or "../bro-pkg”.

Agree about changing “package-manager” to “bro-package-manager”, but do you 
also mean to get rid of the “lib” subdir?  I think that fits within Filesystem 
Hierarchy Standard [1].  For /var/lib that says: 

"State information. Persistent data modified by programs as they run, e.g., 
databases, packaging system metadata, etc.”

There’s probably nuances that let you get away with other locations when 
installing to prefixes other than ‘/‘, but I find it generally works well to 
just replace ‘/‘ with user’s preferred install prefix.  Let me know what you 
think.

> - Once we support dependencies on Bro versions, would be nice if that
>  worked also with the "x.y-z" scheme that git master uses (and maybe
>  it just does anyways).

Should already work via semantic_version.

>- Python 3.x works, right? Then I'd list that explicitly.

Worked for me.  Will do.

>- A quick-start guide would be helpful that just mentions the most
>  important steps, including 

Re: [Bro-Dev] package manager progress

[Matthias Vallentin]

> - Would suggest to rename “pkg.meta” to, say, “bro-pkg.meta”, just to
>   make it more explicit that it's a Bro package. That's something one
>   can also then search for on GitHub.

Just throwing in two more permutations: bro.meta or bro.pkg.

> - For our default package source, do we want to support non-GitHub
>   repositories? If so, a naming scheme by GitHub user won't work.
> 
> - Suggest to rename "/opt/bro/var/lib/package-manager" to
>   "../bro-package-manager" or "../bro-pkg".

Yeah, especially if users don't install into the /opt/bro prefix, not
having "bro" as part of the filename might be confusing.

> For now, I think it's fine to just do your own Sphinx and publish it
> wherever (GitHub, RTM). 

In case you're going with github, here's something non-intuitive that
took me a while to figure out: you need to put an (empty) file .nojekyll
in the document root, otherwise github interprets directories starting
with underscores, which sphinx uses (e.g., _static and _images).

> Given how far you are already, I vote for making it part of 2.5. 

+1

Matthias
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] package manager progress

[Seth Hall]

> On Jul 27, 2016, at 12:15 PM, Johanna Amann  wrote:
> 
> And to add a me three to this - I am also with him on this one. On top of 
> things - I might misremember this, but didn't we plan package names to 
> include the github user name at one point of time? So a package name would be 
> user/redis, for example, and there also could be user2/redis?

I may have lost track of the design so I don't know where things stand now, but 
I think this would make sense too.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/


___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] package manager progress

[Robin Sommer]

On Sun, Jul 24, 2016 at 19:45 +, you wrote:

> The package manager client is at a point now where I think it would be
> usable.

Finally got a chance to play with it a bit. Excellent work, I really
like it!

Belows a list of just smaller things I noticed. The only larger
question I have is regading the use of submodules, also following up
on other parts of this thread. In principle, I actually quite like the
idea of using submodules; git already offers the mechanism, so why not
build on that. That said, seeing how the package manager ends up using
submodules, it's not quite the pure git model actually. If I
understood it right, it's using them really only to *find* the
external repos, but not to pinpoint a particular commit in there; the
package source never even updates the submodules. Given that approach,
I'm now wondering if a custom scheme wouldn't be the more intuitive
solution. My concern is that whoever looks at this submodule usage,
will take a while to understand what's actually happening. One could
argue that's only an implementation detail and shouldn't really matter
to anybody. On the other hand, if, for example, somebody ends up
browsing the package source repository on GitHub, I'm sure they'd be
confused by all the packages pointing to very old versions. So I'm
wondering if it would be worth switching to custom index instead of
submodules; seems that wouldn't be difficult either if indeed all we
need to do is track the external URLs somehow. Also, if you want to
track discoverability metadata there already as well, seems that the
URL could just become part of that, no?

Here's my list of other random things I noticed:

- Would suggest to rename “pkg.meta” to, say, “bro-pkg.meta”, just to
  make it more explicit that it's a Bro package. That's something one
  can also then search for on GitHub.

- Does "upgrade" show the packages affected and ask for confirmation?
  I would suggest either doing that or require an --all option for
  upgrading everything, as that's a potentially dangerous operation.

- I suppose upgrading does (better: will do) dependency checking
  again, including making sure the Bro version matches the one that
  update now requires?

- When installing the package manager as part of Bro, could we pull in
  the Python dependencies automatically, for example by installing
  them into the same prefix? Both GitHub and semantic_version are
  pretty non-standard. Using them is ok I think but it would be nice
  if "bro-pkg" wouldn't abort first thing because they aren't
  installed yet.

- How about adding a note to either packages.bro or the whole
  packages/ directory that's it's automatically maintained and not
  supposed to be manually messed with?

- In bro-pkg.conf, has "default" in "[sources]" a special meaning, or
  could it be any tag? Assuming the latter, I would just call it
  "bro": "bro/jsiwek/bro-test-package" is more intuitive than
  "default/jsiwek/bro-test-package".

- For our default package source, do we want to support non-GitHub
  repositories? If so, a naming scheme by GitHub user won't work.

- Suggest to rename "/opt/bro/var/lib/package-manager" to
  "../bro-package-manager" or "../bro-pkg".

- Once we support dependencies on Bro versions, would be nice if that
  worked also with the "x.y-z" scheme that git master uses (and maybe
  it just does anyways).

- I like the Python API!

- Documentation (nice!):

- Python 3.x works, right? Then I'd list that explicitly.

- A quick-start guide would be helpful that just mentions the most
  important steps, including basic installation along with Bro
  itself (once that's merged).

- The "Installation" section becomes a bit confusing towards with
  the end with all those paths. Maybe split some parts out into an
  advanced section or so?

- How-tos would be helpful that show by example how to create a
  (1) a pure script package, (2) and binary Bro plugin, and (3) a
  BroControl plugin.

Finally, my take on your questions:

> My current idea is that instead of putting this type of data inside
> the package’s metadata, the user puts it in the package source’s
> metadata.

Yeah, I like that.

> Automatic inter-package dependency analysis

Agree on lower priority.

> * Is it acceptable to depend on GitPython and semantic_version python 
> packages?

It is, it just would be nice if we could help people a bit getting
these installed, see above.

> * Documentation is hosted on GitHub at the moment, move to bro.org?

Keeping these docs separate makes sense, although it would also be
nice to have the option to integrate into bro.org at least.  For now,
I think it's fine to just do your own Sphinx and publish it wherever
(GitHub, RTM). We can later see what to do about bro.org. Generally,
our bro.org setup certainly needs work, it's become hard to maintain
and extend. We have been talking about some options here recently but
not settled on anything in particular yet.

> * Thoughts 

Re: [Bro-Dev] package manager progress

[Siwek, Jon]

> On Jul 27, 2016, at 12:59 PM, Robin Sommer  wrote:
> 
> Ah, I see. Would it be better to generally use the full path as the
> name, and not search for submatches, to make it consistent/unambiguous
> what a name refers to?

At least from my usage it’s been convenient to be able to use a short name.  It 
still always accepts full path names for packages even if they’re unambiguous 
when shortened, and the full path is what gets displayed in package listings so 
it’s never inconsistent in that regard.  A user is free to always type full 
paths and for those that like to use short names an occasional “please clarify” 
may be more helpful than annoying: e.g. “oh I didn’t realize there were two, I 
should look into which one is more appropriate for me to use”.

- Jon

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] package manager progress

[Robin Sommer]

Make it four. :) I'm with Seth, too, better not to enforce any naming
scheme because the boundaries are unclear. Also, note that a single
binary Bro plugin can provide multiple quite different things (say, a
reader and an analyzer and a packet source all at the same time, if
one so desires :).

Also agree with Johanna: the username is part of the package name if I
follow correctly, so there's disambiguation there.

I have some more feeback on the package manager and Jon's questions
starting this thread, will send soon.

Robin

On Wed, Jul 27, 2016 at 09:15 -0700, you wrote:

> And to add a me three to this - I am also with him on this one. On top 
> of things - I might misremember this, but didn't we plan package names 
> to include the github user name at one point of time? So a package name 
> would be user/redis, for example, and there also could be user2/redis?
> 
> Johanna
> 
> On 27 Jul 2016, at 9:05, Matthias Vallentin wrote:
> 
> >> I actually don't like this that much because some of these can cross
> >> boundaries and do all sorts of different things in a single plugin.
> >> It makes more sense to me to leave the naming open.
> >
> > I'm with Seth on this one. The reason why I think we should keep the
> > naming open is that it's the job of the meta data tags to take care of
> > the grouping. If someone writes a redis package, then they should 
> > apply
> > the redis package. Encoding this meta data into the package name is
> > quite limited, however.
> >
> > Matthias
> > ___
> > bro-dev mailing list
> > bro-dev@bro.org
> > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
> 


-- 
Robin Sommer * ICSI/LBNL * ro...@icir.org * www.icir.org/robin
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] package manager progress

[Siwek, Jon]

> On Jul 27, 2016, at 11:15 AM, Johanna Amann  wrote:
> 
> And to add a me three to this - I am also with him on this one. On top 
> of things - I might misremember this, but didn't we plan package names 
> to include the github user name at one point of time? So a package name 
> would be user/redis, for example, and there also could be user2/redis?

Yes, package sources support hierarchical package names, but don’t require it.  
The hierarchy for the default package source is currently 
“github_user_name/package_name”.  I’m the only one w/ a package at the moment, 
but you can see the structure here:

https://github.com/bro/packages

Right now, a user of bro-pkg can refer to my package as simply 
“bro-test-package”.  If another user, say “bob”, creates 
“bob/bro-test-package”, then the client will no longer accept 
“bro-test-package” for commands where it is ambiguous and tell the user to 
clarify between either “bob/bro-test-package” or “jsiwek/bro-test-package”.

It’s also not allowed to have two packages with the same shortened name (e.g. 
“bro-test-package”) installed simultaneously.  Interested to hear if people 
have use cases for that, but I expect the common case for same-name packages to 
be forks (either hard forks or just forking to contribute bugfixes) and 
allowing multiple packages of the same name to be installed may make that case 
more confusing/complicated for users/developers.

- Jon

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] package manager progress

[Johanna Amann]

And to add a me three to this - I am also with him on this one. On top 
of things - I might misremember this, but didn't we plan package names 
to include the github user name at one point of time? So a package name 
would be user/redis, for example, and there also could be user2/redis?

Johanna

On 27 Jul 2016, at 9:05, Matthias Vallentin wrote:

>> I actually don't like this that much because some of these can cross
>> boundaries and do all sorts of different things in a single plugin.
>> It makes more sense to me to leave the naming open.
>
> I'm with Seth on this one. The reason why I think we should keep the
> naming open is that it's the job of the meta data tags to take care of
> the grouping. If someone writes a redis package, then they should 
> apply
> the redis package. Encoding this meta data into the package name is
> quite limited, however.
>
> Matthias
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] package manager progress

[Matthias Vallentin]

> I actually don't like this that much because some of these can cross
> boundaries and do all sorts of different things in a single plugin.
> It makes more sense to me to leave the naming open.  

I'm with Seth on this one. The reason why I think we should keep the
naming open is that it's the job of the meta data tags to take care of
the grouping. If someone writes a redis package, then they should apply
the redis package. Encoding this meta data into the package name is
quite limited, however.

Matthias
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] package manager progress

[Azoff, Justin S]

> On Jul 27, 2016, at 11:44 AM, Seth Hall  wrote:
> 
> 
>> On Jul 25, 2016, at 4:49 PM, Azoff, Justin S  wrote:
>> 
>> In one aspect the pktsrc- prefix acts like a tag, but can also help 
>> disambiguate plugins... i.e., a redis log writer plugin vs. a redis data 
>> store plugin vs. a redis protocol analyzer.
> 
> I actually don't like this that much because some of these can cross 
> boundaries and do all sorts of different things in a single plugin.  It makes 
> more sense to me to leave the naming open.  If people want to name a plugin 
> with a prefix, they're free to, but I wouldn't want to discourage people from 
> maintaining individual plugins that provide a variety of features.
> 
>  .Seth

We really need to do this though, the end result otherwise will be chaos.  
Package names shouldn't have a generic name just because it was the first one 
in the repository.

Leaving it open will lead to:

The first person that writes a redis plugin for log writing calls it 'redis'.
Then a redis analyzer is called 'redis-analyzer' 
Then someone writes a redis input source and that gets called 
'input-source-redis'

Then a postgres analyzer is written and named 'postgresql'.
Then a postgres log writer plugin is named 'postgresql-log-writer'. 
Then an input source is written named 'postgresql-input-source'.

So a year later we end up with packages named:

redis
redis-analyzer
input-source-redis
postgresql
postgresql-log-writer
postgresql-input-source


Where 'redis' is a log writer plugin and 'postgresql' is an analyzer. 

Where the input source plugins are interchangeably named input-source-redis and 
postgresql-input-source.

If someone wanted to write a redis plugin that was both an input source, an 
analyzer, and a log writer, that could be called 'redis'... letting anything 
else be called 'redis' is confusing and misleading.


-- 
- Justin Azoff


___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] package manager progress

[Seth Hall]

> On Jul 25, 2016, at 4:49 PM, Azoff, Justin S  wrote:
> 
> In one aspect the pktsrc- prefix acts like a tag, but can also help 
> disambiguate plugins... i.e., a redis log writer plugin vs. a redis data 
> store plugin vs. a redis protocol analyzer.

I actually don't like this that much because some of these can cross boundaries 
and do all sorts of different things in a single plugin.  It makes more sense 
to me to leave the naming open.  If people want to name a plugin with a prefix, 
they're free to, but I wouldn't want to discourage people from maintaining 
individual plugins that provide a variety of features.

  .Seth


--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/


___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] package manager progress

[Matthias Vallentin]

> > I'm not 100% following. Isn't every package recorded as submodule?
> 
> Every package within a package source is recorded as a git submodule
> and that recording happens at the time the package author registers
> their package with a source.  The bro-pkg client itself makes no
> changes to submodules.

Got it, thanks! Also, this page of the manual really helped me fill in
the missing pieces:

https://bro.github.io/package-manager/source.html

> [..] How about this description:
>
> “The ‘list’ command outputs a list of packages that match a given
> category”

Yep, my favorite so far!

> > maybe add an underscore, e.g., script_path and plugin_path?
> 
> Yeah, can do that.  And maybe “dir” is more meaningful than “path”
> since the later may mean file or directory?

Also agreeing here.

Matthias
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] package manager progress

[Siwek, Jon]

> On Jul 25, 2016, at 10:31 PM, Matthias Vallentin  wrote:
> 
>> Right now, packages don’t get downloaded via the submodule, they are
>> cloned directly from the package’s full git URL (which git just
>> happens to encoded within the submodule).
>> 
>> So this means only packages a user is interested in end up getting
>> downloaded.  
> 
> I'm not 100% following. Isn't every package recorded as submodule?

Every package within a package source is recorded as a git submodule and that 
recording happens at the time the package author registers their package with a 
source.  The bro-pkg client itself makes no changes to submodules.

> Is there any use case where you would do a submodule update?

Depends on who “you” refers to:

- a regular bro-pkg user: no, they don’t need to be aware that submodules are 
used

- a package author: no, they only care that submodules are used when they do 
the one-time registration process to add their package to a source

- the bro-pkg developer/maintainer: not currently, but that’s maybe an 
implementation detail.  I don’t currently ever update submodules and instead 
clone packages directly via their full git URL to a separate location because I 
think that’s the more robust implementation.

- some other entity that does periodic analysis on all packages (e.g. web 
frontend): I’d probably expect them to not be using bro-pkg at all, but they 
clone a package source and do recursive submodule updates on it as the easiest 
way of downloading the latest versions of everything.

> Or are the
> packages just recorded there instead of recording them in a separate file?

Right, using git submodules isn’t a requirement for the bro-pkg client to work, 
we could make up a different file/format for registering packages.  But maybe 
submodules do provide some convenience for the last use case mentioned above.

> I think "known" is also ambiguous, because it doesn't clearly convey
> the local aspect. How about just saying "filters installed packages”?

Not all subcategories of “list” are working with just the locally “installed” 
packages.  E.g. “list all” is looking at both installed packages (local git 
repos) and not-installed packages (remote git repos, but we know about them 
because they are registered with a source).  How about this description:

“The ‘list’ command outputs a list of packages that match a given category”

> maybe add an underscore, e.g., script_path and plugin_path?

Yeah, can do that.  And maybe “dir” is more meaningful than “path” since the 
later may mean file or directory?

- Jon

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] package manager progress

[Matthias Vallentin]

> Right now, packages don’t get downloaded via the submodule, they are
> cloned directly from the package’s full git URL (which git just
> happens to encoded within the submodule).
>
> So this means only packages a user is interested in end up getting
> downloaded.  

I'm not 100% following. Isn't every package recorded as submodule? Is
there any use case where you would do a submodule update? Or are the
packages just recorded there instead of recording them in a separate
file?

> The package source just has to have some sort of database that links
> nodes in a package hierarchy (e.g. alice/foo, bob/bar, eve/baz) to
> their actual URLs.  Git submodules just happens to perform this role.

(Yeah, reusing this makes sense)

> >Filters available/installed packages by a chosen category and then
> >outputs that filtered package list.
> > 
> >  I don't understand what "available" means here. It could also mean
> >  "packages that exist remotely but not installed locally" as opposed to
> >  "available for use right now.” 
> 
> It means the former — “list” operates on the combined set of installed and 
> not-yet-installed packages.
> 
> Does wording it like “Filters known packages...” make it clearer for you?

I think "known" is also ambiguous, because it doesn't clearly convey
the local aspect. How about just saying "filters installed packages"?

> [..] but seeing “scripts” as an option, without reading any further
> documentation, implies to me that you might be able to specify a list
> of paths/files there, which you can’t.

Fair point. The reduction certainly omits some semantics. To simplify
reading the options, maybe add an underscore, e.g., script_path and
plugin_path?

Matthias
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] package manager progress

[Azoff, Justin S]

> On Jul 24, 2016, at 3:45 PM, Siwek, Jon  wrote:
> 
> * Add a way for package’s to define “discoverability metadata”.

Kind of related to this, I think we need to define some basic rules for package 
naming.  This can help discoverability and also namespacing issues. Right now 
we have plugins named:

af_packet
elasticsearch
kafka
myricom
netmap
pf_ring
redis
tcprs

But I think they need to be renamed using prefixes like:


af_packet - pktsrc-af_packet
elasticsearch - log-writer-elasticsearch
kafka - log-writer-kafka
myricom   - pktsrc-myricom
netmap- pktsrc-netmap
pf_ring   - pktsrc-pf_ring
redis - log-writer-redis
tcprs - analyzer-tcprs


In one aspect the pktsrc- prefix acts like a tag, but can also help 
disambiguate plugins... i.e., a redis log writer plugin vs. a redis data store 
plugin vs. a redis protocol analyzer.


-- 
- Justin Azoff



___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] package manager progress

[Siwek, Jon]

> On Jul 25, 2016, at 10:18 AM, Matthias Vallentin  wrote:
> 
>> * Add a way for package’s to define “discoverability metadata”.
>> 
>> E.g. following the original plan for this would involve putting
>> something like a “tags” field in each package’s pkg.meta file, but the
>> problem with this is the client would need to either download every
>> package to be able to search this data or have a third-party
>> periodically aggregate it.
> 
> What does "downloading" a package mean? If the package is in the
> .gitmodules of the repo bro/packages, won't it be automatically
> downloaded once the user updates their submodules?

Right now, packages don’t get downloaded via the submodule, they are cloned 
directly from the package’s full git URL (which git just happens to encoded 
within the submodule).

So this means only packages a user is interested in end up getting downloaded.  
I think it also helps in cases where a user installs a package and later it 
gets removed from the package source — so the submodule is gone, but user’s 
installed version is not effected because it’s cloned directly from the 
package’s git URL.  i.e. the package manager doesn’t distinguish between 
packages installed from a package source and packages installed directly from 
git URL.

If we wanted, we could actually use something completely different from git 
submodules to register a package to a package source.  The package source just 
has to have some sort of database that links nodes in a package hierarchy (e.g. 
alice/foo, bob/bar, eve/baz) to their actual URLs.  Git submodules just happens 
to perform this role.  Maybe another added benefit of submodules is that if 
someone (e.g. a web frontend) does want to download the “universe of packages” 
(maybe to do some global stats/analysis on it) its easy to do that with a 
single builtin git command.

> Agreed on inter-package dependencies. How about specifying a specific
> Bro version as "dependency”?

Yep, that’s on also on the TODO list.

> have you thought about publishing it via read-the-docs?

Yeah, just haven’t looked into it.  I’ll do that unless consensus is to host 
the docs on bro.org.

> Some minor feedback:
> 
> - Is the "refresh" command essentially what "update" is to Homebrew? The
>  documentation says:
> 
>Update local package source clones to retrieve information about new
>packages that are available. Also fetches updated package
>information about any installed packages to determine if new
>versions are available.
> 
>  It sounds like this means it's doing submodule update.

I’ll try to clarify it in the docs.  It doesn’t do a recursive submodule 
update, it just updates the source repo itself (so submodule additions/removals 
are visible, but nothing further is actually downloaded).

> - The documentation of the "list" command says:
> 
>Filters available/installed packages by a chosen category and then
>outputs that filtered package list.
> 
>  I don't understand what "available" means here. It could also mean
>  "packages that exist remotely but not installed locally" as opposed to
>  "available for use right now.” 

It means the former — “list” operates on the combined set of installed and 
not-yet-installed packages.

Does wording it like “Filters known packages...” make it clearer for you?

> - Regaring pkg.meta: this is more of a nit/style thing, but I like
>  minimalistic naming of configuration options, e.g.:
> 
>[package]
>version = 1.0.0
>scripts = /path/to/scripts
>plugins = /path/to/plugins

Open to changing it, but seeing “scripts” as an option, without reading any 
further documentation, implies to me that you might be able to specify a list 
of paths/files there, which you can’t.

- Jon

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] package manager progress

[Siwek, Jon]

> On Jul 25, 2016, at 6:53 AM, Jan Grashöfer  wrote:
> 
>> * Add a way for package’s to define “discoverability metadata”.
>> 
>> E.g. following the original plan for this would involve putting something 
>> like a “tags” field in each package’s pkg.meta file, but the problem with 
>> this is the client would need to either download every package to be able to 
>> search this data or have a third-party periodically aggregate it.
> 
> I think this is a question about who should deal with the extra effort:
> On the one hand requiring to spread and sync information between two
> places introduces a burden for the contributors

The idea was not for contributors to have to keep syncing the information 
between two places, the “discoverability” metadata would just be located within 
the “package source” instead of the package itself.

My thinking is that discoverability metadata should be more of a property of a 
package source than the package itself — e.g. if a user is looking at 
discoverability data in a package’s pkg.meta file, it’s not that helpful 
because they’ve already found the package.

Also, some people may initially have no intention of sharing their package, so 
there’s no reason to put discoverability metadata in its pkg.meta.  If they 
later change their mind, and care enough to take the time to register it to a 
package source, then likely they don’t mind adding a few keywords to a new meta 
file as an optional part of the one-time registration process.

> One note: I think the documentation should contain a tremendous warning
> pointing out that the users are responsible for what they are
> installing

Thanks for the suggestion, I’ll do that.

- Jon

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] package manager progress

[Matthias Vallentin]

> The package manager client is at a point now where I think it would be 
> usable.  

Cool!

> * Add a way for package’s to define “discoverability metadata”.
> 
> E.g. following the original plan for this would involve putting
> something like a “tags” field in each package’s pkg.meta file, but the
> problem with this is the client would need to either download every
> package to be able to search this data or have a third-party
> periodically aggregate it.

What does "downloading" a package mean? If the package is in the
.gitmodules of the repo bro/packages, won't it be automatically
downloaded once the user updates their submodules?

> I put it at lower priority since I don’t think it will be common right
> off the bat to have complex package dependencies and users can always
> manually resolve dependencies at the moment.

Agreed on inter-package dependencies. How about specifying a specific
Bro version as "dependency"?

> * Documentation is hosted on GitHub at the moment, move to bro.org?

A key benefit of hosting it at github is reliability and that clients
get good viewing performance thanks to their CDN.

> The current doc/www setup feels like it’s getting rather
> large/monolithic and maybe that contributes to the difficulty of
> approaching/understanding it when there’s breakages.  Just an idea.

Keeping it separate could be an advantage for users, because the current
documentation is a bit unwieldy and confusing. Since you've written it
in RST, have you thought about publishing it via read-the-docs? Their
documentation really reads very smoothly out of the box. CAF, for
example, recently switched to it [1].

Some minor feedback:

- Is the "refresh" command essentially what "update" is to Homebrew? The
  documentation says:

Update local package source clones to retrieve information about new
packages that are available. Also fetches updated package
information about any installed packages to determine if new
versions are available.

  It sounds like this means it's doing submodule update.

- The documentation of the "list" command says:
  
Filters available/installed packages by a chosen category and then
outputs that filtered package list.

  I don't understand what "available" means here. It could also mean
  "packages that exist remotely but not installed locally" as opposed to
  "available for use right now." To avoid ambiguity and clearly
  distinguish it from "search", I would make that clear in the
  documention.

- Regaring pkg.meta: this is more of a nit/style thing, but I like
  minimalistic naming of configuration options, e.g.:

[package]
version = 1.0.0
scripts = /path/to/scripts
plugins = /path/to/plugins

  I find them easier to remember. But Robin would probably disagree with
  me here :-).

Looking forward to see it shaping up!

Matthias

[1] http://actor-framework.readthedocs.io/en/latest/
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] package manager progress

[Jan Grashöfer]

Amazing work! I really like the package manager and I am looking forward
to contributing a script.

> * Add a way for package’s to define “discoverability metadata”.
> 
> E.g. following the original plan for this would involve putting something 
> like a “tags” field in each package’s pkg.meta file, but the problem with 
> this is the client would need to either download every package to be able to 
> search this data or have a third-party periodically aggregate it.

I think this is a question about who should deal with the extra effort:
On the one hand requiring to spread and sync information between two
places introduces a burden for the contributors, on the other hand
(automatic) aggregation of information makes it harder to maintain a
source including metadata. I am in favor of putting that information
into pkg.meta to make contributing as easy as possible.

One note: I think the documentation should contain a tremendous warning
pointing out that the users are responsible for what they are
installing. One scenario that came instantly to my mind: Someone is
contributing a small and useful script, waits for its distribution and
than updates his repository, adding e.g. a malicious build command. In
that context it would be nice if the package manager would ask the user
before executing the build command. For the official repository also
some automatic checks would be nice (e.g. indicating in case a script
executes shell commands). I think that was discussed before.

All in all I think the package manager design is intuitive and really
easy to use. Having central repositories will be great!

Thanks,
Jan
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] package manager progress

[Siwek, Jon]

The package manager client is at a point now where I think it would be usable.  
Documentation is here:

https://bro.github.io/package-manager/

There is a branch in the ‘bro’ repo called ‘package-manager’ that simply 
changes CMake scripts to install ‘bro-pkg’ along with bro.  Here’s an example 
usage/session:

$ git clone --recursive --branch=package-manager git://bro.org/bro
...
$ cd bro && ./configure && make install
...
$ /usr/local/bro/bin/bro-pkg list all
default/jsiwek/bro-test-package
$ /usr/local/bro/bin/bro-pkg install bro-test-package
installed "bro-test-package"
loaded "bro-test-package"
$ /usr/local/bro/bin/bro packages
loaded bro-test-package plugin
loaded bro-test-package scripts
$ /usr/local/bro/bin/broctl
Test package: initialized
…

That test package shows that bro-pkg was able to install a package containing 
Bro scripts, a Bro plugin, and a BroControl plugin and everything should “just 
work” without needing any configuration.

Roadmap/TODO/Questions:

* Add a way for package’s to define “discoverability metadata”.

E.g. following the original plan for this would involve putting something like 
a “tags” field in each package’s pkg.meta file, but the problem with this is 
the client would need to either download every package to be able to search 
this data or have a third-party periodically aggregate it.

My current idea is that instead of putting this type of data inside the 
package’s metadata, the user puts it in the package source’s metadata.  They do 
this on first registration and may update it whenever.  That way, bro-pkg 
always has access to latest discoverability metadata, no need for a separate 
aggregation process.  It’s also something that will rarely change, so not a 
problem for that data to live in a repo not owned by the package author and not 
much increased burden for Bro Team to accept pull requests to update this data. 
 Thoughts?

* Automatic inter-package dependency analysis

Simply a TODO.  I put it at lower priority since I don’t think it will be 
common right off the bat to have complex package dependencies and users can 
always manually resolve dependencies at the moment.

* Is it acceptable to depend on GitPython and semantic_version python packages?

Both are replaceable implementation details, just didn’t want to write 
something myself if not necessary and in interest of time.

* Documentation is hosted on GitHub at the moment, move to bro.org?

Mostly just on GitHub now to be able to show something without having to touch 
any of the master bro/www doc generation processes, but maybe it’s a nice thing 
to start keeping docs more compartmentalized?  The current doc/www setup feels 
like it’s getting rather large/monolithic and maybe that contributes to the 
difficulty of approaching/understanding it when there’s breakages.  Just an 
idea.

* Thoughts on when to merge ‘package-manager’ branch in ‘bro’ ?

IMO, it can be done now or soon after I address responses/feedback to this 
email.

- Jon

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev