Re: [Bro-Dev] Broker data layouts

[Jon Siwek]

On Tue, Aug 21, 2018 at 1:09 PM Robin Sommer  wrote:

> Also, this question is about events, not logs, right? Logs have a
> different wire format and they actually come with meta data describing
> their columns.

Though the Broker data corresponding to log entry content is also
opaque at the moment (I recall that was maybe for performance or
message volume optimization), but I suppose same reasoning as before
could apply: this info is internal to Bro operation unless one wants
to explicitly re-publish it via their own event for external
consumption.

- Jon
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Broker data layouts

[Robin Sommer]

On Tue, Aug 21, 2018 at 12:34 -0500, Jonathan Siwek wrote:

> Maybe there's a more standardized approach that could be worked
> towards, but likely we just need more experience in understanding and
> defining common use-cases for external Bro data consumption.

Dominik, wasn't the original idea for VAST to provide an event
description language that would create the link between the values
coming over the wire and their interpretation? Such a specification
could be auto-generated from Bro's knowledge about the events it
generates.

Also, this question is about events, not logs, right? Logs have a
different wire format and they actually come with meta data describing
their columns.

Robin

-- 
Robin Sommer * Corelight, Inc. * ro...@corelight.com * www.corelight.com
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Broker data layouts

[Jon Siwek]

On Tue, Aug 21, 2018 at 8:54 AM Dominik Charousset
 wrote:

> This raises a couple of questions. Primarily: where can Broker users learn 
> the layouts to interpret received data?

broker/bro.hh is basically all there is right now.  e.g. if you
construct a broker::bro::Event from a received broker::data, you get
access to event name + "interpretable" arguments.

> There’s essentially no hope in deferring a layout, since broker::data doesn’t 
> include any meta information such as field names. Is meta information stored 
> somewhere?

No, nothing like that is in implicit in message content.

> Is there a convention how to place and retrieve such meta information in 
> Broker’s data stores?

No, and any stores created by Bro don't even have such meta info.  The
types stored in them are just documented like "the keys are type Foo
and values are type Bar".

> How does Bro itself make such information available?

Nothing beyond documentation or the Bro -> Broker type mapping that's
implicit in events themselves (or as given in docs for data stores).

> Is there a document that lists all topics used by Bro with their respective 
> broker::data layout?

I don't think there's a plan to keep such an up-to-date document.

A basic usage premise that I'm wondering about is that none of the
current Broker usage in Bro actually seems suitable for generic/public
consumption as it is.  It's maybe more implementation details of doing
cluster-enabled network traffic analysis, so also not a primary goal
to make interpretation of those communications easy for
external/direct Broker users.  (You can ingest it if you want and do
the work of "manually" interpreting it all, but maybe won't be a
stable/transparent source of data going forward).

However, one can still use Bro + Broker to create their own
events/stores in a way that does contain the meta information required
for easier/programmatic interpretation on the receiving side.  e.g. I
think, at the moment, if one is interested in ingesting data produced
by Bro, they are best served by explicitly defining topic names,
event/data types, and explicitly producing those messages at suitable
places within Bro scripts themselves.  Then, one can be in control of
defining a common/expected data format and include whatever meta
information is necessary to help receivers interpret the data.

Maybe there's a more standardized approach that could be worked
towards, but likely we just need more experience in understanding and
defining common use-cases for external Bro data consumption.

Or if we were just talking about Broker-only usage independent of Bro,
then I think it's still the same ideas/answers: currently up to user
to decide how to encode broker::data in a way that defines
common/expected layouts + any required meta info.

Does that help at all?

- Jon

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] Broker data layouts

[Dominik Charousset]

We are currently writing code for ingesting data directly using Broker’s API. 
From the docs, it seems that Broker assumes that publishers and subscribers 
somehow agree on one layout per topic: "senders and receivers will need to 
agree on a specific data layout for the values exchanged, so that they 
interpret them in the same way.” [1]

This raises a couple of questions. Primarily: where can Broker users learn the 
layouts to interpret received data? There’s essentially no hope in deferring a 
layout, since broker::data doesn’t include any meta information such as field 
names. Is meta information stored somewhere? Is there a convention how to place 
and retrieve such meta information in Broker’s data stores? How does Bro itself 
make such information available? Is there a document that lists all topics used 
by Bro with their respective broker::data layout?

Dominik


[1] https://bro-broker.readthedocs.io/en/stable/comm.html#exchanging-bro-events 
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev