On Tue, Aug 28, 2018 at 17:12 +0200, Dominik Charousset wrote:
> 1) Matthias threw in memory-mapping, but I’m not so sure if this is
> actually feasible for you.
Yeah, our normal use case is different, memory-mapping won't help much
with that.
> 2) CAF already does batching. Ideally, Broker
>> Okay. In the future, we probably need some form of
>> "serialization-free" batching mechanism to ship data more efficiently.
>
> Do you guys have a sense of how load splits up between serialization
> and batching/communication? My hope has been that batching itself can
> take care of the
On Sat, Aug 25, 2018 at 17:42 +0200, Matthias Vallentin wrote:
> Okay. In the future, we probably need some form of
> "serialization-free" batching mechanism to ship data more efficiently.
Do you guys have a sense of how load splits up between serialization
and batching/communication? My hope
> Agree. Right now a newly connecting peer gets a round of explicit
> LogCreates, but that's probably not the best way forward for larger
> topologies.
Okay. In the future, we probably need some form of
"serialization-free" batching mechanism to ship data more efficiently.
There exist
On Fri, Aug 24, 2018 at 16:32 +0200, Matthias Vallentin wrote:
> It sounds like this is critical also for regular operation:
Agree. Right now a newly connecting peer gets a round of explicit
LogCreates, but that's probably not the best way forward for larger
topologies.
> is it currently
> I don't really see a way around that without substantially increasing
> volume. We could send LogCreate updates regularly, so that it's easier
> to synchronize with an ongoing stream.
It sounds like this is critical also for regular operation: (1) when
an endpoint bootstraps slowly and the
> Dominik, wasn't the original idea for VAST to provide an event
> description language that would create the link between the values
> coming over the wire and their interpretation? Such a specification
> could be auto-generated from Bro's knowledge about the events it
> generates.
We were
On Thu, Aug 23, 2018 at 10:01 -0500, Jonathan Siwek wrote:
> Yeah, that's one problem, but a bigger issue is you can't parse
> LogWrite because the content is a serial blob whose format is another
> thing not intended for public consumption.
I guess my earlier comment might have been
On Thu, Aug 23, 2018 at 15:32 +0200, Dominik Charousset wrote:
> Does that mean I need to receive the LogCreate even first to
> understand successive LogWrite events?
I don't really see a way around that without substantially increasing
volume. We could send LogCreate updates regularly, so
On Thu, Aug 23, 2018 at 8:32 AM Dominik Charousset
wrote:
> I’m a bit hesitant to rely on this header at the moment, because of:
>
> /// A Bro log-write message. Note that at the moment this should be used only
> /// by Bro itself as the arguments aren't publicly defined.
>
> Is the API stable
On Tue, Aug 21, 2018 at 14:05 -0500, Jonathan Siwek wrote:
> Though the Broker data corresponding to log entry content is also
> opaque at the moment (I recall that was maybe for performance or
> message volume optimization),
Yeah, but generally this is something I could see opening up. The
On Tue, Aug 21, 2018 at 1:09 PM Robin Sommer wrote:
> Also, this question is about events, not logs, right? Logs have a
> different wire format and they actually come with meta data describing
> their columns.
Though the Broker data corresponding to log entry content is also
opaque at the
On Tue, Aug 21, 2018 at 12:34 -0500, Jonathan Siwek wrote:
> Maybe there's a more standardized approach that could be worked
> towards, but likely we just need more experience in understanding and
> defining common use-cases for external Bro data consumption.
Dominik, wasn't the original idea
On Tue, Aug 21, 2018 at 8:54 AM Dominik Charousset
wrote:
> This raises a couple of questions. Primarily: where can Broker users learn
> the layouts to interpret received data?
broker/bro.hh is basically all there is right now. e.g. if you
construct a broker::bro::Event from a received
14 matches
Mail list logo
