Re: [Bro-Dev] "delete" of entire tables/sets

On Thu, Dec 6, 2018 at 4:09 PM Vern Paxson wrote: > This means I have to first build up a *separate* vector of all the indexes, > then iterate over that to remove them. I guess re-assigning a new, empty table to the variable could be analogous to deleting all entries and also avoids the

Re: [Bro-Dev] Broker cluster discovery and load balancing

On Thu, Dec 6, 2018 at 11:16 AM Hosom, Stephen M wrote: > I have looked at the implementations of publish_hrw and publish_rr in bro. I > could easily implement those features in my application if that is the > recommended way to handle this issue. There's been some ideas on pushing loading

Re: [Bro-Dev] Consistent error handling for scripting mistakes

On Tue, Nov 13, 2018 at 4:32 PM Vern Paxson wrote: > > I like what you & Robin sketch. FWIW, it's hard for me to get excited > over the issue of leaks-in-the-face-of-error-recovery. Yeah, it's not great, but intention would still be to eventually fix the leakage problem, too. Anyway, made a

Re: [Bro-Dev] Consistent error handling for scripting mistakes

On Mon, Nov 12, 2018 at 1:44 PM Jim Mellander wrote: > I was tinkering with the sumstats code, and inadvertantly deleted the final > "}" closing out the last function. When running the code, the misleading > error message is received: Yes, that's a bit of a different topic, but still tracked

[Bro-Dev] Consistent error handling for scripting mistakes

Trying to broaden the scope of: https://github.com/bro/bro/issues/208 I recently noticed there's a range of behaviors in how various scripting mistakes are treated. They may (1) abort, as in case of bad subnet mask or incompatible vector element assignment (2) skip over evaluating

Re: [Bro-Dev] attributes & named types

On Tue, Nov 6, 2018 at 3:00 PM Vern Paxson wrote: > I think what we need to do is rethink the basic architecture/structure of > attributes. In particular, types in general (not just named types) should > be able to have attributes associated with them. The attributes associated > with an

[Bro-Dev] Any 2.6 release blockers?

Anyone have any last minute issues that would block the 2.6 release? Or any remaining changes (bug fixes) they want to still try to get into it? - Jon ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] bro-pkg Bro version requirements

On Tue, Oct 16, 2018 at 2:23 PM Vlad Grigorescu wrote: >> @if ( Version::at_least("2.6") ) >> event ssl_client_hello(c: connection, version: count, record_version:count, >> possible_ts: time, client_random: string, session_id: string, ciphers: >> index_vec, comp_methods: vector of count) =1 >>

Re: [Bro-Dev] Intermittent bro crashes

On Tue, Oct 16, 2018 at 9:38 AM McMullan, Tim wrote: > We have been seeing occasional core dumps from bro, currently running on > 2.5-870. May be nice to try most recent master version to see if it still pops up. > We’ve tried a few things to reproduce it on-demand but haven’t been >

Re: [Bro-Dev] SSH Capabilities Bug: Fix for 2.6?

On Mon, Oct 15, 2018 at 3:33 PM Vlad Grigorescu wrote: > The SSH Capabilities record has the following field, which is being set > incorrectly: > >> ## Are these the capabilities of the server? >> is_server: bool; > >> result->Assign(6, new Val(${msg.is_orig},

Re: [Bro-Dev] Building Bro with openssl 1.1.1

On Fri, Oct 12, 2018 at 6:33 PM Craig Leres wrote: > Is openssl 1.1.1 support on the roadmap? Bro 2.6 will support OpenSSL 1.1. Possibly only a couple more weeks of beta remain before that release. - Jon ___ bro-dev mailing list bro-dev@bro.org

Re: [Bro-Dev] Event handling within script modules

On Thu, Oct 11, 2018 at 3:53 AM Steffen Haas wrote: > # Problem 1: Event Handling within module > The handler for "my_event" is not invoked as long as it is defined > within the same module "module_test". > - Workaround a): Do not use modules at all > - Workaround b): Define the handler in a

Re: [Bro-Dev] Module prefix in sending and receiving Broker events

On Thu, Sep 27, 2018 at 3:47 AM Matthias Vallentin wrote: > I hope this illustrates the issue a bit better. Thanks, that helps. Underlying issue/pitfall still seems related to module interactions with event handler definition/declaration, so added a link back from [#163]. Namely, seems you

Re: [Bro-Dev] 2.6-beta build error

On Fri, Sep 21, 2018 at 12:00 PM Azoff, Justin S wrote: > I think I remember the older geoip bifs logging this as well, but only once > at startup, not once per call to lookup_location. Yes, looks like it used to be that way. Changed via [1] now in master. - Jon [1]

Re: [Bro-Dev] 2.6-beta build error

On Thu, Sep 20, 2018 at 2:14 PM Michael Dopheide wrote: > > bro.bif:3630:47: error: cannot convert ‘stat’ to ‘__dev_t {aka long unsigned > int}’ in initialization Thanks, it's fixed by a simple patch [1] now in master. - Jon [1]

[Bro-Dev] 2.6-beta and 2.7 parallel development

Due to how our version tagging works and to prevent things destined for 2.7 getting incorrectly labelled as part of 2.6-beta in CHANGES, I'm going to start a dev/2.7 branch. * Any new commits that should be included for final 2.6: merge to master * Any new commits that are meant for 2.7: merge to

Re: [Bro-Dev] JIRA to GitHub ticket migration plan

On Tue, Sep 18, 2018 at 7:35 AM Vlad Grigorescu wrote: > > On Sat, Sep 15, 2018 at 1:28 AM Robin Sommer wrote: >> >> Are Jenkins and Coverity already pulling from GitHub? > > No, I thought Jenkins was pushing to Coverity. Travis is triggered by commits on GitHub and looks like it has a monthly

Re: [Bro-Dev] JIRA to GitHub ticket migration plan

On Fri, Sep 14, 2018 at 9:54 AM Robin Sommer wrote: > > real question here is if we want to switch repositories before or > after 2.6? Forgot to say that my sentiment is conditional -- if I find that it's not disruptive/risky to the release process (e.g. whether I need to make commits within bro

Re: [Bro-Dev] JIRA to GitHub ticket migration plan

On Fri, Sep 14, 2018 at 9:54 AM Robin Sommer wrote: > > On Thu, Sep 13, 2018 at 19:39 -0500, Jon Siwek wrote: > > > A preview of what migrated issues will look like along with new labeling > > scheme: > > The only thing I noticed is that the labels are > quite lo

[Bro-Dev] JIRA to GitHub ticket migration plan

A preview of what migrated issues will look like along with new labeling scheme: https://github.com/jsiwek/test/issues The selection strategy of which tickets to migrate is something like "anything that is a reproducible bug or a simple/uncontroversial enhancement". Any tickets not in that

Re: [Bro-Dev] Module prefix in sending and receiving Broker events

On Thu, Sep 13, 2018 at 4:28 AM Matthias Vallentin wrote: > > > Does the > > suggestion [1] to always explicitly scope events by their > > namespace/module address your problem? > > That's what I thought would work, but it's the opposite: when I add > the module name as prefix, Bro silently

Re: [Bro-Dev] Bro 2.6-beta plans

On Wed, Sep 12, 2018 at 11:58 AM Alan Commike wrote: > Are there still parent/child processes handling comms/work? No. Single process, configurable number of threads (default 1). > Is there a mechanism today for per node type tuneables? One should be able to use an @if directive [1] to tune

Re: [Bro-Dev] Module prefix in sending and receiving Broker events

On Wed, Sep 12, 2018 at 10:09 AM Matthias Vallentin wrote: > > When I receive events from Bro via Broker, they have the prefix of the > enclosing module: > > module Foo; > > event foo() { ... } > event bar() { ... } > > When I publish "foo" via Broker, it arrives as "Foo::foo".

Re: [Bro-Dev] Bro 2.6-beta plans

On Wed, Sep 12, 2018 at 9:18 AM Azoff, Justin S wrote: > > Just finished the migration to master across the board, and it's looking > REALLY good. Great, thanks for helping test and provide performance data. > The manager box in this cluster only runs the manager and logger processes, > no

Re: [Bro-Dev] How to use Broker::Data in an event handler?

On Tue, Sep 11, 2018 at 5:52 AM Matthias Vallentin wrote: > One more question: how would I capture a default-constructed > broker::Data() in a case statement? This would happen when I publish > just "None" on the Python side. In Bro, it shows up on the command > line as "broker::data{nil}".

Re: [Bro-Dev] switch/case type recognition

On Tue, Sep 11, 2018 at 5:42 AM Matthias Vallentin wrote: > I am aware that this is a somewhat pathological case, because 'case type > any' is probably equivalent to the 'default' case. A 'vector of any' also qualifies as an 'any', so while the error message of "duplicate case label" could

Re: [Bro-Dev] attributes & named types

On Mon, Sep 10, 2018 at 5:08 PM Vern Paxson wrote: > > At least that's how I think it's currently working, so are you going > > start using TypeType as a means of type aliasing in addition to adding > > attributes to them? > > Not quite. Rather, the type of "mytype" would be a TypeType, which

Re: [Bro-Dev] attributes & named types

On Mon, Sep 10, 2018 at 4:16 PM Vern Paxson wrote: > Seems simplest to me to have TypeType's (and only those) include attributes. > The rest of it is easy to do from there. We could also do this with a > separate NameType, but I don't see what that gains, since TypeType's already > come into

Re: [Bro-Dev] How to use Broker::Data in an event handler?

On Mon, Sep 10, 2018 at 8:01 AM Matthias Vallentin wrote: > > I'm trying to figure out if/how it is possible to use Broker::Data in an > event handler as follows: > > event foo(x: Broker::Data) > { > print x; > } No, but you can try to use 'any' instead of 'Broker::Data'.

Re: [Bro-Dev] Bro 2.6-beta plans

On Fri, Sep 7, 2018 at 3:41 PM Azoff, Justin S wrote: > One thing I'm still seeing when I switch from an old version to latest master > is that huge spike > in Content switches/interrupts and cpu spent in the kernel. I just updated the default tuning parameters for CAF's scheduling policy and

Re: [Bro-Dev] Bro 2.6-beta plans

On Thu, Sep 6, 2018 at 3:40 PM Azoff, Justin S wrote: >8842 > broker::topic+broker::internal_comma...@u32.bro/known/certs/<$>/data/clone Thanks, there was an unintended forwarding loop in data store communication. It's fixed in master now, but I've also just reverted to generally

Re: [Bro-Dev] Bro 2.6-beta plans

On Thu, Sep 6, 2018 at 3:14 PM Azoff, Justin S wrote: > I tested an almost stock local.bro (a few additional things disabled) and saw > the same thing. > > fa7fa5aa is fine, but with 452eb0cb everything is working really hard to do > something. Thanks for that, I'll start looking into it,

Re: [Bro-Dev] Bro 2.6-beta plans

On Thu, Sep 6, 2018 at 2:47 PM Azoff, Justin S wrote: > I just got 2 clusters upgraded from > > fa7fa5aa to > 452eb0cb > > And now everything is broken.. > > cpu and memory are through the roof across the board, as well as network > traffic, but it's not logging much. > > I may have created a

Re: [Bro-Dev] Bro 2.6-beta plans

On Wed, Sep 5, 2018 at 5:43 PM Michael Dopheide wrote: > > To be clear, Cluster::relay_rr() is gone forever? I’ll need to rewrite some > policies, but also update the blog to be correct. Yes. - Jon ___ bro-dev mailing list bro-dev@bro.org

Re: [Bro-Dev] attributes & named types

On Wed, Sep 5, 2018 at 6:47 PM Vern Paxson wrote: > > > I think the right solution for this is to introduce a new BroType called > > a NamedType. A NamedType has an identifier, an underlying BroType, and a > > set of attributes. When a NamedType is constructed ... > > Huh, turns out there's

[Bro-Dev] Bro 2.6-beta plans

There's no significant code changes/features planned to get added to the master branch from now until the 2.6-beta gets released (maybe in about a week). Until that happens, please help test the latest master branch and provide any feedback about how it's working if you can. - Jon

Re: [Bro-Dev] [Bro-Commits] [git/bro] master: Allow loading policy/protocols/smb once again (57a505b0e)

On Thu, Aug 30, 2018 at 5:06 PM Johanna Amann wrote: > > To pick up the idea that you mentioned before - do we also want to make > the new policy/protocols/smb/__load__.bro trigger a reporter warning > that it is deprecated? Sounds right -- unlikely it will ever be used in the future and should

Re: [Bro-Dev] Compatibilty script for policy/protocols/smb?

On Thu, Aug 30, 2018 at 9:50 AM Azoff, Justin S wrote: > fatal error in /bro/share/bro/site/local.bro, line 88: can't open > /bro/share/bro/policy/protocols/smb/__load__.bro > > I see in NEWS we have > > - The SMB scripts in policy/protocols/smb are now moved into > base/protocols/smb > and

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/johanna/tls-more-data: Update NEWS for ssl changes. (3c7c60cf6)

On Wed, Aug 29, 2018 at 11:02 AM Johanna Amann wrote: > I actually tested it - and it works fine with old versions as long as > you use the @if this way round. Ah, tricky. I can see how that would work now, thanks for clarifying. - Jon ___ bro-dev

Re: [Bro-Dev] [Bro-Commits] [git/bro] topic/johanna/tls-more-data: Update NEWS for ssl changes. (3c7c60cf6)

On Tue, Aug 28, 2018 at 6:35 PM Johanna Amann wrote: > + If you use these events, you can make your scripts work on old and new > versions > + of Bro by wrapping the event definition in an @if, for example: > + > +@if ( Version::at_least("2.6") || ( Version::number == 20500 && >

Re: [Bro-Dev] Jira filter results

On Tue, Aug 28, 2018 at 12:48 PM Johanna Amann wrote: > "The filter configured for this gadget could not be retrieved. Please > verify it is still valid on the issue navigator.". Should be showing merge requests again. - Jon ___ bro-dev mailing list

Re: [Bro-Dev] libmaxminddb configure issue

Thanks, does look like these wouldn't work as intended for CMake < 3.3, but I've merged Daniel's patch in to master now. - Jon On Thu, Aug 23, 2018 at 3:36 PM Michael Dopheide wrote: > > Yeah, I just figured that out myself and rebuilt... > > bro -e "print lookup_location(8.8.8.8);" >

Re: [Bro-Dev] Broker data layouts

On Thu, Aug 23, 2018 at 8:32 AM Dominik Charousset wrote: > I’m a bit hesitant to rely on this header at the moment, because of: > > /// A Bro log-write message. Note that at the moment this should be used only > /// by Bro itself as the arguments aren't publicly defined. > > Is the API stable

Re: [Bro-Dev] Broker data layouts

On Tue, Aug 21, 2018 at 1:09 PM Robin Sommer wrote: > Also, this question is about events, not logs, right? Logs have a > different wire format and they actually come with meta data describing > their columns. Though the Broker data corresponding to log entry content is also opaque at the

Re: [Bro-Dev] Broker data layouts

On Tue, Aug 21, 2018 at 8:54 AM Dominik Charousset wrote: > This raises a couple of questions. Primarily: where can Broker users learn > the layouts to interpret received data? broker/bro.hh is basically all there is right now. e.g. if you construct a broker::bro::Event from a received

Re: [Bro-Dev] Use of 'any' type

In the master branch, there are also type checking/casting 'is' and 'as' operators [1] and type-based switch statement [2] that may be be useful. - Jon [1] https://www.bro.org/sphinx-git/script-reference/operators.html [2]

Re: [Bro-Dev] reproducible segfault in master branch

On Tue, Aug 14, 2018 at 10:26 PM Michael Dopheide wrote: > > Somehow related to Broker stores and/or casting. You'll get a better error message/behavior now using: https://github.com/bro/bro/commit/f336c8c710bdeb41eb0aba88967ee90da24848b2 But ultimately, you likely want to do something like

Re: [Bro-Dev] Broker::publish API

On Tue, Aug 14, 2018 at 12:09 PM Jan Grashöfer wrote: > > On 13/08/18 18:24, Jon Siwek wrote: > > Old Worker: > > > >Cluster::relay_rr(Cluster::proxy_pool, my_event); > > > > New Worker: > > > >Broker::publish(Cluster::rr_topic(Cluster::pr

Re: [Bro-Dev] Broker::publish API

On Tue, Aug 14, 2018 at 10:13 AM Robin Sommer wrote: > One more question: what about raising published events locally as well > if the sending node is subscribed to the topic? I'm kind of torn on > that. I don't think we want that as a default, but perhaps as an > option, either with the

Re: [Bro-Dev] Writing SumStats plugin

On Tue, Aug 7, 2018 at 5:15 PM Jim Mellander wrote: > Incidentally, I think theres a bug in the observe() function: > > These two lines are run in the loop thru the reducers: >if ( r?$normalize_key ) > key = r$normalize_key(copy(key)); > which has the

Re: [Bro-Dev] Broker::publish API

intelligence framework. Either for > querying it similar to workers of for managing purposes. If possible, it > could be beneficial for such an application to receive only the relevant > parts of cluster communication. > > On 10/08/18 17:52, Jon Siwek wrote: > > (1) if the even

Re: [Bro-Dev] Broker::publish API

On Fri, Aug 10, 2018 at 11:47 AM Azoff, Justin S wrote: > If relay is removed how does a script writer efficiently get an event from > one worker (or manager) > to all of the other workers? Old Worker: Cluster::relay_rr(Cluster::proxy_pool, my_event); New Worker:

Re: [Bro-Dev] Broker::publish API

On Fri, Aug 10, 2018 at 8:29 AM Jan Grashöfer wrote: > > Let me try to phrase it differently: If there's already a topic for a > > use case, it's better to use it. That's easier and less error-prone. > > So if, e.g., I want to send my script's data to all workers, > > publishing to

Re: [Bro-Dev] Broker::publish API

On Thu, Aug 9, 2018 at 1:29 PM Robin Sommer wrote: > > (1) enable the "explicit/manual" forwarding by default? > > Coming from that assumption above, I'd say yes here, doing it like you > suggest: differentiate between forwarding and locally raising an event > by topic. Maybe instead of adding

Re: [Bro-Dev] Broker::publish API

On Wed, Aug 8, 2018 at 2:50 PM Robin Sommer wrote: > > * enable message forwarding by default (meaning re-implement the one > > or two subscription patterns that might create a cycle) > > Haven't quite made up my mind on this one. In principlel yes, but > right now a host needs to be subscribed

Re: [Bro-Dev] Broker::publish API

On Wed, Aug 8, 2018 at 10:55 AM Robin Sommer wrote: > That's actually something I realized yesterday: we don't have direct > worker-to-worker communication right now, correct? A worker cannot > just publish to "bro/cluster/workers". Right, here's a crude graphic of the cluster layout from the

Re: [Bro-Dev] Broker::publish API

On Wed, Aug 8, 2018 at 10:53 AM Robin Sommer wrote: > > Yeah, I realize that. A direct port of the old logic was of course the > goal so far, with the drawbacks of that approach accepted & > understood. That's what's in place now; that's great and exactly as > planned. We can get 2.6 out this

Re: [Bro-Dev] Broker::publish API

On Mon, Aug 6, 2018 at 3:00 PM Robin Sommer wrote: > Overall I have to say I found it pretty hard to follow this all > because we don't have much consistency right now in how scripts > structure their communication. That's not surprising, given that we're > just starting to use all this, but it

Re: [Bro-Dev] Broker::publish API

On Mon, Aug 6, 2018 at 1:57 PM Robin Sommer wrote: > I have another question about this specific case: we use relay_rr() > only for sending Intel::insert_indicator. Intel::remove_indicator gets > published normally through auto_publish(). Why the difference? Potentially no reason other than no

Re: [Bro-Dev] Broker::publish API

On Fri, Aug 3, 2018 at 12:22 PM Robin Sommer wrote: > On Fri, Jul 27, 2018 at 10:39 -0700, I wrote: > > > Broker::relay(change_topic, change_topic, Config::cluster_set_option, ID, > > val, location); > > Can somebody remind me what the use-case is for changing the topic on > relay? Grepping

Re: [Bro-Dev] Broker::publish API

On Mon, Jul 30, 2018 at 12:16 PM Robin Sommer wrote: > > I'd be more comfortable if one could automate answering the question: > > "if I add a subscription to a given node in the network, will I create > > a cycle?". > > Hmm ... What about a test mode where we'd spin up a dummy cluster >

Re: [Bro-Dev] Broker::publish API

On Mon, Jul 30, 2018 at 11:02 AM Robin Sommer wrote: > True, although it's not cycles in the connection topology that matter, > it's cycles in topic subscriptions. Right, good point. > I need to think about this a bit > more (and I need to remind myself how our topics currently look like) I

Re: [Bro-Dev] Broker::publish API

On Fri, Jul 27, 2018 at 7:30 PM Azoff, Justin S wrote: > > > > On Jul 27, 2018, at 6:10 PM, Jon Siwek wrote: > > > > On Fri, Jul 27, 2018 at 3:55 PM Azoff, Justin S wrote: > > > >> I do agree that there's room for a lot of simplification, for exampl

Re: [Bro-Dev] Broker::publish API

On Fri, Jul 27, 2018 at 3:55 PM Azoff, Justin S wrote: > I do agree that there's room for a lot of simplification, for example a > worker broadcasting a message efficiently to all > other workers needs to do something like this from the docs: > > Cluster::relay_rr(Cluster::proxy_pool,

Re: [Bro-Dev] Broker::publish API

On Fri, Jul 27, 2018 at 12:40 PM Robin Sommer wrote: > I'm wondering if we should give it another try to simply this API > while we still can (i.e., before 2.6 goes out). To me, the most > intuitive publish operation is "send to topic T and propagate to > everybody subscribed to that topic". I'd

Re: [Bro-Dev] error compiling master

On Tue, Jul 10, 2018 at 2:10 PM Aashish Sharma wrote: > [ 96%] Building CXX object > libcaf_openssl/CMakeFiles/libcaf_openssl_shared.dir/src/manager.cpp.o > clang: warning: argument unused during compilation: '-L/lib' >

Re: [Bro-Dev] patterns and &&/|| vs. &/| operators

On Thu, Jun 21, 2018 at 4:25 PM Vern Paxson wrote: > > > though maybe p1 + p2 would be even better at expressing that > > concatenation is happening? > > I think this is somewhat problematic, since '+' already has a > regular-expression meaning which is different. In addition, '&' is > a more

Re: [Bro-Dev] v += e (Re: set and vector operators)

On Thu, Jun 21, 2018 at 4:23 PM Vern Paxson wrote: > > > > (3) Implement "v += e" to mean "append the element e to the vector v". > > > > Do we want to do this now, or should we potentially wait a release-cycle > > with it (to prevent the situation where v + e and v+= e means something > >

Re: [Bro-Dev] patterns and &&/|| vs. &/| operators

On Thu, Jun 21, 2018 at 1:34 PM Vern Paxson wrote: > > > The meaning of "p1 & p2" would be "yields a pattern that matches both > > p1 and p2" versus the meaning of "p1 && p2" currently being "yields a > > pattern that matches a p1 followed by a p2" ? > > No, p1 & p2 would be the new way to

Re: [Bro-Dev] set and vector operators

On Fri, Jun 15, 2018 at 7:40 PM Vern Paxson wrote: > (1) Add bitwise operators on "count" variables for &, | and ~. Yeah, looks like everyone was in favor of those. > (2) Deprecate element-wise arithmetic operations on vectors, such > as "v * 3" meaning "multiply each element of v by 3".

Re: [Bro-Dev] patterns and &&/|| vs. &/| operators

On Tue, Jun 19, 2018 at 1:21 PM Vern Paxson wrote: > Proposal: as part of adding bitwise &/| operators for counts, I'll > also implement &/| operators for patterns, and remove the current > &&/|| functionality. The meaning of "p1 & p2" would be "yields a pattern that matches both p1 and p2"

Re: [Bro-Dev] Performance Issues after the fe7e1ee commit?

On Tue, Jun 12, 2018 at 4:26 PM Robin Sommer wrote: > One question about Broker's endpoint::advance_time(): that's locking > on each call when in pcap mode, i.e., once per packet. Could we limit > that to cases where something actually needs to be done? Quick idea: > use an atomic<> for

Re: [Bro-Dev] Performance Issues after the fe7e1ee commit?

On Mon, Jun 11, 2018 at 6:30 PM Robin Sommer wrote: > Confirmed that I'm still seeing that difference even when using the > options to turn the stores. Tried it on two different Fedora 28 > systems, with similar results. I had previously tried on a Fedora 28 VM w/ 4 CPUs and didn't reproduce

Re: [Bro-Dev] Performance Issues after the fe7e1ee commit?

On Fri, Jun 8, 2018 at 10:23 AM Robin Sommer wrote: > > > # zcat 2009-M57-day11-18.trace.gz | time bro -r - tests/m57-long.bro > > Known::use_host_store=F Known::use_service_store=F > > Known::use_cert_store=F > > That indeed gets it way down, though still not back to the same level > on my box:

Re: [Bro-Dev] Performance Issues after the fe7e1ee commit?

On Thu, Jun 7, 2018 at 12:21 PM Robin Sommer wrote: > > Hmm, I'm still seeing much larger runtimes on that M57 trace using our > test configuration, even with yesterday's change: > > ; Master, pre-Broker > # zcat 2009-M57-day11-18.trace.gz | time bro -r - tests/m57-long.bro > 73.72user 7.90system

Re: [Bro-Dev] Performance Issues after the fe7e1ee commit?

On Wed, Jun 6, 2018 at 2:10 PM Azoff, Justin S wrote: > I haven't noticed a huge difference on a real multi process cluster, at least > not a 30x change, so this is odd that pcap processing is so much slower. > Especially since broker should be completely disabled when pcap files are > being

Re: [Bro-Dev] Performance Issues after the fe7e1ee commit?

On Wed, Jun 6, 2018 at 12:01 PM McMullan, Tim wrote: > Is the Bro development team still optimizing the Broker/Actor framework? Yes, in the sense that optimizations will be done according to feedback. No, in the sense that there was no one actively looking into it at the moment you asked. >

Re: [Bro-Dev] bro-devel package?

On Thu, May 24, 2018 at 2:50 PM, Vlad Grigorescu wrote: > There are a couple of cases where I think it'd be useful to have a bro-devel > package -- a package that I can install on a system, and then be able to > build plugins against Bro. (This is the same model as other *-devel >

Re: [Bro-Dev] Broker has landed in master, please test

On 5/23/18 3:12 PM, Michael Dopheide wrote: > For here though, can you elaborate on the going down to one proxy?  My > understanding still isn't strong, but that seems to be opposed to the > idea of using Cluster::publish_hrw to spread memory across proxies. The idea is to try starting with

Re: [Bro-Dev] Moving to GitHub?

On 5/18/18 11:11 AM, Robin Sommer wrote: > What I was envisioning is more or less a clean slate: we'd migrate > over a few tickets, but essentially we'd start with an empty list. I > realize that sounds pretty harsh. However, I hardly ever see any > activity on older tickets in JIRA, and I

Re: [Bro-Dev] Moving to GitHub?

On 5/17/18 6:27 PM, Robin Sommer wrote: > That may be a bit too broad though. How about "still valid and either > (1) quite important or (2) something we expect will be addresses > reasonably soon"? We have many old tickets that are technically still > valid but unlikely to see any work anytime

Re: [Bro-Dev] Moving to GitHub?

On 5/15/18 7:19 PM, Robin Sommer wrote: > What do people think? Any support, or concerns? Yeah, generally in favor with some comments: * For porting over JIRA tickets to GitHub, "most recent" doesn't seem like a good metric to use. e.g. BIT-1829 (pcap triggering assertion in binpac) seems

Re: [Bro-Dev] Broker data store use case and questions

On 5/11/18 6:33 PM, Michael Dopheide wrote: > First, can Cluster::default_master_node be changed to default to the > name of the current manager node rather than specifying the name as > 'manager'? Maybe. I'll try having broctl communicate that to Bro via a new environment variable. >

Re: [Bro-Dev] Broker data store use case and questions

On 5/11/18 1:38 PM, Azoff, Justin S wrote: > >> On May 11, 2018, at 10:13 AM, Jon Siwek <jsi...@corelight.com> wrote: >> >> >> There's no check against the local cache to first see if the key exists >> as going down that path leads to race condit

Re: [Bro-Dev] Broker data store use case and questions

On 5/11/18 10:15 AM, Michael Dopheide wrote: > Let me clarify point 4, my goal is just to keep the knownhosts data > persistent across restarts.  (Or any data set in the general case.)  So > if HRW is the best way to keep data in memory I need a way to write it > out to disk on Bro exit so I

Re: [Bro-Dev] Broker data store use case and questions

On 5/10/18 3:53 PM, Michael Dopheide wrote: > 1) My initial gut feeling was that all of the when() calls for insertion > could get really expensive on a brand new cluster before the store is > populated. I've not tried to explicitly measure differences yet, though my hunch is that the

Re: [Bro-Dev] Final Broker branch testing

On 5/2/18 9:59 AM, Johanna Amann wrote: >>> (3) I need to try to hack our CMake system more to try to get back down >>> to 2.8.12 while still being able to embed CAF. I think (hope!) I was mistaken and everything already works with 2.8.12 (structure of CMake docs previously led me to think it

Re: [Bro-Dev] set and vector operators

On 4/30/18 9:54 AM, Robin Sommer wrote: > One additional piece of context here: That vector(...) syntax could > then be used more broadly in the sense of creating a different > semantic context for the operations inside. That kind of opens up a > whole new set of of type-specific operator

Re: [Bro-Dev] set and vector operators

On 4/30/18 9:10 AM, Vern Paxson wrote: > The question then was what would be the new "v op e" syntax. > The best we could come up with (which we both found not-too-awful) is > "vector(v op e)". Wrapped in "vector(...)", the operation becomes the > current semantics (apply "op e" separately to

Re: [Bro-Dev] Final Broker branch testing

On 4/26/18 4:30 PM, Johanna Amann wrote: > It might be. I am honestly not sure - I suspect that this still will > mean that some places might not be able to easily use Bro > anymore--adding external package sources does not seem to be a viable > option everywhere. They could still build

Re: [Bro-Dev] set and vector operators

On 4/26/18 3:43 PM, Vern Paxson wrote: >> E.g. say you come back to some code after a few months and see "foo += >> 1". Not obvious what 'foo' is anymore. > > I don't think it's reasonable to have the bar be "can you tell what's going > on in isolation". It should include consideration of

Re: [Bro-Dev] Final Broker branch testing

On 4/26/18 2:04 PM, Johanna Amann wrote: > With this change, we Bro cannot be compiled out of the Box on > RedHat/Centos 7 anymore. Since that is the latest release of RedHat and > probably used in production by quite a few people a potentially > significant amount of people might not be

Re: [Bro-Dev] How to deal with stale branches?

On 4/26/18 11:06 AM, Vlad Grigorescu wrote: > I'm torn between deleting the branches, in an effort to not clog up git > with unneeded branches, and leaving them around or perhaps archiving > them somewhere, in order to not completely lose the work in case it's of > value to someone down the

Re: [Bro-Dev] set and vector operators

On 4/26/18 11:29 AM, Vern Paxson wrote: >> Just one more thing still: I'm actually feeling pretty strongly >> against having multiple different operators for the same operation >> (set union, set addition/removal). I'm maybe convincing myself that it's at least not that useful or there's

[Bro-Dev] Final Broker branch testing

The latest version of the new Broker-ized cluster/communication system for Bro in 'topic/actor-system' branch is wrapping up and, in my opinion, ready to be merged into Bro's 'master' branch. However, since it's such a big change, I'd like a last round of feedback before merging. If you want

Re: [Bro-Dev] set and vector operators

On 4/26/18 12:19 AM, Vern Paxson wrote: > Hmmm thinking about it, we can get away with '&' with minimal keyword > conflict because there's such an easy (and natural-to-presume) fix - > namely, rather than "x" you use "x & attrkeyword". Now > there's no problem, since the lexer only recognizes

Re: [Bro-Dev] scheduling events vs using _func ?

On 4/13/18 6:14 PM, Aashish Sharma wrote: > I have a aggregation policy where I am trying to keep counts of number of > connections an IP made in a cluster setup. > > For now, I am using table on workers and manager and using expire_func to > trigger worker2manager and manager2worker events. >

Re: [Bro-Dev] Overload Bro Events

On 4/12/18 2:23 PM, DW wrote: > Hello, > > it is possible to overload events in Bro based on the event-parameter > and trigger the "right" event based on the given parameter? > > E.g. I would define events like this > > event overload%(c: connection%); > event overload%(c: connection, h:

Re: [Bro-Dev] Broker port status

On Thu, Mar 8, 2018 at 6:11 PM, Azoff, Justin S wrote: >> redef Known::use_host_store = F; >> redef Known::use_cert_store = F; >> redef Known::use_device_store = F; >> redef Known::use_service_store = F; > but if you load the script first, the > > @if (

Re: [Bro-Dev] Broker port status

On Thu, Mar 8, 2018 at 5:07 PM, Azoff, Justin S wrote: > One thing I notice is that the traffic to/from the manager box and cpu has > increased quite a bit. > > Ignoring the large CPU spikes from building the new branch just before the > switch at 15:30, the overall cpu

  1   2   3   4   5   6   7   >