Re: [Bro-Dev] libmaxminddb configure issue
Thanks, does look like these wouldn't work as intended for CMake <
3.3, but I've merged Daniel's patch in to master now.
- Jon
On Thu, Aug 23, 2018 at 3:36 PM Michael Dopheide wrote:
>
> Yeah, I just figured that out myself and rebuilt...
>
> bro -e "print lookup_location(8.8.8.8);"
> [country_code=US, region=, city=,
> latitude=37.751, longitude=-97.822]
>
> Looks like you'll have the same issue with LibKRB5_FOUND (I didn't look for
> others).
>
> -Dop
>
> On Thu, Aug 23, 2018 at 3:10 PM, Thayer, Daniel N
> wrote:
>>
>> Could you try the following patch and let me know if it works for you:
>>
>> --- a/CMakeLists.txt
>> +++ b/CMakeLists.txt
>> @@ -134,7 +134,7 @@ include_directories(BEFORE
>>
>> set(USE_GEOIP false)
>> find_package(LibMMDB)
>> -if (LibMMDB_FOUND)
>> +if (LIBMMDB_FOUND)
>> set(USE_GEOIP true)
>> include_directories(BEFORE ${LibMMDB_INCLUDE_DIR})
>> list(APPEND OPTLIBS ${LibMMDB_LIBRARY})
>>
>>
>> --
>>
>> From: bro-dev-boun...@bro.org [bro-dev-boun...@bro.org] on behalf of Michael
>> Dopheide [dophe...@es.net]
>>
>> Sent: Thursday, August 23, 2018 2:16 PM
>>
>> To:
>>
>> Subject: [Bro-Dev] libmaxminddb configure issue
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Johanna mentioned to me that libmaxminddb should be working now in master...
>>
>>
>>
>>
>>
>> So far I haven't been able to get 'configure' to find it, neither with the
>> OS packages nor by installing libmaxminddb in /usr/local/ and specifying
>> --with-geoip.
>>
>>
>>
>>
>>
>> This is CentOS 7.5.
>>
>>
>>
>>
>>
>> -Dop
>>
>>
>>
>>
>>
>>
>>
>
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] libmaxminddb configure issue
Yeah, I just figured that out myself and rebuilt...
bro -e "print lookup_location(8.8.8.8);"
[country_code=US, region=, city=,
latitude=37.751, longitude=-97.822]
Looks like you'll have the same issue with LibKRB5_FOUND (I didn't look for
others).
-Dop
On Thu, Aug 23, 2018 at 3:10 PM, Thayer, Daniel N
wrote:
> Could you try the following patch and let me know if it works for you:
>
> --- a/CMakeLists.txt
> +++ b/CMakeLists.txt
> @@ -134,7 +134,7 @@ include_directories(BEFORE
>
> set(USE_GEOIP false)
> find_package(LibMMDB)
> -if (LibMMDB_FOUND)
> +if (LIBMMDB_FOUND)
> set(USE_GEOIP true)
> include_directories(BEFORE ${LibMMDB_INCLUDE_DIR})
> list(APPEND OPTLIBS ${LibMMDB_LIBRARY})
>
>
> --
>
> From: bro-dev-boun...@bro.org [bro-dev-boun...@bro.org] on behalf of
> Michael Dopheide [dophe...@es.net]
>
> Sent: Thursday, August 23, 2018 2:16 PM
>
> To:
>
> Subject: [Bro-Dev] libmaxminddb configure issue
>
>
>
>
>
>
>
>
>
> Johanna mentioned to me that libmaxminddb should be working now in
> master...
>
>
>
>
>
> So far I haven't been able to get 'configure' to find it, neither with the
> OS packages nor by installing libmaxminddb in /usr/local/ and specifying
> --with-geoip.
>
>
>
>
>
> This is CentOS 7.5.
>
>
>
>
>
> -Dop
>
>
>
>
>
>
>
>
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] libmaxminddb configure issue
Could you try the following patch and let me know if it works for you:
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -134,7 +134,7 @@ include_directories(BEFORE
set(USE_GEOIP false)
find_package(LibMMDB)
-if (LibMMDB_FOUND)
+if (LIBMMDB_FOUND)
set(USE_GEOIP true)
include_directories(BEFORE ${LibMMDB_INCLUDE_DIR})
list(APPEND OPTLIBS ${LibMMDB_LIBRARY})
--
From: bro-dev-boun...@bro.org [bro-dev-boun...@bro.org] on behalf of Michael
Dopheide [dophe...@es.net]
Sent: Thursday, August 23, 2018 2:16 PM
To:
Subject: [Bro-Dev] libmaxminddb configure issue
Johanna mentioned to me that libmaxminddb should be working now in master...
So far I haven't been able to get 'configure' to find it, neither with the OS
packages nor by installing libmaxminddb in /usr/local/ and specifying
--with-geoip.
This is CentOS 7.5.
-Dop
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] libmaxminddb configure issue
More info: *snip* libmaxminddb: false <--- THIS Kerberos: false gperftools found: true tcmalloc: true debugging: false jemalloc: false -- Configuring done -- Generating done -- Build files have been written to: /usr/local/src/bro/build [root@sec-gpu bro]# grep MMDB build/CMakeCache.txt LibMMDB_INCLUDE_DIR:PATH=/usr/include LibMMDB_LIBRARY:FILEPATH=/usr/lib64/libmaxminddb.so LibMMDB_ROOT_DIR:PATH=/usr //Details about finding LibMMDB FIND_PACKAGE_MESSAGE_DETAILS_LibMMDB:INTERNAL=[/usr/lib64/libmaxminddb.so][/usr/include][v()] //ADVANCED property for variable: LibMMDB_INCLUDE_DIR LibMMDB_INCLUDE_DIR-ADVANCED:INTERNAL=1 //ADVANCED property for variable: LibMMDB_LIBRARY LibMMDB_LIBRARY-ADVANCED:INTERNAL=1 //ADVANCED property for variable: LibMMDB_ROOT_DIR LibMMDB_ROOT_DIR-ADVANCED:INTERNAL=1 Clearly it found the libraries just fine... this is where my cmake debugging ability starts to fall apart... -Dop On Thu, Aug 23, 2018 at 2:16 PM, Michael Dopheide wrote: > Johanna mentioned to me that libmaxminddb should be working now in > master... > > So far I haven't been able to get 'configure' to find it, neither with the > OS packages nor by installing libmaxminddb in /usr/local/ and specifying > --with-geoip. > > This is CentOS 7.5. > > -Dop > > ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] libmaxminddb configure issue
Johanna mentioned to me that libmaxminddb should be working now in master... So far I haven't been able to get 'configure' to find it, neither with the OS packages nor by installing libmaxminddb in /usr/local/ and specifying --with-geoip. This is CentOS 7.5. -Dop ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Broker data layouts
> Dominik, wasn't the original idea for VAST to provide an event > description language that would create the link between the values > coming over the wire and their interpretation? Such a specification > could be auto-generated from Bro's knowledge about the events it > generates. We were actually thinking about auto-generating the schema. But broker::data simply has no meta information that we can use. Even distinguishing records/tuples from actual lists is impossible, because broker::vector is used for both. Of course we can make a couple of assumptions (the top-level vector is a record, for example), but then VAST users only ever can use type queries. In other words, they can only ask for IP addresses for example, but not specifically for originator IPs. In a sense, broker’s representation is an inverted JSON. In JSON, we have field names but no type information (everything is a string), whereas in broker we have (ambiguous) type information but no field names. :) >> Though the Broker data corresponding to log entry content is also >> opaque at the moment (I recall that was maybe for performance or >> message volume optimization), > > Yeah, but generally this is something I could see opening up. The log > structure is pretty straight-forward and self-describing, it'd be > mostly a matter of clean up and documentation to make that directly > accessible to external consumers I think. Events, on the other hands, > are semantically tied very closely to the scripts generating them, and > also much more diverse so that self-description doesn't really seem > feasible/useful. Republishing a relevant subset certainly sounds > better for that; or, if it's really a bulk feed that's desired, some > out-of-band mechanism to convey the schema information somehow. Opening that up would be great. However, our goal was to have Broker as a source for structured data that we can import in a generic fashion for later analysis. Of course that relies on a standard / convention / best practice for making schema programmatically accessible. Currently, it seems that we need a schema definition provided by the user offline. This will work as long as all published data for a given topic is uniform. Multiplexing multiple event types already makes things complicated, but it seems like this is actually the standard use case. OSQuery, for example, will generate different events that we than either need to separate into different topics or multiplex in a single topic but merge-in some meta information. And once we mix in meta information with actual data, a simple schema definition no longer cuts it. At worst, importing data from Broker requires a separate parser for each import format. > broker/bro.hh is basically all there is right now I’m a bit hesitant to rely on this header at the moment, because of: /// A Bro log-write message. Note that at the moment this should be used only /// by Bro itself as the arguments aren't publicly defined. Is the API stable enough on your end at this point to make it public? Also, there are LogCreate and LogWrite events. The LogCreate has the `fields_data` (a list of field names?). Does that mean I need to receive the LogCreate even first to understand successive LogWrite events? That would mean I cannot parse logs that had their LogCreate event before I was able to subscribe to the topic. Dominik ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Broker data layouts
On Thu, Aug 23, 2018 at 10:01 -0500, Jonathan Siwek wrote: > Yeah, that's one problem, but a bigger issue is you can't parse > LogWrite because the content is a serial blob whose format is another > thing not intended for public consumption. I guess my earlier comment might have been misleading: there's certaily work that needs to be done to open this up. Right now, it's probably not even realistic at all because we still have a work around in place in there that uses the old (non-Broker) serialization code for creating that blob. That was to get around a performance issue, and still needs to be addressed. As part of upgrading that, I think it can make sense to think about documenting the format we end up chosing. Robin -- Robin Sommer * Corelight, Inc. * ro...@corelight.com * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Broker data layouts
On Thu, Aug 23, 2018 at 15:32 +0200, Dominik Charousset wrote: > Does that mean I need to receive the LogCreate even first to > understand successive LogWrite events? I don't really see a way around that without substantially increasing volume. We could send LogCreate updates regularly, so that it's easier to synchronize with an ongoing stream. Robin -- Robin Sommer * Corelight, Inc. * ro...@corelight.com * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Broker data layouts
On Thu, Aug 23, 2018 at 8:32 AM Dominik Charousset wrote: > I’m a bit hesitant to rely on this header at the moment, because of: > > /// A Bro log-write message. Note that at the moment this should be used only > /// by Bro itself as the arguments aren't publicly defined. > > Is the API stable enough on your end at this point to make it public? The comment is just pointing out what was said about the log message formats being opaque at the moment. It's expected only Bro will be able to make sense of the content. > Also, there are LogCreate and LogWrite events. The LogCreate has the > `fields_data` (a list of field names?). Yeah, there's some field info in there: names, types, optionality. The type info in particularly doesn't seem good to treat as intended for public consumption. > Does that mean I need to receive the LogCreate even first to understand > successive LogWrite events? That would mean I cannot parse logs that had > their LogCreate event before I was able to subscribe to the topic. Yeah, that's one problem, but a bigger issue is you can't parse LogWrite because the content is a serial blob whose format is another thing not intended for public consumption. - Jon ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
