Please start a new thread
On December 5, 2019 at 02:07:53, Farrukh Naveed Anjum (
anjum.farr...@gmail.com) wrote:
I am not receiving data from Bro to Kafka
# @load packages/metron-bro-plugin-kafka/Apache/Kafka
redef Kafka::logs_to_send = set(SSH::LOG, RDP::LOG, KRB::LOG, SSL::LOG,
DHCP::LOG, Cluster::LOG, Syslog::LOG, SNMP::LOG, Reporter::LOG, DNP3::LOG,
RADIUS::LOG, Tunnel::LOG, Conn::LOG, HTTP::LOG, DNS::LOG, Software::LOG,
Intel::LOG, Notice::LOG, Signatures::LOG);
redef Kafka::send_all_active_logs = T;
redef Kafka::topic_name = "bro";
redef Kafka::tag_json = T;
redef Kafka::kafka_conf = table(
["metadata.broker.list"] = "localhost:6667",
["client.id"] = "bro"
);
Commented out line as per your recommendation. Still not getting any data
in Kafka Topic.... Any suggestions ?
On Thu, Jul 4, 2019 at 5:08 PM zeo...@gmail.com <zeo...@gmail.com> wrote:
> If you had the all active logs set to true it should send everything.
> What is the latest commit of the version of plugin are you running? I see
> it's 0.3 but since that hasn't been "released" (tagged) I'm assuming you
> are installing from master?
>
> Jon Zeolla
>
> On Wed, Jul 3, 2019, 5:57 PM Sanket Sharma <sanket.sha...@dukstra.com>
> wrote:
>
>> Seems like all I had to do was to specify the exact logs that I wanted to
>> export. All working now.
>>
>>
>>
>> Thanks for the help @Jon Zeolla
>>
>>
>>
>>
>>
>> Best regards,
>>
>> Sanket
>>
>>
>>
>>
>>
>> *From:* Sanket Sharma <sanket.sha...@dukstra.com>
>> *Reply-To:* "user@metron.apache.org" <user@metron.apache.org>
>> *Date:* Wednesday, 03 July 2019 at 19:47
>> *To:* "user@metron.apache.org" <user@metron.apache.org>
>> *Subject:* Re: metron-bro-plugin-kafka error
>>
>>
>>
>> Okay, I figured it out. There was a mismatch in my install bro (yum
>> installed), the source (git cloned) and the plugin version. I removed
>> everything and them compiled both zeek and the plugin from source and the
>> issue seems to have gone. I can run the test command I get the following
>> output.
>>
>>
>>
>> # zeek -N Apache::Kafka
>>
>> Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3.0)
>>
>>
>>
>> However, now I can't seem to get alerts/logs to Kafka. Here's the config
>> I'm using in /usr/local/zeek/share/zeek/site/local.zeek
>>
>>
>>
>> #This doesn't work in the new version anymore.
>>
>> #@load packages/metron-bro-plugin-kafka/Apache/Kafka
>>
>>
>>
>> #Tried added this line to ensure all packages are automatically loaded.
>>
>> #@load packages
>>
>>
>>
>> #Then tried loading the specific module
>>
>> #@load metron-bro-plugin-kafka
>>
>> #And then I eventually removed the three previous load lines
>>
>>
>>
>> redef Kafka::send_all_active_logs = T;
>>
>> redef Kafka::tag_json = T;
>>
>> redef Kafka::kafka_conf = table(
>>
>> ["metadata.broker.list"] = "mysecrethost:6667",
>>
>> ["client.id"] = "bro"
>>
>> );
>>
>>
>>
>> Even when I have the `@loads` disabled, I still see the script being
>> loaded (see logs below).
>>
>>
>>
>> To start, I did the following:
>>
>>
>>
>> zeekctl> deploy
>>
>> zeekctl> restart --clean
>>
>> zeekctl> start
>>
>>
>>
>> I can see the following in startup logs:
>>
>>
>>
>> starting ...
>>
>> starting zeek ...
>>
>> [ZeekControl] > diag
>>
>> [zeek]
>>
>>
>>
>> No core file found.
>>
>>
>>
>> Zeek 2.6-558
>>
>> Linux 3.10.0-957.21.3.el7.x86_64
>>
>>
>>
>> Zeek plugins:
>>
>> Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3.0)
>>
>>
>>
>> ==== No reporter.log
>>
>>
>>
>> ==== stderr.log
>>
>> listening on em1
>>
>>
>>
>>
>>
>> ==== stdout.log
>>
>> max memory size (kbytes, -m) unlimited
>>
>> data seg size (kbytes, -d) unlimited
>>
>> virtual memory (kbytes, -v) unlimited
>>
>> core file size (blocks, -c) unlimited
>>
>>
>>
>> ==== .cmdline
>>
>> -i em1 -U .status -p zeekctl -p zeekctl-live -p standalone -p local -p
>> zeek local.zeek zeekctl zeekctl/standalone zeekctl/auto
>>
>>
>>
>> ==== .env_vars
>>
>>
>> PATH=/usr/local/zeek/bin:/usr/local/zeek/share/zeekctl/scripts:/usr/local/openssl/bin:/opt/apache-maven-3.3.9/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/zeek/bin
>>
>>
>> ZEEKPATH=/usr/local/zeek/spool/installed-scripts-do-not-touch/site::/usr/local/zeek/spool/installed-scripts-do-not-touch/auto:/usr/local/zeek/share/zeek:/usr/local/zeek/share/zeek/policy:/usr/local/zeek/share/zeek/site
>>
>> CLUSTER_NODE=
>>
>>
>>
>> ==== .status
>>
>> RUNNING [net_run]
>>
>>
>>
>> ==== No prof.log
>>
>>
>>
>> ==== packet_filter.log
>>
>> #separator \x09
>>
>> #set_separator ,
>>
>> #empty_field (empty)
>>
>> #unset_field -
>>
>> #path packet_filter
>>
>> #open 2019-07-03-19-36-56
>>
>> #fields ts node filter init success
>>
>> #types time string string bool bool
>>
>> 1562175416.590048 zeek ip or not ip T T
>>
>>
>>
>> ==== loaded_scripts.log
>>
>> #separator \x09
>>
>> #set_separator ,
>>
>> #empty_field (empty)
>>
>> #unset_field -
>>
>> #path loaded_scripts
>>
>> #open 2019-07-03-19-36-56
>>
>> #fields name
>>
>> #types string
>>
>> /usr/local/zeek/lib/zeek/plugins/APACHE_KAFKA/lib/bif/__load__.zeek
>>
>> /usr/local/zeek/lib/zeek/plugins/APACHE_KAFKA/lib/bif/kafka.bif.zeek
>>
>> /usr/local/zeek/lib/zeek/plugins/APACHE_KAFKA/scripts/__load__.bro
>>
>> /usr/local/zeek/lib/zeek/plugins/APACHE_KAFKA/scripts/init.bro
>>
>>
>>
>>
>>
>> it starts up fine, no error messages. running "diag" in zeekctl just
>> gives a long list of plugins that were loaded.
>>
>>
>>
>> If I tail logs in I see new connection logs being added. However, I dont
>> see any messages in Kafka console consumer. What am I missing? How do I go
>> about debugging this?
>>
>>
>>
>> Thank you for your help and assistance.
>>
>>
>>
>> Best regards,
>>
>> Sanket
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ------------------------------
>>
>> *From:* zeo...@gmail.com <zeo...@gmail.com>
>> *Sent:* Tuesday, July 2, 2019 11:46 AM
>> *To:* user@metron.apache.org
>> *Subject:* Re: metron-bro-plugin-kafka error
>>
>>
>>
>> Did you install it manual or with bro-pkg/zkg? I believe bro-pkg was
>> renamed to zkg as of their 2.0 release but I haven't used it in a little
>> while. Any more details regarding the installation process, or versions of
>> software in use may be helpful
>>
>> Jon Zeolla
>>
>>
>>
>> On Tue, Jul 2, 2019, 12:26 AM Sanket Sharma <sanket.sha...@dukstra.com>
>> wrote:
>>
>> Hi,
>>
>>
>>
>> I’m trying to configure Metron bro plugin by following instructions here:
>> https://github.com/apache/metron-bro-plugin-kafka
>>
>>
>>
>>
>>
>> I’m able to build/install the plugin successfully but when I test it
>> using the command:
>>
>>
>>
>> $ bro -N Apache::Kafka
>>
>>
>>
>>
>>
>> I get the following error:
>>
>>
>>
>> fatal error in /opt/bro/share/bro/base/init-bare.bro, line 1: cannot load
>> plugin library /opt/bro/lib/bro/plugins/APACHE_KAFKA//lib/
>> APACHE-KAFKA.linux-x86_64.so: /opt/bro/lib/bro/plugins/APACHE_KAFKA//lib/
>> APACHE-KAFKA.linux-x86_64.so: undefined symbol:
>> bro_version_2_6_558_plugin_7
>>
>>
>>
>> Not sure what am I missing? Any help would be greatly appreciated.
>>
>>
>>
>>
>>
>> Best regards,
>>
>> Sanket
>>
>>
>>
>>
>>
>>
--
*Best Regards*
Farrukh Naveed Anjum
*M:* +92 321 5083954 (WhatsApp Enabled)
*W:* https://www.farrukh.cc/
