Jan Grashoefer created BIT-1441:
-----------------------------------
Summary: Logrotation cannot be set when using path_func
Key: BIT-1441
URL: https://bro-tracker.atlassian.net/browse/BIT-1441
Project: Bro Issue Tracker
Issue Type: Problem
Components: Bro
Affects Versions: 2.4
Environment: SLC6, PF_RING, broctl
Reporter: Jan Grashoefer
I had a problem using Bro's filtering on my Bro cluster (using broctl). I
wanted to create separate logfiles in JSON format for some streams. As the file
name should include the current date, I specified a path_func. So far
everything worked as expected. Then I tried to disable the logrotation for
these files by setting interv = 0. Unfortunately this did not work. Setting a
fixed path, disabling logrotation worked as intended (see
[http://try.bro.org/#/trybro/saved/14143] an example of the code I used).
I investigated this issue and think, I have discovered a problem. The rotation
interval for a writer is determined in CreateWriter in manager.cc (see
[https://github.com/bro/bro/blob/2b1cd66f17194a30b90490965cbdffdd71c18c09/src/logging/Manager.cc#L1064])
based on the filter. The filter again is determined by writer and path (I
don't understand why the name of the filter is not used but there may be
reasons). To see whether the interval is set correctly I added some debug
output here. Then I did a test specifying a filter for HTTP using path_func and
a filter for CONN using a fixed path.
On my worker I get the expected output (except the interval seems wrong):
{quote} 0.000000/1437813255.656896 [logging] Set interval for
'packet_filter' (filter 'default') to '86400.000000'
0.000000/1437813255.658523 [logging] Set interval for 'loaded_scripts'
(filter 'default') to '86400.000000'
0.000000/1437813255.685123 [logging] Set interval for 'communication'
(filter 'default') to '86400.000000'
1437813255.644956/1437813255.709181 [logging] Set interval for 'stats' (filter
'default') to '86400.000000'
1437813255.644965/1437813255.710468 [logging] Set interval for 'weird' (filter
'default') to '86400.000000'
1437813255.822196/1437813255.834760 [logging] Set interval for 'reporter'
(filter 'default') to '86400.000000'
1437813256.015793/1437813256.027556 [logging] Set interval for 'software'
(filter 'default') to '86400.000000'
1437813256.015793/1437813256.039455 [logging] Set interval for 'files' (filter
'default') to '86400.000000'
1437813256.015793/1437813256.040269 [logging] Set interval for 'http' (filter
'default') to '86400.000000'
1437813256.015793/1437813256.040504 [logging] Set interval for
'/var/opt/bro/logs-json/http-2015-07-25' (filter 'http_json') to '0.000000'
1437813257.512453/1437813257.523782 [logging] Set interval for 'x509' (filter
'default') to '86400.000000'
1437813260.645607/1437813260.656385 [logging] Set interval for 'conn' (filter
'default') to '86400.000000'
1437813260.645607/1437813260.656526 [logging] Set interval for
'/var/opt/bro/logs-json/conn' (filter 'conn_json') to '0.000000'
1437813262.827012/1437813262.839179 [logging] Set interval for 'dns' (filter
'default') to '86400.000000'
1437813263.401981/1437813263.411552 [logging] Set interval for 'ssl' (filter
'default') to '86400.000000'
1437813293.565530/1437813293.575182 [logging] Set interval for 'kerberos'
(filter 'default') to '86400.000000'{quote}
But on the manager I get the following:
{quote}1437813085.377826/1437813085.387819 [logging] Set interval for
'loaded_scripts' (filter 'default') to '3600.000000'
1437813085.377826/1437813085.400927 [logging] Set interval for 'communication'
(filter 'default') to '3600.000000'
1437813089.408731/1437813089.409921 [logging] Set interval for 'reporter'
(filter '') to '3600.000000'
1437813089.410046/1437813089.411141 [logging] Set interval for 'weird' (filter
'') to '3600.000000'
1437813089.410046/1437813089.411314 [logging] Set interval for 'packet_filter'
(filter '') to '3600.000000'
1437813089.411802/1437813089.412948 [logging] Set interval for 'stats' (filter
'') to '3600.000000'
1437813089.444066/1437813089.445155 [logging] Set interval for 'files' (filter
'') to '3600.000000'
1437813089.453163/1437813089.454249 [logging] Set interval for 'software'
(filter '') to '3600.000000'
1437813089.472973/1437813089.474123 [logging] Set interval for 'dns' (filter
'') to '3600.000000'
1437813089.507522/1437813089.508617 [logging] Set default interval for
'/var/opt/bro/logs-json/http-2015-07-25' (filter '')
1437813089.508759/1437813089.509852 [logging] Set interval for 'http' (filter
'') to '3600.000000'
1437813089.523751/1437813089.524868 [logging] Set interval for 'x509' (filter
'') to '3600.000000',
1437813089.983185/1437813089.984342 [logging] Set interval for 'ssl' (filter
'') to '3600.000000'
1437813093.316215/1437813093.317350 [logging] Set interval for 'ftp' (filter
'') to '3600.000000'
1437813094.076354/1437813094.077442 [logging] Set interval for 'conn' (filter
'') to '3600.000000'
1437813094.077580/1437813094.078657 [logging] Set interval for
'/var/opt/bro/logs-json/conn' (filter '') to '0.000000'
1437813100.949465/1437813100.950567 [logging] Set interval for 'syslog' (filter
'') to '3600.000000'{quote}
On the manager you can see, that for all worker-generated logs the filter is
not known and that the interval for my HTTP-JSON log is set to the default
value (Note: The instantiating filter is not known because it is not set in the
call in SendAllWritersTo - see
[https://github.com/bro/bro/blob/2b1cd66f17194a30b90490965cbdffdd71c18c09/src/logging/Manager.cc#L1174]).
So why does it work on the worker? Its because the path of the filter is
determined and set during the write: The first write triggers determining the
path by the filter. Then the writer is created and path of writer and filter
match. The writers on the manager seem to be created without a write and
therefore the filter cannot be determined.
At first I tried to fix the issue by using the name of the filter but as seen
in the debug output, the name is not set. I also thought about setting the
interval using the WriterBackend::WriterInfo, which is passed to CreateWriter
and has a field for the interval, but there is also the postprocessor set in
the CreateWriter method. Unfortunately I don't understand how logging is
distributed between manager and worker in detail, so I do not know how I can
fix this issue.
--
This message was sent by Atlassian JIRA
(v6.5-OD-08-001#65007)
_______________________________________________
bro-dev mailing list
[email protected]
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
