[
https://issues.apache.org/jira/browse/METRON-815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15952965#comment-15952965
]
ASF GitHub Bot commented on METRON-815:
---------------------------------------
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/503#discussion_r109336484
--- Diff: metron-deployment/roles/sensor-stubs/templates/start-bro-stub ---
@@ -47,8 +47,8 @@ TOPIC="bro"
while true; do
# transform the bro timestamp and push to kafka
- SEARCH="\"ts\"\:[0-9]\+.[0-9]\{6\}"
- REPLACE="\"ts\"\:`date +%s`.000000"
+ SEARCH="\"ts\"\:[0-9]\+\."
+ REPLACE="\"ts\"\:`date +%s`\."
--- End diff --
@JonZeolla , good catch. Leaving the fractional portion of the timestamp
the same as it is, is appealing. However, since the granularity of `date +%s`
is only seconds, and we might transform a bunch of timestamps in one second of
wallclock realtime, this may result in apparently out-of-order timestamps, no?
Eg, if we start with data whose first three records have timestamps:
1491190032.222222 1491190032.777777 1491190033.111111
The transformed data will have timestamps
1491190442.222222 1491190442.777777 1491190442.111111
with later ones being (at least potentially) out of order. The original
code would have generated
1491190442.000000 1491190442.000000 1491190442.000000
which is rather monotone, but at least not out of order.
Is this okay, or potentially bad?
Perhaps it would be better to just change the `.[0-9]\{6\}` to `\.[0-9]\+`
in line 50, and leaving line 51 unchanged?
(I'm asking, I don't know. Maybe bro data can naturally be out of order?)
> sensor-stubs sometimes send malformed bro timestamps
> ----------------------------------------------------
>
> Key: METRON-815
> URL: https://issues.apache.org/jira/browse/METRON-815
> Project: Metron
> Issue Type: Bug
> Reporter: Jon Zeolla
> Assignee: Jon Zeolla
>
> The bro sensor-stub sends malformed timestamps when transforming an input
> timestamp that has less than 6 digits. For instance:
> [vagrant@node1 bin]$ SEARCH="\"ts\"\:[0-9]\+.[0-9]\{6\}"
> [vagrant@node1 bin]$ REPLACE="\"ts\"\:`date +%s`.000000"
> [vagrant@node1 bin]$ cat /opt/sensor-stubs/data/bro.out | sed -e
> "s/$SEARCH/$REPLACE/g"
> ...
> {"dns":
> {"ts":1491064638.000000.38621,"uid":"CQ5vBa2GcEToa4NKt5","id.orig_h":"192.168.66.1","id.orig_p":5353,"id.resp_h":"224.0.0.251","id.resp_p":5353,"proto":"udp","trans_id":0,"query":"_googlecast._tcp.local","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
