Re: [DISCUSS] System time vs. Event Time

I am a huge fan of these ideas and your comments Matt. These use cases have been in the back of my head for a while, so I'm happy to see them getting discussed. It would be a huge step forward for Metron capabilities. I see connections between this discussion and both METRON-192

Re: [DISCUSS] System time vs. Event Time

Before the thought becomes obsolete, I’d like to say that I agree with Nick about the replay scenario and threat signature databases. I think a principal use case is replaying old data with new threat signatures, to detect problems that were undetectable at the time they happened. The use

Re: [DISCUSS] System time vs. Event Time

I'm just going to throw out a few of questions, that I don't have good answers to. Casey and Nick, given your familiarity with the systems involved, do you have any thoughts? - What's the smallest unit of work we can do to enable at least a useful subset of a fully featured term batch

Re: [DISCUSS] System time vs. Event Time

@Jon, it looks like it is based on system date. >From ElasticsearchWriter.write: String indexPostfix = dateFormat.format(new Date()); ... indexName = indexName + "_index_" + indexPostfix; ... IndexRequestBuilder indexRequestBuilder = client.prepareIndex(indexName, sensorType + "_doc"); Justin

Re: [DISCUSS] System time vs. Event Time

I'm actually a bit surprised to see METRON-691, because I know a while back I did some experiments to ensure that data was being written to the indexes that relate to the timestamp in the message, not the current time, and I thought that messages were getting written to the proper historical

Re: [DISCUSS] System time vs. Event Time

Let's make sure we have a common understanding of the use case (there are likely many). What you mentioned was replaying historical data, which is very cool, but can mean a lot of different things (being that we all have very active imaginations). Here are a few broad strokes of what I have been

Re: [DISCUSS] System time vs. Event Time

I think this is a really tricky topic, but necessary. I've given it a bit of thought over the last few months and I don't really see a great way to do it given the Profiler. Here's what I've come up with so far, though, in my thinking. - Replaying events will compress events in time (e.g. 2

[DISCUSS] System time vs. Event Time

There's a couple JIRAs related to the use of system time vs event time. METRON-590 Enable Use of Event Time in Profiler METRON-691 Elastic Writer index partitions on system time, not event time