[
https://issues.apache.org/jira/browse/METRON-477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16282970#comment-16282970
]
Otto Fowler commented on METRON-477:
------------------------------------
Another idea could be to just store the data in all levels of fidelity all the
time, and have different ( tunable ) expiry policies.
Functionally, the system could offer different levels of functionality, perhaps
specific to each level of fidelity or by speed or purpose ( at the lowest level
of fidelity, the data is some statistical representation of 'counts' of data
that could back simple visualization and queries for example).
This approach may work better because:
* it is simple, there are less moving parts
* it allows for all the *newest* data to always be available to all the
features. The idea that older data expires is more readily understandable than
the idea that data does not show up until it is 'old' enough.
Data expiration could be 'plugin', such that you can choose to expire data to
multiple stores of different types.
> Support lower fidelity retention of network traffic over time
> -------------------------------------------------------------
>
> Key: METRON-477
> URL: https://issues.apache.org/jira/browse/METRON-477
> Project: Metron
> Issue Type: Improvement
> Reporter: Jon Zeolla
>
> Currently fastcapa supports full pcap capture. I would like to see the
> ability to retain network traffic for longer periods of time but at
> increasing less fidelity.
> For instance:
> - Full PCAP is ingested and stored in bucket 1
> - Transition "Full PCAP" to "Truncated PCAP" after bucket 1 hits X size,
> stored in bucket 2
> - Transform the truncated PCAP into flows or daily summaries after bucket 2
> hits X size, stored in bucket 3
> This system should be setup so that the transition jobs are highly
> configurable (as in sizes for each bucket, truncation cutoffs length,
> transition ordering, etc.). In addition, both the full pcap and truncated
> pcap should be able to be retrieved using the same method (CLI, UI, etc.).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
