I don't think we have the functionality that you are looking for yet. Feel free to open a JIRA to help define exactly what that might look like.
I think providing some means of automatically managing data as it ages would be interesting. One approach I've been thinking about is trading some fidelity for space as it ages. For example, once I hit a ceiling on the amount of raw pcap that I can store, the system could potentially reprocess that raw pcap data to convert it to header-only pcap, flow records or some other form that consumes less storage, but loses some fidelity. On Thu, Jun 16, 2016 at 2:50 PM, Kuba Sienkiewicz <[email protected]> wrote: > What Merton thinks is old? (I mean when Merton purges data) About backup > data I mean having some space for hdfs and having separate partition (or > machine) for historical data. > On 14 Jun 2016 22:10, "Nick Allen" <[email protected]> wrote: > > > The standard deployment does setup some cleanup tasks that purges 'old' > > data. This seems to be different from what you're asking though. What > do > > you imagine using as your "backup space" if not /dev/null? > > > > > > > > > > On Tue, Jun 14, 2016 at 7:45 AM, Kuba Sienkiewicz <[email protected]> > > wrote: > > > > > PS. I'm saying about automatically moving data away from hdfs to some > > sort > > > of backup space. Sorry for inaccuracy in previous email (removing). > > > > > > wt., 14.06.2016 o 13:32 użytkownik Kuba Sienkiewicz < > > [email protected]> > > > napisał: > > > > > > > Hi all, > > > > I've tested metron for 2 days and I already have over 20GB of data > (w/o > > > > any special network traffic, server just stood untouched). > > > > What are good practices with such big amounts of data that metron > > stores? > > > > Also do metron support removing historical data automatically? > > > > > > > > Best Regards, > > > > Jakub Sienkiewicz > > > > > > > > > > > > > > > -- > > Nick Allen <[email protected]> > > > -- Nick Allen <[email protected]>
