Yes, I agree with you Jon. I think it would be interesting to have a mechanism that gradually trades-off fidelity for disk space.
Of course, you would start with raw pcap. After a threshold is crossed, the oldest data would be transformed to header-only pcap. From there you could transform the header-only pcap to flow records. The specific ways in which the data is transformed/degraded would have to be configurable. Maybe someone doesn't find header-only pcap useful at all and simply goes from full pcap to flow records. On Wed, Jul 20, 2016 at 2:01 PM, [email protected] <[email protected]> wrote: > Hi All, > > Has there been any discussion on truncating or extracting some payload from > PCAP on data expiration instead of doing the standard YAF flow IPFIX? I > would like to have longer retention of PCAP, but don't have an empty > multi-PB storage array at our disposal to handle full captures. We have > used time-machine to do bpf-based connection cutoffs this in the past > (relevant > documentation > < > https://github.com/bro/time-machine/blob/master/doc/howto.rst#class-section > > > ), however that product does not support IPv6 (unless you use the LBNL > non-sanctified branch) and seems to be mostly dead. Time-machine has been > mostly eclipsed by Stenographer as much as I can tell, but they do not > support this feature either (relevant issue > <https://github.com/google/stenographer/issues/123>). > > I spoke with the YAF/SiLK developers earlier today about this and they > mentioned that it is possible to have YAF export the first X bytes of the > payload and store it in their IPFIX output. Perhaps this could become a > configurable change, where you specify the amount of payload to retain? > Interested in getting input on this - if feedback is positive we may be > able to dedicate some development cycles to it. > > Jon > -- > > Jon > -- Nick Allen <[email protected]>
