Hi Tarik, You’re quite right, ES costs can get very high, which is why most Metron users store a short term amount of data in ES and use the HDFS store for longer term data access.
Most I know of keep 1 to 3 months in elastic, and use HDFS to store data for, in some cases, years. This is usually done by deleting older ES indexes via curator. Some people also limit the fields stored in ES through templates, which is something we’ve talked about making even more efficient with field level filtering in Metron. That helps keep the hot layer storage costs (ES) down. Simon > On 12 Sep 2018, at 20:08, Tarik Courdy <[email protected]> wrote: > > Hello - > > I was wondering if I could receive an invite to the apache metron slack > organization? > > I also wanted to ask a question about supported data stores. One of the > supported data stores is elasticsearch. Does this mean that all data is > stored directly in elasticsearch or is some other approach taken? If all of > the data is stored in ES, I imagine that scaling costs could get out of hand > as the size of the data continues to grow. Is this a valid concern with > metron or no? > > Thank you for your time. > > -Tarik
