This is what we used to have in a very old version of Metron, and is what I was referring to above. Not sure how applicable it still is.
https://github.com/apache/metron/blob/Metron_0.3.1/metron-deployment/roles/metron_streaming/tasks/hdfs_purge.yml Jon On Mon, Dec 31, 2018 at 12:33 AM Farrukh Naveed Anjum < [email protected]> wrote: > Any suggestions ? to get ride of classic data and save up some space. > > On Wed, Dec 26, 2018 at 8:09 PM Farrukh Naveed Anjum < > [email protected]> wrote: > >> Any exact command / example ? I am using BRO Tool for Log Collection >> >> On Wed, Dec 26, 2018 at 6:03 PM [email protected] <[email protected]> >> wrote: >> >>> Aren't the logs stored in date-based directories? I thought we used to >>> deploy a daily cron with a hadoop fs -rmr based on today's date minus 30 >>> days or so. >>> >>> Jon >>> >>> On Wed, Dec 26, 2018 at 6:20 AM Farrukh Naveed Anjum < >>> [email protected]> wrote: >>> >>>> what about data in HDFS and Other Logs ? Any idea to purne them ? >>>> >>>> On Wed, Dec 26, 2018 at 3:51 PM [email protected] <[email protected]> >>>> wrote: >>>> >>>>> There used to be something that was automatically deployed as a cron >>>>> that did that. For ES I know it moved to curator >>>>> https://github.com/apache/metron/blob/master/metron-platform/metron-data-management/README.md#pruning-data-from-elasticsearch >>>>> >>>>> I don't know if the HDFS job still exists or not, but it used to. >>>>> >>>>> Jon >>>>> >>>>> On Wed, Dec 26, 2018, 2:06 AM Farrukh Naveed Anjum < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> How can write a data retention script for metron. So that safe hard >>>>>> disk space. Like I want to Store only last N Days (Say 30 days) data in >>>>>> HDFS, Elasticssearch and Logs in Metron. >>>>>> >>>>>> Right now, Metron eats up my all hard drive space on server. >>>>>> >>>>>> Any Idea / Help will be appreciated. >>>>>> >>>>>> >>>>>> -- >>>>>> With Regards >>>>>> Farrukh Naveed Anjum >>>>>> >>>>> -- >>>>> >>>>> Jon Zeolla >>>>> >>>> >>>> >>>> -- >>>> With Regards >>>> Farrukh Naveed Anjum >>>> >>> -- >>> >>> Jon Zeolla >>> >> >> >> -- >> With Regards >> Farrukh Naveed Anjum >> > > > -- > With Regards > Farrukh Naveed Anjum > -- Jon Zeolla
