This is what we used to have in a very old version of Metron, and is what I
was referring to above.  Not sure how applicable it still is.

https://github.com/apache/metron/blob/Metron_0.3.1/metron-deployment/roles/metron_streaming/tasks/hdfs_purge.yml

Jon

On Mon, Dec 31, 2018 at 12:33 AM Farrukh Naveed Anjum <
[email protected]> wrote:

> Any suggestions ? to get ride of classic data and save up some space.
>
> On Wed, Dec 26, 2018 at 8:09 PM Farrukh Naveed Anjum <
> [email protected]> wrote:
>
>> Any exact command / example ? I am using BRO Tool for Log Collection
>>
>> On Wed, Dec 26, 2018 at 6:03 PM [email protected] <[email protected]>
>> wrote:
>>
>>> Aren't the logs stored in date-based directories?  I thought we used to
>>> deploy a daily cron with a hadoop fs -rmr based on today's date minus 30
>>> days or so.
>>>
>>> Jon
>>>
>>> On Wed, Dec 26, 2018 at 6:20 AM Farrukh Naveed Anjum <
>>> [email protected]> wrote:
>>>
>>>> what about data in HDFS and Other Logs ? Any idea to purne them ?
>>>>
>>>> On Wed, Dec 26, 2018 at 3:51 PM [email protected] <[email protected]>
>>>> wrote:
>>>>
>>>>> There used to be something that was automatically deployed as a cron
>>>>> that did that.  For ES I know it moved to curator
>>>>> https://github.com/apache/metron/blob/master/metron-platform/metron-data-management/README.md#pruning-data-from-elasticsearch
>>>>>
>>>>> I don't know if the HDFS job still exists or not, but it used to.
>>>>>
>>>>> Jon
>>>>>
>>>>> On Wed, Dec 26, 2018, 2:06 AM Farrukh Naveed Anjum <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> How can write a data retention script for metron. So that safe hard
>>>>>> disk space. Like I want to Store only last N Days (Say 30 days) data in
>>>>>> HDFS, Elasticssearch and Logs in  Metron.
>>>>>>
>>>>>> Right now, Metron  eats up my all hard drive space on server.
>>>>>>
>>>>>> Any Idea / Help will be appreciated.
>>>>>>
>>>>>>
>>>>>> --
>>>>>> With Regards
>>>>>> Farrukh Naveed Anjum
>>>>>>
>>>>> --
>>>>>
>>>>> Jon Zeolla
>>>>>
>>>>
>>>>
>>>> --
>>>> With Regards
>>>> Farrukh Naveed Anjum
>>>>
>>> --
>>>
>>> Jon Zeolla
>>>
>>
>>
>> --
>> With Regards
>> Farrukh Naveed Anjum
>>
>
>
> --
> With Regards
> Farrukh Naveed Anjum
>
-- 

Jon Zeolla

Reply via email to