Re: Rolling Window

2016-03-08 Thread François Méthot
ueries are in "running" state or schedule a delete outside business hours. On Tue, Mar 1, 2016 at 11:25 AM, François Méthot wrote: > Hi, > > We need to manage a rolling window of parquet data within drill. > > Our parquet files are partitioned by hour, > Once hd

Rolling Window

2016-03-01 Thread François Méthot
Hi, We need to manage a rolling window of parquet data within drill. Our parquet files are partitioned by hour, Once hdfs reach a certain usage threshold, we want to delete the oldest partition folder. A simple approach would be to run a cron job that check the hdfs usage and delete the