Hi, I have been using drill for more than year now, we are running 1.10.
My queries can spend from 5 to 10 minutes for planning because I am dealing with lots of file in HDFS. (then 5 min to 60 min for execution) I maintain a rolling window of data partitionned by the epoch seconds rounded to the hour. /mydata/3/ -> Next partition to be deleted (nightly check) /mydata/4/ /mydata/.../ /mydata/109/ /mydata/110/ -> current hour, this is where new parquet files are added I am considering using REFRESH TABLE METADATA. Is it beneficial at all in a situation where new files are added constantly, (but only to the latest partition, older partition are set in stone)? Will drill detect that new files are added to the latest partition (110) ? -Will it trigger a refresh metadata on all the directory, on just on /mydata/110? Thanks for your help François
