Hi Joel,

This is my understanding:
We have list of all directories (i.e. all subdirectories and their
subdirectories etc.) in the metadata
cache file of each directory. We go through that list of directories and
check
directory modification time against modification time of metadata cache
file in that directory.
If this does not match for any of the directories, we build the metadata
cache for the whole hierarchy.
The reason we have to do this adding new files will only update
modification time of immediate
parent directory and not the whole hierarchy.

Regarding your proposal, some random thoughts:
How will you get current time that can be compared against last
modification time set by the file system ?
I think you meant compare current system time of the running java process
i.e. drillbit
against last time we checked if metadata cache needs to be updated for that
directory.
What issues we need to worry about if different directories in the
hierarchy are checked last at different times ?
Do we have to worry about coordinating against multiple drillbits ?

Another option is if the difference between modification time of directory
and metadata cache file is within
TTL limit, do not do anything. If we do that, do we get errors during
execution (I think so) ?
Also, how to reset that state and do metadata cache refresh eventually ?
We are not saving time for modification time checks here.

Instead of TTL, I think having a system/session option that will let us
skip this check altogether would be a
good thing to have. So, if we know we are not adding new data, we can set
that option.

Instead of saving this TTL in metadata cache file for each table(directory),
is it better to have this TTL as global system or session option ?
In that case, we cannot have a different TTL for each table, but it makes
it much simpler.
Otherwise, there are some complications to think about.
We have a root metadata file per directory with each of the subdirectories
underneath having their own metadata file.
So, if we update the TTL of the root directory, do we update for all the
subdirectories or just the top level directory ?
What issues we need to think about if TTL of the root directory and
subdirectories are different ?


Thanks
Padma




On Thu, Jul 12, 2018 at 8:07 AM, Joel Pfaff <joel.pf...@gmail.com> wrote:

> Hello,
>
> Thanks for the feedback.
>
> The logic I had in mind was to add the TTL, as a refresh_interval field in
> the root metadata file.
>
> At each query, the current time would be compared to the addition of the
> modification time of the root metadata file and the refresh_interval.
> If the current time is greater, it would mean the metadata may be invalid,
> so the regular process would apply: recursively going through the file to
> check for updates, and trig a full metadata cache refresh any change is
> detected, or just touch the metadata file to align its modification time
> with the current time if no change is detected.
> If the current time is smaller, the root metadata would be trusted (without
> additional checks) and the planning would continue.
>
> So in most of the cases, only the timestamp of the root metadata file would
> be checked. In the worst case (at most once per TTL), all the timestamps
> would be checked.
>
> Regards, Joel
>
> On Thu, Jul 12, 2018 at 4:47 PM, Vitalii Diravka <
> vitalii.dira...@gmail.com>
> wrote:
>
> > Hi Joel,
> >
> > Sounds reasonable.
> > But if Drill checks this TTL property from metadata cache file for every
> > query and for every file instead of file timestamp, it will not give the
> > benefit.
> > I suppose we can add this TTL property to only root metadata cache file
> and
> > check it only once per query.
> >
> > Could you clarify the details, what is the TTL time?
> > How TTL info could be used to determine whether refresh is needed for the
> > query?
> >
> > Kind regards
> > Vitalii
> >
> >
> > On Thu, Jul 12, 2018 at 4:40 PM Joel Pfaff <joel.pf...@gmail.com> wrote:
> >
> > > Hello,
> > >
> > > Today, on a table for which we have created statistics (through the
> > REFRESH
> > > TABLE METADATA <path to table> command), Drill validates the timestamp
> of
> > > every files or directory involved in the scan.
> > >
> > > If the timestamps of the files are greater than the one of the metadata
> > > file, then a re-regeneration of the meta-data file is triggered.
> > > In the case the timestamp of the metadata file is the greatest, then
> the
> > > planning continues without regenerating the metadata.
> > >
> > > When the number of files to be queried increases, this operation can
> > take a
> > > significant amount of time.
> > > We have seen cases where this validation step alone is taking 3 to 5
> > > seconds (just checking the timestamps), meaning the planning time was
> > > taking way more time than the querying time.
> > > And this can be problematic in some usecases where the response time is
> > > favored compared to the `accuracy` of the data.
> > >
> > > What would you think about adding an option to the metadata generation,
> > so
> > > that the metadata is trusted for a configurable time period
> > > Example : REFRESH TABLE METADATA <path to table> WITH TTL='15m'
> > > The exact syntax, of course, needs to be thought through.
> > >
> > > This TTL would be stored in the metadata file, and used to determine
> if a
> > > refresh is needed at each query. And this would significantly decrease
> > the
> > > planning time when the number of files represented in the metadata file
> > is
> > > important.
> > >
> > > Of course, this means that there could be cases where the metadata
> would
> > be
> > > wrong, so cases like the one below would need to be solved (since they
> > may
> > > happen much more frequently):
> > > https://issues.apache.org/jira/browse/DRILL-6194
> > > But my feeling is that since we already do have a kind of race
> condition
> > > between the view of the file system at the planning time, and the state
> > > that will be found during the execution, we could gracefully accept
> that
> > > some files may have disappeared between the planning and the execution.
> > >
> > > In the case the TTL would need to be changed, or be removed completely,
> > > this could be done by re-issuing a REFRESH TABLE METADATA, either with
> a
> > > new TTL, or without TTL at all.
> > >
> > > What do you think?
> > >
> > > Regards, Joel
> > >
> >
>

Reply via email to