Very good points, Jinfeng!
However, short lived queries can also provide valuable insight, so persisting
them makes sense. For such queries, where the throughput of executed queries is
high and we don't want to incur the penalty of writing profiles to disk, there
is already an in-memory profil
two things that might be worth considering.
1. Add an option to persistent the profile to a file, only if query elapse
time exceeds a certain threshold (say 10 seconds)? Normally, people will
need look into profiles just to figure out performance bottleneck. It
probably makes sense to only store p
This would be a great improvement (and long overdue). Thanks for working
on it.
I would be inclined to option #2 and perhaps add an option to drillbit
startup that allows partitioning all existing profiles in a forced manner
(default can be the 1000 profiles that you proposed).
The option makes t
Hi guys
I'm working on a draft PR to improve the management of Drill's query profiles.
https://github.com/apache/drill/pull/1750
[https://github.com/apache/drill/pull/1750]
The design basically partitions existing profiles into sub-directories based on
the structure '/MM/dd' (can be custom
