Hmm. I think you're talking about the SegmentMetadata queries that DruidSchema runs. The intent is that they include an empty analysisTypes list, so they only use cached metadata and don't actually read segments, and are pretty resource-light on historicals. But if you implemented some sort of lazy loading for that metadata, those wouldn't play well together. I'm not sure what the best approach is here. What's the purpose of the lazy loading? If we need to make them play better together, one way to do that could be to add the information that the broker needs to the segment-level "Metadata" object, which I think is probably going to be faster to load, and then keep loading that eagerly.
On Tue, Feb 26, 2019 at 11:11 AM liliule...@gmail.com <liliule...@gmail.com> wrote: > Hi, I noticed that the query node will send a metadata query to the > historical node once new segments are published, and this will trigger an > analysis method to read all segments in memory and do the analysis. I tried > to do lazy cache for columns, which will read segments from disk once they > are called, however, the analysis method will read all data in memory to > process the metadata query. Is it a good idea to migrate this analysis > process to the middlemanager and persist the result in deep storage? So the > historical node can just read that file to answer metadata query. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org > For additional commands, e-mail: dev-h...@druid.apache.org > >
