clintropolis commented on issue #18446: URL: https://github.com/apache/druid/issues/18446#issuecomment-3444460732
>I also see that you have rendered isLazyLoadOnStart() introduced in https://github.com/apache/druid/pull/6988 at SegmentLocalCacheManager obsolete This was an accident, fixed in #18637. Also, your understanding of how the virtual storage feature works looks correct to me. >I'm not really that familiar with the load rules, will it work for Historicals during start-up, and not just for Historicals that have been running for a while? I think the solution I was thinking of should work for startup since it involves the coordinator wrapping the load spec with another load spec to indicate stuff (like load as a weak reference, or in this case, lazy deserialize the columns on loading) and sending that to the historicals instead of the direct load spec. >A picture of the problem we are trying to solve: Historical start-up takes a very long time (20min), and we also do not want queries for the past 7 days to be affected. So we want to implement a functionality where we can lazily load segments that are more than 7 days old, while eagerly loading the past 7 days of data... Will your proposed virtualStorage be able to help with this? Right now virtual storage could only help with this if you had separate tiers of historicals, a regular tier with the past 7 days, and another virtual storage mode tier that has the other data. Later, the load rules thing I'm thinking about would probably help more, since there will be a lot finer control over how stuff is loaded, but haven't got to this yet. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
