FrankChen021 commented on PR #18687: URL: https://github.com/apache/druid/pull/18687#issuecomment-3450301084
> I guess it is natural to allow more start up strategies since we already support - loadAllLazily and loadAllEagerly. But the approach here feels a little complicated. We are also trying to re-invent the things which load rules already do. It would perhaps make more sense to use "Virtual Storage Fabric" + load rule combo since they are much more powerful and will only evolve over time. > > IIUC, the use case here is that we frequently query the data of the last `x` days and also query the data of the last `x + y` days but less frequently so. Rather than doing the eager vs lazy load of the segments in the `y` days only on startup, can we _always_ load these segments lazily? > > So, maybe the solution could be a mix of the recently added "Virtual Storage Fabric" + some load rules. > > Load rules would perhaps look something like: > > Rule 1: `loadByPeriod` last 7 days (i.e. load eagaerly on startup and otherwise) Rule 2: `loadLazilyByPeriod` last 30 days (i.e. these segments are present on virtual storage fabric and will be loaded when the first query is received for them. Coordinator remains agnostic of this fact. The virtual storage can also be reclaimed to load other virtual segments.). Rule 3: `dropForever` any data beyond the last 30 days. > > @FrankChen021 , @GWphua , would this satisfy your use case? @clintropolis , @abhishekrb19 , what are your thoughts? I think virtual storage + loading rule is different from what this PR is doing. under the virtual storage mode, the segment is not loaded from deep storage until first query comes. Here, the segment is not loaded into page `CACHE` during start up of historical, after a segment has been loaded by a historical node. the implementation here is not re-invent, and not complicated. If the goal of virtual storage will be the dominant mode in future, introducing `loadLazilyByPeriod` to loading rules makes sense. Virtual storage is just merged and is still under experiemental, I don't know when it will be production ready, and what's the roadmap for it. Making a small change to existing segment cache loading is still worthy. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
