> > > > Thanks, I missed that point. However, there's still a point at which the > consumer fetches start getting served from remote storage (even if that > point isn't as soon as the local log retention time/size). This represents > a kind of performance cliff edge and what I'm really interested in is how > easy it is for a consumer which falls off that cliff to catch up and so its > fetches again come from local storage. Obviously this can depend on all > sorts of factors (like production rate, consumption rate), so it's not > guaranteed (just like it's not guaranteed for Kafka today), but this would > represent a new failure mode. >
As I have explained in the last mail, it's a very rare case that a consumer need to read remote data. With our experience at Uber, this only happens when the consumer service had an outage for several hours. There is not a "performance cliff" as you assume. The remote storage is even faster than local disks in terms of bandwidth. Reading from remote storage is going to have higher latency than local disk. But since the consumer is catching up several hours data, it's not sensitive to the sub-second level latency, and each remote read request will read a large amount of data to make the overall performance better than reading from local disks. > Another aspect I'd like to understand better is the effect of serving fetch > request from remote storage has on the broker's network utilization. If > we're just trimming the amount of data held locally (without increasing the > overall local+remote retention), then we're effectively trading disk > bandwidth for network bandwidth when serving fetch requests from remote > storage (which I understand to be a good thing, since brokers are > often/usually disk bound). But if we're increasing the overall local+remote > retention then it's more likely that network itself becomes the bottleneck. > I appreciate this is all rather hand wavy, I'm just trying to understand > how this would affect broker performance, so I'd be grateful for any > insights you can offer. > > Network bandwidth is a function of produce speed, it has nothing to do with remote retention. As long as the data is shipped to remote storage, you can keep the data there for 1 day or 1 year or 100 years, it doesn't consume any network resources.
