pabrahamusa commented on issue #6317: URL: https://github.com/apache/incubator-pinot/issues/6317#issuecomment-743384566
@fx19880617 Tiered Storage plus Lazy loading will surely help. However if a raw index is considerably small we can preferably store it in SSDs and separate the data out to object storage. The only caveat here is how to map the index to data locations in object store. There should be some way to mount the object store locally to pull the corresponding data. Either by defining partitions based on the index which helps to locate the object directly. Also there is a possibility of introducing zstd to further reduce the size and increase the speed of transfer. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
