Hi xuchianyin, Thanks for the question
1. Currently implementation is no need to load all the segments, only required will be loaded during filter and all segments will be loaded during query like count *. 2. Cache loading is fired during pruning phase in query, it will go to index server prune and load to cache , if index server is disabled and if distributed pruning is enabled then distributed pruning happens else driver side pruning, please check the index server Design doc for more info on this. For auto compaction, no need to load to index server, because internally one more level of compaction can happen and old loaded segments can become invalid, I will handle this is Design document. 3. Index server is a separate spark application meant for caching , so for SDK , spark session doesn't come into picture, so SDK not applicable, for file format case we will handle. Please get back for any clarifications or inputs. Thanks and Regards Akash R Nilugal From: xuchuanyin <xuchuan...@apache.org> > Date: Sat, 17 Aug, 2019, 11:55 AM > Subject: Re: [DISCUSSION] Cache Pre Priming > To: <dev@carbondata.apache.org> > > > Hi, I've two questions about the current index server implementation: > > 1. Currently do we need to load all the index data of all segments to cache > server while doing filter query OR only load the segments required by this > query? > > 2. When do we trigger the cache loading action during the query? > > As for the proposal in this mail, what will happen if auto-compaction occur > for this loading? > > 3. Since we want to preload the index to cache, maybe we need to handle all > the scenarios that causing data ingestion, so it seems you forget the SDK > scenario. > > > > -- > Sent from: > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/ >
