This parameter is under the sort namespace. It applies to the TopN sort operator. While computing the TopN, we hold onto incoming batches and maintain a list of references to the records which make up the current TopN. Periodically, we will copy the records that we want to keep, and release the batches to free up memory. We call this purging.. The purge threshold is how many batches we will hold onto before purging.
On Tue, Mar 8, 2016 at 11:34 AM, Vamsi Pendyala <[email protected]> wrote: > Hi > > Can someone help me to understand what does below parameter > purge.threshold :100 signifies , does it mean it will purge 100 days older > logs or 100hrs?Any insight is appreciated. Thanks. > > sys.store.provider.zk.blobroot: "maprfs:///drill<maprfs://drill>" > sort: { > purge.threshold : 100, > external: { > batch.size : 4000, > spill: { > batch.size : 4000, > group.size : 100, > threshold : 20000, > directories : [ "/drill/spill" ], > fs : "maprfs:///<maprfs://>" > } > } > }, > sys.store.provider: { > local: { > path: "maprfs:///drill<maprfs://drill>", > write: true > } > } > } > > -Vamsi >
