Thanks to Sattam, here is the revised version. Feel free to revise this. I will upload a patch set after some revision is done.
*storage.memorycomponent.numpages* The number of pages to allocate for a memory component. (Default = 256) This budget is shared by all the memory components of the primary index and all its secondary indexes across all I/O devices on a node. Note: in-memory components usually has fill factor of 75% since the pages are 75% full and the remaining 25% is un-utilized. *storage.memorycomponent.globalbudget* [4GB + 100MB] The total size of memory in bytes that the sum of all open memory components cannot exceed. (Default = 512MB) Consider this as the buffer cache for all memory components of all indexes in a node. When this budget is fully used, a victim dataset will be chosen. It must be evicted and closed to make a space for another dataset. Best, Taewoo On Mon, Sep 12, 2016 at 4:10 PM, Mike Carey <dtab...@gmail.com> wrote: > +1 > > > > On 9/12/16 3:42 PM, Taewoo Kim wrote: > >> It would be really helpful this conversation can be applied in the >> description of each parameter. Currently, I think that is too short. >> >> Best, >> Taewoo >> >> On Mon, Sep 12, 2016 at 2:19 PM, Jianfeng Jia <jianfeng....@gmail.com> >> wrote: >> >> Clear. Thanks. >>> >>> And Ian’s parameters works. I can have a on-disk components around 128M. >>> Thanks! >>> >>> On Sep 12, 2016, at 12:50 PM, Sattam Alsubaiee <salsuba...@gmail.com> >>>> >>> wrote: >>> >>>> This is the total memory size given for all datasets. Think of it as the >>>> buffer cache for all memory components of all indexes in that machine. >>>> >>> When >>> >>>> it is exhausted, a victim dataset must be evicted and closed to have a >>>> space for another dataset. >>>> >>>> On Mon, Sep 12, 2016 at 12:29 PM, Jianfeng Jia <jianfeng....@gmail.com> >>>> wrote: >>>> >>>> I was a little confused, there is another configuration: >>>>> >>>>> storage.memorycomponent.globalbudget ( which I set to 4G) >>>>> >>>>> I was thinking this is the budget that every component on one partition >>>>> >>>> is >>> >>>> shared. Is that the case? >>>>> >>>>> On Sep 12, 2016, at 12:16 PM, Sattam Alsubaiee <salsuba...@gmail.com> >>>>>> >>>>> wrote: >>>>> >>>>>> The 128M is shared by all the memory components of the primary index >>>>>> >>>>> and >>> >>>> all its secondary indexes across all io devices on that node. >>>>>> Also the in-memory components usually usually has fill factor of 75% >>>>>> >>>>> since >>>>> >>>>>> the pages are 75% full and the remaining 25% is un-utilized. >>>>>> >>>>>> The page size that you have set 128KB looks reasonable for most cases. >>>>>> >>>>> Your >>>>> >>>>>> best bet is to increase the value of storage.memorycomponent.numpage >>>>>> >>>>> to >>> >>>> a >>>>> >>>>>> higher number. >>>>>> >>>>>> Sattam >>>>>> >>>>>> >>>>>> On Mon, Sep 12, 2016 at 11:33 AM, Jianfeng Jia < >>>>>> jianfeng....@gmail.com >>>>>> wrote: >>>>>> >>>>>> Dear devs, >>>>>>> >>>>>>> I’m using the `no-merge` compaction policy and find that the physical >>>>>>> flushed on-disk component is smaller than I was expected. >>>>>>> >>>>>>> Here are my related configurations >>>>>>> >>>>>>> <property> >>>>>>> <name>storage.memorycomponent.pagesize</name> >>>>>>> <value>128KB</value> >>>>>>> <description>The page size in bytes for pages allocated to memory >>>>>>> components. (Default = "131072" // 128KB) >>>>>>> </description> >>>>>>> </property> >>>>>>> >>>>>>> <property> >>>>>>> <name>storage.memorycomponent.numpages</name> >>>>>>> <value>1024</value> >>>>>>> <description>The number of pages to allocate for a memory >>>>>>> component. >>>>>>> (Default = 256) >>>>>>> </description> >>>>>>> </property> >>>>>>> >>>>>>> With these two settings, I’m expecting the lsm component should be >>>>>>> >>>>>> 128M. >>> >>>> However, the flushed one is about 16M~ 20M. Do we have some >>>>>>> >>>>>> compression >>> >>>> for >>>>> >>>>>> the on-disk components? If so, it will be good. Otherwise, could >>>>>>> >>>>>> someone >>> >>>> help me to increase the component size? Thanks! >>>>>>> >>>>>>> Best, >>>>>>> >>>>>>> Jianfeng Jia >>>>>>> PhD Candidate of Computer Science >>>>>>> University of California, Irvine >>>>>>> >>>>>>> >>>>>>> >>>>> >>>>> Best, >>>>> >>>>> Jianfeng Jia >>>>> PhD Candidate of Computer Science >>>>> University of California, Irvine >>>>> >>>>> >>>>> >>> >>> Best, >>> >>> Jianfeng Jia >>> PhD Candidate of Computer Science >>> University of California, Irvine >>> >>> >>> >
