The explanation changes for the two parameters have been merged into the master.
https://asterix-gerrit.ics.uci.edu/#/c/1281/3/asterixdb/asterix-installer/src/main/resources/conf/asterix-configuration.xml Best, Taewoo On Mon, Sep 12, 2016 at 5:02 PM, Taewoo Kim <[email protected]> wrote: > Thanks to Sattam, here is the revised version. Feel free to revise this. I > will upload a patch set after some revision is done. > > *storage.memorycomponent.numpages* > > The number of pages to allocate for a memory component. (Default = 256) > This budget is shared by all the memory components of the primary index > and all its secondary indexes across all I/O devices on a node. > Note: in-memory components usually has fill factor of 75% since the pages > are 75% full and the remaining 25% is un-utilized. > > > *storage.memorycomponent.globalbudget* > > [4GB + 100MB] The total size of memory in bytes that the sum of all open > memory components cannot exceed. (Default = 512MB) > Consider this as the buffer cache for all memory components of all indexes > in a node. > When this budget is fully used, a victim dataset will be chosen. It must > be evicted and closed to make a space for another dataset. > > > Best, > Taewoo > > On Mon, Sep 12, 2016 at 4:10 PM, Mike Carey <[email protected]> wrote: > >> +1 >> >> >> >> On 9/12/16 3:42 PM, Taewoo Kim wrote: >> >>> It would be really helpful this conversation can be applied in the >>> description of each parameter. Currently, I think that is too short. >>> >>> Best, >>> Taewoo >>> >>> On Mon, Sep 12, 2016 at 2:19 PM, Jianfeng Jia <[email protected]> >>> wrote: >>> >>> Clear. Thanks. >>>> >>>> And Ian’s parameters works. I can have a on-disk components around 128M. >>>> Thanks! >>>> >>>> On Sep 12, 2016, at 12:50 PM, Sattam Alsubaiee <[email protected]> >>>>> >>>> wrote: >>>> >>>>> This is the total memory size given for all datasets. Think of it as >>>>> the >>>>> buffer cache for all memory components of all indexes in that machine. >>>>> >>>> When >>>> >>>>> it is exhausted, a victim dataset must be evicted and closed to have a >>>>> space for another dataset. >>>>> >>>>> On Mon, Sep 12, 2016 at 12:29 PM, Jianfeng Jia <[email protected] >>>>> > >>>>> wrote: >>>>> >>>>> I was a little confused, there is another configuration: >>>>>> >>>>>> storage.memorycomponent.globalbudget ( which I set to 4G) >>>>>> >>>>>> I was thinking this is the budget that every component on one >>>>>> partition >>>>>> >>>>> is >>>> >>>>> shared. Is that the case? >>>>>> >>>>>> On Sep 12, 2016, at 12:16 PM, Sattam Alsubaiee <[email protected]> >>>>>>> >>>>>> wrote: >>>>>> >>>>>>> The 128M is shared by all the memory components of the primary index >>>>>>> >>>>>> and >>>> >>>>> all its secondary indexes across all io devices on that node. >>>>>>> Also the in-memory components usually usually has fill factor of 75% >>>>>>> >>>>>> since >>>>>> >>>>>>> the pages are 75% full and the remaining 25% is un-utilized. >>>>>>> >>>>>>> The page size that you have set 128KB looks reasonable for most >>>>>>> cases. >>>>>>> >>>>>> Your >>>>>> >>>>>>> best bet is to increase the value of storage.memorycomponent.numpage >>>>>>> >>>>>> to >>>> >>>>> a >>>>>> >>>>>>> higher number. >>>>>>> >>>>>>> Sattam >>>>>>> >>>>>>> >>>>>>> On Mon, Sep 12, 2016 at 11:33 AM, Jianfeng Jia < >>>>>>> [email protected] >>>>>>> wrote: >>>>>>> >>>>>>> Dear devs, >>>>>>>> >>>>>>>> I’m using the `no-merge` compaction policy and find that the >>>>>>>> physical >>>>>>>> flushed on-disk component is smaller than I was expected. >>>>>>>> >>>>>>>> Here are my related configurations >>>>>>>> >>>>>>>> <property> >>>>>>>> <name>storage.memorycomponent.pagesize</name> >>>>>>>> <value>128KB</value> >>>>>>>> <description>The page size in bytes for pages allocated to memory >>>>>>>> components. (Default = "131072" // 128KB) >>>>>>>> </description> >>>>>>>> </property> >>>>>>>> >>>>>>>> <property> >>>>>>>> <name>storage.memorycomponent.numpages</name> >>>>>>>> <value>1024</value> >>>>>>>> <description>The number of pages to allocate for a memory >>>>>>>> component. >>>>>>>> (Default = 256) >>>>>>>> </description> >>>>>>>> </property> >>>>>>>> >>>>>>>> With these two settings, I’m expecting the lsm component should be >>>>>>>> >>>>>>> 128M. >>>> >>>>> However, the flushed one is about 16M~ 20M. Do we have some >>>>>>>> >>>>>>> compression >>>> >>>>> for >>>>>> >>>>>>> the on-disk components? If so, it will be good. Otherwise, could >>>>>>>> >>>>>>> someone >>>> >>>>> help me to increase the component size? Thanks! >>>>>>>> >>>>>>>> Best, >>>>>>>> >>>>>>>> Jianfeng Jia >>>>>>>> PhD Candidate of Computer Science >>>>>>>> University of California, Irvine >>>>>>>> >>>>>>>> >>>>>>>> >>>>>> >>>>>> Best, >>>>>> >>>>>> Jianfeng Jia >>>>>> PhD Candidate of Computer Science >>>>>> University of California, Irvine >>>>>> >>>>>> >>>>>> >>>> >>>> Best, >>>> >>>> Jianfeng Jia >>>> PhD Candidate of Computer Science >>>> University of California, Irvine >>>> >>>> >>>> >> >
