Re: How to set the lsm component size?
Nice! > On Oct 13, 2016, at 2:56 PM, Taewoo Kim wrote: > > The explanation changes for the two parameters have been merged into the > master. > > https://asterix-gerrit.ics.uci.edu/#/c/1281/3/asterixdb/asterix-installer/src/main/resources/conf/asterix-configuration.xml > > Best, > Taewoo > > On Mon, Sep 12, 2016 at 5:02 PM, Taewoo Kim wrote: > >> Thanks to Sattam, here is the revised version. Feel free to revise this. I >> will upload a patch set after some revision is done. >> >> *storage.memorycomponent.numpages* >> >> The number of pages to allocate for a memory component. (Default = 256) >> This budget is shared by all the memory components of the primary index >> and all its secondary indexes across all I/O devices on a node. >> Note: in-memory components usually has fill factor of 75% since the pages >> are 75% full and the remaining 25% is un-utilized. >> >> >> *storage.memorycomponent.globalbudget* >> >> [4GB + 100MB] The total size of memory in bytes that the sum of all open >> memory components cannot exceed. (Default = 512MB) >> Consider this as the buffer cache for all memory components of all indexes >> in a node. >> When this budget is fully used, a victim dataset will be chosen. It must >> be evicted and closed to make a space for another dataset. >> >> >> Best, >> Taewoo >> >> On Mon, Sep 12, 2016 at 4:10 PM, Mike Carey wrote: >> >>> +1 >>> >>> >>> >>> On 9/12/16 3:42 PM, Taewoo Kim wrote: >>> It would be really helpful this conversation can be applied in the description of each parameter. Currently, I think that is too short. Best, Taewoo On Mon, Sep 12, 2016 at 2:19 PM, Jianfeng Jia wrote: Clear. Thanks. > > And Ian’s parameters works. I can have a on-disk components around 128M. > Thanks! > > On Sep 12, 2016, at 12:50 PM, Sattam Alsubaiee >> > wrote: > >> This is the total memory size given for all datasets. Think of it as >> the >> buffer cache for all memory components of all indexes in that machine. >> > When > >> it is exhausted, a victim dataset must be evicted and closed to have a >> space for another dataset. >> >> On Mon, Sep 12, 2016 at 12:29 PM, Jianfeng Jia >> >> wrote: >> >> I was a little confused, there is another configuration: >>> >>> storage.memorycomponent.globalbudget ( which I set to 4G) >>> >>> I was thinking this is the budget that every component on one >>> partition >>> >> is > >> shared. Is that the case? >>> >>> On Sep 12, 2016, at 12:16 PM, Sattam Alsubaiee >>> wrote: >>> The 128M is shared by all the memory components of the primary index >>> and > >> all its secondary indexes across all io devices on that node. Also the in-memory components usually usually has fill factor of 75% >>> since >>> the pages are 75% full and the remaining 25% is un-utilized. The page size that you have set 128KB looks reasonable for most cases. >>> Your >>> best bet is to increase the value of storage.memorycomponent.numpage >>> to > >> a >>> higher number. Sattam On Mon, Sep 12, 2016 at 11:33 AM, Jianfeng Jia < jianfeng@gmail.com wrote: Dear devs, > > I’m using the `no-merge` compaction policy and find that the > physical > flushed on-disk component is smaller than I was expected. > > Here are my related configurations > > > storage.memorycomponent.pagesize > 128KB > The page size in bytes for pages allocated to memory > components. (Default = "131072" // 128KB) > > > > > storage.memorycomponent.numpages > 1024 > The number of pages to allocate for a memory > component. > (Default = 256) > > > > With these two settings, I’m expecting the lsm component should be > 128M. > >> However, the flushed one is about 16M~ 20M. Do we have some > compression > >> for >>> the on-disk components? If so, it will be good. Otherwise, could > someone > >> help me to increase the component size? Thanks! > > Best, > > Jianfeng Jia > PhD Candidate of Computer Science > University of California, Irvine > > > >>> >>> Best, >>> >>> Jianfeng Jia >>> PhD Candidate of Computer Science >>> University of California, Irvine >>> >>> >>> > > Best, > > Jianfeng Jia > P
Re: How to set the lsm component size?
The explanation changes for the two parameters have been merged into the master. https://asterix-gerrit.ics.uci.edu/#/c/1281/3/asterixdb/asterix-installer/src/main/resources/conf/asterix-configuration.xml Best, Taewoo On Mon, Sep 12, 2016 at 5:02 PM, Taewoo Kim wrote: > Thanks to Sattam, here is the revised version. Feel free to revise this. I > will upload a patch set after some revision is done. > > *storage.memorycomponent.numpages* > > The number of pages to allocate for a memory component. (Default = 256) > This budget is shared by all the memory components of the primary index > and all its secondary indexes across all I/O devices on a node. > Note: in-memory components usually has fill factor of 75% since the pages > are 75% full and the remaining 25% is un-utilized. > > > *storage.memorycomponent.globalbudget* > > [4GB + 100MB] The total size of memory in bytes that the sum of all open > memory components cannot exceed. (Default = 512MB) > Consider this as the buffer cache for all memory components of all indexes > in a node. > When this budget is fully used, a victim dataset will be chosen. It must > be evicted and closed to make a space for another dataset. > > > Best, > Taewoo > > On Mon, Sep 12, 2016 at 4:10 PM, Mike Carey wrote: > >> +1 >> >> >> >> On 9/12/16 3:42 PM, Taewoo Kim wrote: >> >>> It would be really helpful this conversation can be applied in the >>> description of each parameter. Currently, I think that is too short. >>> >>> Best, >>> Taewoo >>> >>> On Mon, Sep 12, 2016 at 2:19 PM, Jianfeng Jia >>> wrote: >>> >>> Clear. Thanks. And Ian’s parameters works. I can have a on-disk components around 128M. Thanks! On Sep 12, 2016, at 12:50 PM, Sattam Alsubaiee > wrote: > This is the total memory size given for all datasets. Think of it as > the > buffer cache for all memory components of all indexes in that machine. > When > it is exhausted, a victim dataset must be evicted and closed to have a > space for another dataset. > > On Mon, Sep 12, 2016 at 12:29 PM, Jianfeng Jia > > wrote: > > I was a little confused, there is another configuration: >> >> storage.memorycomponent.globalbudget ( which I set to 4G) >> >> I was thinking this is the budget that every component on one >> partition >> > is > shared. Is that the case? >> >> On Sep 12, 2016, at 12:16 PM, Sattam Alsubaiee >>> >> wrote: >> >>> The 128M is shared by all the memory components of the primary index >>> >> and > all its secondary indexes across all io devices on that node. >>> Also the in-memory components usually usually has fill factor of 75% >>> >> since >> >>> the pages are 75% full and the remaining 25% is un-utilized. >>> >>> The page size that you have set 128KB looks reasonable for most >>> cases. >>> >> Your >> >>> best bet is to increase the value of storage.memorycomponent.numpage >>> >> to > a >> >>> higher number. >>> >>> Sattam >>> >>> >>> On Mon, Sep 12, 2016 at 11:33 AM, Jianfeng Jia < >>> jianfeng@gmail.com >>> wrote: >>> >>> Dear devs, I’m using the `no-merge` compaction policy and find that the physical flushed on-disk component is smaller than I was expected. Here are my related configurations storage.memorycomponent.pagesize 128KB The page size in bytes for pages allocated to memory components. (Default = "131072" // 128KB) storage.memorycomponent.numpages 1024 The number of pages to allocate for a memory component. (Default = 256) With these two settings, I’m expecting the lsm component should be >>> 128M. > However, the flushed one is about 16M~ 20M. Do we have some >>> compression > for >> >>> the on-disk components? If so, it will be good. Otherwise, could >>> someone > help me to increase the component size? Thanks! Best, Jianfeng Jia PhD Candidate of Computer Science University of California, Irvine >> >> Best, >> >> Jianfeng Jia >> PhD Candidate of Computer Science >> University of California, Irvine >> >> >> Best, Jianfeng Jia PhD Candidate of Computer Science University of California, Irvine >> >
Re: How to set the lsm component size?
Thanks to Sattam, here is the revised version. Feel free to revise this. I will upload a patch set after some revision is done. *storage.memorycomponent.numpages* The number of pages to allocate for a memory component. (Default = 256) This budget is shared by all the memory components of the primary index and all its secondary indexes across all I/O devices on a node. Note: in-memory components usually has fill factor of 75% since the pages are 75% full and the remaining 25% is un-utilized. *storage.memorycomponent.globalbudget* [4GB + 100MB] The total size of memory in bytes that the sum of all open memory components cannot exceed. (Default = 512MB) Consider this as the buffer cache for all memory components of all indexes in a node. When this budget is fully used, a victim dataset will be chosen. It must be evicted and closed to make a space for another dataset. Best, Taewoo On Mon, Sep 12, 2016 at 4:10 PM, Mike Carey wrote: > +1 > > > > On 9/12/16 3:42 PM, Taewoo Kim wrote: > >> It would be really helpful this conversation can be applied in the >> description of each parameter. Currently, I think that is too short. >> >> Best, >> Taewoo >> >> On Mon, Sep 12, 2016 at 2:19 PM, Jianfeng Jia >> wrote: >> >> Clear. Thanks. >>> >>> And Ian’s parameters works. I can have a on-disk components around 128M. >>> Thanks! >>> >>> On Sep 12, 2016, at 12:50 PM, Sattam Alsubaiee >>> wrote: >>> This is the total memory size given for all datasets. Think of it as the buffer cache for all memory components of all indexes in that machine. >>> When >>> it is exhausted, a victim dataset must be evicted and closed to have a space for another dataset. On Mon, Sep 12, 2016 at 12:29 PM, Jianfeng Jia wrote: I was a little confused, there is another configuration: > > storage.memorycomponent.globalbudget ( which I set to 4G) > > I was thinking this is the budget that every component on one partition > is >>> shared. Is that the case? > > On Sep 12, 2016, at 12:16 PM, Sattam Alsubaiee >> > wrote: > >> The 128M is shared by all the memory components of the primary index >> > and >>> all its secondary indexes across all io devices on that node. >> Also the in-memory components usually usually has fill factor of 75% >> > since > >> the pages are 75% full and the remaining 25% is un-utilized. >> >> The page size that you have set 128KB looks reasonable for most cases. >> > Your > >> best bet is to increase the value of storage.memorycomponent.numpage >> > to >>> a > >> higher number. >> >> Sattam >> >> >> On Mon, Sep 12, 2016 at 11:33 AM, Jianfeng Jia < >> jianfeng@gmail.com >> wrote: >> >> Dear devs, >>> >>> I’m using the `no-merge` compaction policy and find that the physical >>> flushed on-disk component is smaller than I was expected. >>> >>> Here are my related configurations >>> >>> >>>storage.memorycomponent.pagesize >>>128KB >>>The page size in bytes for pages allocated to memory >>> components. (Default = "131072" // 128KB) >>> >>> >>> >>> >>>storage.memorycomponent.numpages >>>1024 >>>The number of pages to allocate for a memory >>> component. >>> (Default = 256) >>> >>> >>> >>> With these two settings, I’m expecting the lsm component should be >>> >> 128M. >>> However, the flushed one is about 16M~ 20M. Do we have some >>> >> compression >>> for > >> the on-disk components? If so, it will be good. Otherwise, could >>> >> someone >>> help me to increase the component size? Thanks! >>> >>> Best, >>> >>> Jianfeng Jia >>> PhD Candidate of Computer Science >>> University of California, Irvine >>> >>> >>> > > Best, > > Jianfeng Jia > PhD Candidate of Computer Science > University of California, Irvine > > > >>> >>> Best, >>> >>> Jianfeng Jia >>> PhD Candidate of Computer Science >>> University of California, Irvine >>> >>> >>> >
Re: How to set the lsm component size?
+1 On 9/12/16 3:42 PM, Taewoo Kim wrote: It would be really helpful this conversation can be applied in the description of each parameter. Currently, I think that is too short. Best, Taewoo On Mon, Sep 12, 2016 at 2:19 PM, Jianfeng Jia wrote: Clear. Thanks. And Ian’s parameters works. I can have a on-disk components around 128M. Thanks! On Sep 12, 2016, at 12:50 PM, Sattam Alsubaiee wrote: This is the total memory size given for all datasets. Think of it as the buffer cache for all memory components of all indexes in that machine. When it is exhausted, a victim dataset must be evicted and closed to have a space for another dataset. On Mon, Sep 12, 2016 at 12:29 PM, Jianfeng Jia wrote: I was a little confused, there is another configuration: storage.memorycomponent.globalbudget ( which I set to 4G) I was thinking this is the budget that every component on one partition is shared. Is that the case? On Sep 12, 2016, at 12:16 PM, Sattam Alsubaiee wrote: The 128M is shared by all the memory components of the primary index and all its secondary indexes across all io devices on that node. Also the in-memory components usually usually has fill factor of 75% since the pages are 75% full and the remaining 25% is un-utilized. The page size that you have set 128KB looks reasonable for most cases. Your best bet is to increase the value of storage.memorycomponent.numpage to a higher number. Sattam On Mon, Sep 12, 2016 at 11:33 AM, Jianfeng Jia Dear devs, I’m using the `no-merge` compaction policy and find that the physical flushed on-disk component is smaller than I was expected. Here are my related configurations storage.memorycomponent.pagesize 128KB The page size in bytes for pages allocated to memory components. (Default = "131072" // 128KB) storage.memorycomponent.numpages 1024 The number of pages to allocate for a memory component. (Default = 256) With these two settings, I’m expecting the lsm component should be 128M. However, the flushed one is about 16M~ 20M. Do we have some compression for the on-disk components? If so, it will be good. Otherwise, could someone help me to increase the component size? Thanks! Best, Jianfeng Jia PhD Candidate of Computer Science University of California, Irvine Best, Jianfeng Jia PhD Candidate of Computer Science University of California, Irvine Best, Jianfeng Jia PhD Candidate of Computer Science University of California, Irvine
Re: How to set the lsm component size?
It would be really helpful this conversation can be applied in the description of each parameter. Currently, I think that is too short. Best, Taewoo On Mon, Sep 12, 2016 at 2:19 PM, Jianfeng Jia wrote: > Clear. Thanks. > > And Ian’s parameters works. I can have a on-disk components around 128M. > Thanks! > > > On Sep 12, 2016, at 12:50 PM, Sattam Alsubaiee > wrote: > > > > This is the total memory size given for all datasets. Think of it as the > > buffer cache for all memory components of all indexes in that machine. > When > > it is exhausted, a victim dataset must be evicted and closed to have a > > space for another dataset. > > > > On Mon, Sep 12, 2016 at 12:29 PM, Jianfeng Jia > > wrote: > > > >> I was a little confused, there is another configuration: > >> > >> storage.memorycomponent.globalbudget ( which I set to 4G) > >> > >> I was thinking this is the budget that every component on one partition > is > >> shared. Is that the case? > >> > >>> On Sep 12, 2016, at 12:16 PM, Sattam Alsubaiee > >> wrote: > >>> > >>> The 128M is shared by all the memory components of the primary index > and > >>> all its secondary indexes across all io devices on that node. > >>> Also the in-memory components usually usually has fill factor of 75% > >> since > >>> the pages are 75% full and the remaining 25% is un-utilized. > >>> > >>> The page size that you have set 128KB looks reasonable for most cases. > >> Your > >>> best bet is to increase the value of storage.memorycomponent.numpage > to > >> a > >>> higher number. > >>> > >>> Sattam > >>> > >>> > >>> On Mon, Sep 12, 2016 at 11:33 AM, Jianfeng Jia > > >>> wrote: > >>> > Dear devs, > > I’m using the `no-merge` compaction policy and find that the physical > flushed on-disk component is smaller than I was expected. > > Here are my related configurations > > > storage.memorycomponent.pagesize > 128KB > The page size in bytes for pages allocated to memory > components. (Default = "131072" // 128KB) > > > > > storage.memorycomponent.numpages > 1024 > The number of pages to allocate for a memory component. > (Default = 256) > > > > With these two settings, I’m expecting the lsm component should be > 128M. > However, the flushed one is about 16M~ 20M. Do we have some > compression > >> for > the on-disk components? If so, it will be good. Otherwise, could > someone > help me to increase the component size? Thanks! > > Best, > > Jianfeng Jia > PhD Candidate of Computer Science > University of California, Irvine > > > >> > >> > >> > >> Best, > >> > >> Jianfeng Jia > >> PhD Candidate of Computer Science > >> University of California, Irvine > >> > >> > > > > Best, > > Jianfeng Jia > PhD Candidate of Computer Science > University of California, Irvine > >
Re: How to set the lsm component size?
Clear. Thanks. And Ian’s parameters works. I can have a on-disk components around 128M. Thanks! > On Sep 12, 2016, at 12:50 PM, Sattam Alsubaiee wrote: > > This is the total memory size given for all datasets. Think of it as the > buffer cache for all memory components of all indexes in that machine. When > it is exhausted, a victim dataset must be evicted and closed to have a > space for another dataset. > > On Mon, Sep 12, 2016 at 12:29 PM, Jianfeng Jia > wrote: > >> I was a little confused, there is another configuration: >> >> storage.memorycomponent.globalbudget ( which I set to 4G) >> >> I was thinking this is the budget that every component on one partition is >> shared. Is that the case? >> >>> On Sep 12, 2016, at 12:16 PM, Sattam Alsubaiee >> wrote: >>> >>> The 128M is shared by all the memory components of the primary index and >>> all its secondary indexes across all io devices on that node. >>> Also the in-memory components usually usually has fill factor of 75% >> since >>> the pages are 75% full and the remaining 25% is un-utilized. >>> >>> The page size that you have set 128KB looks reasonable for most cases. >> Your >>> best bet is to increase the value of storage.memorycomponent.numpage to >> a >>> higher number. >>> >>> Sattam >>> >>> >>> On Mon, Sep 12, 2016 at 11:33 AM, Jianfeng Jia >>> wrote: >>> Dear devs, I’m using the `no-merge` compaction policy and find that the physical flushed on-disk component is smaller than I was expected. Here are my related configurations storage.memorycomponent.pagesize 128KB The page size in bytes for pages allocated to memory components. (Default = "131072" // 128KB) storage.memorycomponent.numpages 1024 The number of pages to allocate for a memory component. (Default = 256) With these two settings, I’m expecting the lsm component should be 128M. However, the flushed one is about 16M~ 20M. Do we have some compression >> for the on-disk components? If so, it will be good. Otherwise, could someone help me to increase the component size? Thanks! Best, Jianfeng Jia PhD Candidate of Computer Science University of California, Irvine >> >> >> >> Best, >> >> Jianfeng Jia >> PhD Candidate of Computer Science >> University of California, Irvine >> >> Best, Jianfeng Jia PhD Candidate of Computer Science University of California, Irvine
Re: How to set the lsm component size?
This is the total memory size given for all datasets. Think of it as the buffer cache for all memory components of all indexes in that machine. When it is exhausted, a victim dataset must be evicted and closed to have a space for another dataset. On Mon, Sep 12, 2016 at 12:29 PM, Jianfeng Jia wrote: > I was a little confused, there is another configuration: > > storage.memorycomponent.globalbudget ( which I set to 4G) > > I was thinking this is the budget that every component on one partition is > shared. Is that the case? > > > On Sep 12, 2016, at 12:16 PM, Sattam Alsubaiee > wrote: > > > > The 128M is shared by all the memory components of the primary index and > > all its secondary indexes across all io devices on that node. > > Also the in-memory components usually usually has fill factor of 75% > since > > the pages are 75% full and the remaining 25% is un-utilized. > > > > The page size that you have set 128KB looks reasonable for most cases. > Your > > best bet is to increase the value of storage.memorycomponent.numpage to > a > > higher number. > > > > Sattam > > > > > > On Mon, Sep 12, 2016 at 11:33 AM, Jianfeng Jia > > wrote: > > > >> Dear devs, > >> > >> I’m using the `no-merge` compaction policy and find that the physical > >> flushed on-disk component is smaller than I was expected. > >> > >> Here are my related configurations > >> > >> > >>storage.memorycomponent.pagesize > >>128KB > >>The page size in bytes for pages allocated to memory > >> components. (Default = "131072" // 128KB) > >> > >> > >> > >> > >>storage.memorycomponent.numpages > >>1024 > >>The number of pages to allocate for a memory component. > >> (Default = 256) > >> > >> > >> > >> With these two settings, I’m expecting the lsm component should be 128M. > >> However, the flushed one is about 16M~ 20M. Do we have some compression > for > >> the on-disk components? If so, it will be good. Otherwise, could someone > >> help me to increase the component size? Thanks! > >> > >> Best, > >> > >> Jianfeng Jia > >> PhD Candidate of Computer Science > >> University of California, Irvine > >> > >> > > > > Best, > > Jianfeng Jia > PhD Candidate of Computer Science > University of California, Irvine > >
Re: How to set the lsm component size?
I was a little confused, there is another configuration: storage.memorycomponent.globalbudget ( which I set to 4G) I was thinking this is the budget that every component on one partition is shared. Is that the case? > On Sep 12, 2016, at 12:16 PM, Sattam Alsubaiee wrote: > > The 128M is shared by all the memory components of the primary index and > all its secondary indexes across all io devices on that node. > Also the in-memory components usually usually has fill factor of 75% since > the pages are 75% full and the remaining 25% is un-utilized. > > The page size that you have set 128KB looks reasonable for most cases. Your > best bet is to increase the value of storage.memorycomponent.numpage to a > higher number. > > Sattam > > > On Mon, Sep 12, 2016 at 11:33 AM, Jianfeng Jia > wrote: > >> Dear devs, >> >> I’m using the `no-merge` compaction policy and find that the physical >> flushed on-disk component is smaller than I was expected. >> >> Here are my related configurations >> >> >>storage.memorycomponent.pagesize >>128KB >>The page size in bytes for pages allocated to memory >> components. (Default = "131072" // 128KB) >> >> >> >> >>storage.memorycomponent.numpages >>1024 >>The number of pages to allocate for a memory component. >> (Default = 256) >> >> >> >> With these two settings, I’m expecting the lsm component should be 128M. >> However, the flushed one is about 16M~ 20M. Do we have some compression for >> the on-disk components? If so, it will be good. Otherwise, could someone >> help me to increase the component size? Thanks! >> >> Best, >> >> Jianfeng Jia >> PhD Candidate of Computer Science >> University of California, Irvine >> >> Best, Jianfeng Jia PhD Candidate of Computer Science University of California, Irvine
Re: How to set the lsm component size?
This parameter is related to the maximum allowed size for a disk component. Any disk component that is larger than this, won't be merged. On Mon, Sep 12, 2016 at 11:47 AM, Ian Maxon wrote: > It is a parameter to the compaction policy, which is an option during > dataset creation. The syntax is 'using compaction policy prefix > ((max-mergeable-component-size=" ( size in b) ", > ("max-tolerance-component- > count"="3"))' > > On Mon, Sep 12, 2016 at 11:33 AM, Jianfeng Jia > wrote: > > > Dear devs, > > > > I’m using the `no-merge` compaction policy and find that the physical > > flushed on-disk component is smaller than I was expected. > > > > Here are my related configurations > > > > > > storage.memorycomponent.pagesize > > 128KB > > The page size in bytes for pages allocated to memory > > components. (Default = "131072" // 128KB) > > > > > > > > > > storage.memorycomponent.numpages > > 1024 > > The number of pages to allocate for a memory component. > > (Default = 256) > > > > > > > > With these two settings, I’m expecting the lsm component should be 128M. > > However, the flushed one is about 16M~ 20M. Do we have some compression > for > > the on-disk components? If so, it will be good. Otherwise, could someone > > help me to increase the component size? Thanks! > > > > Best, > > > > Jianfeng Jia > > PhD Candidate of Computer Science > > University of California, Irvine > > > > >
Re: How to set the lsm component size?
The 128M is shared by all the memory components of the primary index and all its secondary indexes across all io devices on that node. Also the in-memory components usually usually has fill factor of 75% since the pages are 75% full and the remaining 25% is un-utilized. The page size that you have set 128KB looks reasonable for most cases. Your best bet is to increase the value of storage.memorycomponent.numpage to a higher number. Sattam On Mon, Sep 12, 2016 at 11:33 AM, Jianfeng Jia wrote: > Dear devs, > > I’m using the `no-merge` compaction policy and find that the physical > flushed on-disk component is smaller than I was expected. > > Here are my related configurations > > > storage.memorycomponent.pagesize > 128KB > The page size in bytes for pages allocated to memory > components. (Default = "131072" // 128KB) > > > > > storage.memorycomponent.numpages > 1024 > The number of pages to allocate for a memory component. > (Default = 256) > > > > With these two settings, I’m expecting the lsm component should be 128M. > However, the flushed one is about 16M~ 20M. Do we have some compression for > the on-disk components? If so, it will be good. Otherwise, could someone > help me to increase the component size? Thanks! > > Best, > > Jianfeng Jia > PhD Candidate of Computer Science > University of California, Irvine > >
Re: How to set the lsm component size?
It is a parameter to the compaction policy, which is an option during dataset creation. The syntax is 'using compaction policy prefix ((max-mergeable-component-size=" ( size in b) ", ("max-tolerance-component- count"="3"))' On Mon, Sep 12, 2016 at 11:33 AM, Jianfeng Jia wrote: > Dear devs, > > I’m using the `no-merge` compaction policy and find that the physical > flushed on-disk component is smaller than I was expected. > > Here are my related configurations > > > storage.memorycomponent.pagesize > 128KB > The page size in bytes for pages allocated to memory > components. (Default = "131072" // 128KB) > > > > > storage.memorycomponent.numpages > 1024 > The number of pages to allocate for a memory component. > (Default = 256) > > > > With these two settings, I’m expecting the lsm component should be 128M. > However, the flushed one is about 16M~ 20M. Do we have some compression for > the on-disk components? If so, it will be good. Otherwise, could someone > help me to increase the component size? Thanks! > > Best, > > Jianfeng Jia > PhD Candidate of Computer Science > University of California, Irvine > >