Re: How to set the lsm component size?

2016-10-13 Thread Jianfeng Jia
Nice!
> On Oct 13, 2016, at 2:56 PM, Taewoo Kim  wrote:
> 
> The explanation changes for the two parameters have been merged into the
> master.
> 
> https://asterix-gerrit.ics.uci.edu/#/c/1281/3/asterixdb/asterix-installer/src/main/resources/conf/asterix-configuration.xml
> 
> Best,
> Taewoo
> 
> On Mon, Sep 12, 2016 at 5:02 PM, Taewoo Kim  wrote:
> 
>> Thanks to Sattam, here is the revised version. Feel free to revise this. I
>> will upload a patch set after some revision is done.
>> 
>> *storage.memorycomponent.numpages*
>> 
>> The number of pages to allocate for a memory component. (Default = 256)
>> This budget is shared by all the memory components of the primary index
>> and all its secondary indexes across all I/O devices on a node.
>> Note: in-memory components usually has fill factor of 75% since the pages
>> are 75% full and the remaining 25% is un-utilized.
>> 
>> 
>> *storage.memorycomponent.globalbudget*
>> 
>> [4GB + 100MB] The total size of memory in bytes that the sum of all open
>> memory components cannot exceed. (Default = 512MB)
>> Consider this as the buffer cache for all memory components of all indexes
>> in a node.
>> When this budget is fully used, a victim dataset will be chosen. It must
>> be evicted and closed to make a space for another dataset.
>> 
>> 
>> Best,
>> Taewoo
>> 
>> On Mon, Sep 12, 2016 at 4:10 PM, Mike Carey  wrote:
>> 
>>> +1
>>> 
>>> 
>>> 
>>> On 9/12/16 3:42 PM, Taewoo Kim wrote:
>>> 
 It would be really helpful this conversation can be applied in the
 description of each parameter. Currently, I think that is too short.
 
 Best,
 Taewoo
 
 On Mon, Sep 12, 2016 at 2:19 PM, Jianfeng Jia 
 wrote:
 
 Clear. Thanks.
> 
> And Ian’s parameters works. I can have a on-disk components around 128M.
> Thanks!
> 
> On Sep 12, 2016, at 12:50 PM, Sattam Alsubaiee 
>> 
> wrote:
> 
>> This is the total memory size given for all datasets. Think of it as
>> the
>> buffer cache for all memory components of all indexes in that machine.
>> 
> When
> 
>> it is exhausted, a victim dataset must be evicted and closed to have a
>> space for another dataset.
>> 
>> On Mon, Sep 12, 2016 at 12:29 PM, Jianfeng Jia >> 
>> wrote:
>> 
>> I was a little confused, there is another configuration:
>>> 
>>> storage.memorycomponent.globalbudget ( which I set to 4G)
>>> 
>>> I was thinking this is the budget that every component on one
>>> partition
>>> 
>> is
> 
>> shared. Is that the case?
>>> 
>>> On Sep 12, 2016, at 12:16 PM, Sattam Alsubaiee 
 
>>> wrote:
>>> 
 The 128M is shared by all the memory components of the primary index
 
>>> and
> 
>> all its secondary indexes across all io devices on that node.
 Also the in-memory components usually usually has fill factor of 75%
 
>>> since
>>> 
 the pages are 75% full and the remaining 25% is un-utilized.
 
 The page size that you have set 128KB looks reasonable for most
 cases.
 
>>> Your
>>> 
 best bet is to increase the value of storage.memorycomponent.numpage
 
>>> to
> 
>> a
>>> 
 higher number.
 
 Sattam
 
 
 On Mon, Sep 12, 2016 at 11:33 AM, Jianfeng Jia <
 jianfeng@gmail.com
 wrote:
 
 Dear devs,
> 
> I’m using the `no-merge` compaction policy and find that the
> physical
> flushed on-disk component is smaller than I was expected.
> 
> Here are my related configurations
> 
> 
>   storage.memorycomponent.pagesize
>   128KB
>   The page size in bytes for pages allocated to memory
> components. (Default = "131072" // 128KB)
>   
> 
> 
> 
>   storage.memorycomponent.numpages
>   1024
>   The number of pages to allocate for a memory
> component.
> (Default = 256)
>   
> 
> 
> With these two settings, I’m expecting the lsm component should be
> 
 128M.
> 
>> However, the flushed one is about 16M~ 20M. Do we have some
> 
 compression
> 
>> for
>>> 
 the on-disk components? If so, it will be good. Otherwise, could
> 
 someone
> 
>> help me to increase the component size? Thanks!
> 
> Best,
> 
> Jianfeng Jia
> PhD Candidate of Computer Science
> University of California, Irvine
> 
> 
> 
>>> 
>>> Best,
>>> 
>>> Jianfeng Jia
>>> PhD Candidate of Computer Science
>>> University of California, Irvine
>>> 
>>> 
>>> 
> 
> Best,
> 
> Jianfeng Jia
> P

Re: How to set the lsm component size?

2016-10-13 Thread Taewoo Kim
The explanation changes for the two parameters have been merged into the
master.

https://asterix-gerrit.ics.uci.edu/#/c/1281/3/asterixdb/asterix-installer/src/main/resources/conf/asterix-configuration.xml

Best,
Taewoo

On Mon, Sep 12, 2016 at 5:02 PM, Taewoo Kim  wrote:

> Thanks to Sattam, here is the revised version. Feel free to revise this. I
> will upload a patch set after some revision is done.
>
> *storage.memorycomponent.numpages*
>
> The number of pages to allocate for a memory component. (Default = 256)
> This budget is shared by all the memory components of the primary index
> and all its secondary indexes across all I/O devices on a node.
> Note: in-memory components usually has fill factor of 75% since the pages
> are 75% full and the remaining 25% is un-utilized.
>
>
> *storage.memorycomponent.globalbudget*
>
> [4GB + 100MB] The total size of memory in bytes that the sum of all open
> memory components cannot exceed. (Default = 512MB)
> Consider this as the buffer cache for all memory components of all indexes
> in a node.
> When this budget is fully used, a victim dataset will be chosen. It must
> be evicted and closed to make a space for another dataset.
>
>
> Best,
> Taewoo
>
> On Mon, Sep 12, 2016 at 4:10 PM, Mike Carey  wrote:
>
>> +1
>>
>>
>>
>> On 9/12/16 3:42 PM, Taewoo Kim wrote:
>>
>>> It would be really helpful this conversation can be applied in the
>>> description of each parameter. Currently, I think that is too short.
>>>
>>> Best,
>>> Taewoo
>>>
>>> On Mon, Sep 12, 2016 at 2:19 PM, Jianfeng Jia 
>>> wrote:
>>>
>>> Clear. Thanks.

 And Ian’s parameters works. I can have a on-disk components around 128M.
 Thanks!

 On Sep 12, 2016, at 12:50 PM, Sattam Alsubaiee 
>
 wrote:

> This is the total memory size given for all datasets. Think of it as
> the
> buffer cache for all memory components of all indexes in that machine.
>
 When

> it is exhausted, a victim dataset must be evicted and closed to have a
> space for another dataset.
>
> On Mon, Sep 12, 2016 at 12:29 PM, Jianfeng Jia  >
> wrote:
>
> I was a little confused, there is another configuration:
>>
>> storage.memorycomponent.globalbudget ( which I set to 4G)
>>
>> I was thinking this is the budget that every component on one
>> partition
>>
> is

> shared. Is that the case?
>>
>> On Sep 12, 2016, at 12:16 PM, Sattam Alsubaiee 
>>>
>> wrote:
>>
>>> The 128M is shared by all the memory components of the primary index
>>>
>> and

> all its secondary indexes across all io devices on that node.
>>> Also the in-memory components usually usually has fill factor of 75%
>>>
>> since
>>
>>> the pages are 75% full and the remaining 25% is un-utilized.
>>>
>>> The page size that you have set 128KB looks reasonable for most
>>> cases.
>>>
>> Your
>>
>>> best bet is to increase the value of storage.memorycomponent.numpage
>>>
>> to

> a
>>
>>> higher number.
>>>
>>> Sattam
>>>
>>>
>>> On Mon, Sep 12, 2016 at 11:33 AM, Jianfeng Jia <
>>> jianfeng@gmail.com
>>> wrote:
>>>
>>> Dear devs,

 I’m using the `no-merge` compaction policy and find that the
 physical
 flushed on-disk component is smaller than I was expected.

 Here are my related configurations

 
storage.memorycomponent.pagesize
128KB
The page size in bytes for pages allocated to memory
  components. (Default = "131072" // 128KB)

 

 
storage.memorycomponent.numpages
1024
The number of pages to allocate for a memory
 component.
  (Default = 256)

 

 With these two settings, I’m expecting the lsm component should be

>>> 128M.

> However, the flushed one is about 16M~ 20M. Do we have some

>>> compression

> for
>>
>>> the on-disk components? If so, it will be good. Otherwise, could

>>> someone

> help me to increase the component size? Thanks!

 Best,

 Jianfeng Jia
 PhD Candidate of Computer Science
 University of California, Irvine



>>
>> Best,
>>
>> Jianfeng Jia
>> PhD Candidate of Computer Science
>> University of California, Irvine
>>
>>
>>

 Best,

 Jianfeng Jia
 PhD Candidate of Computer Science
 University of California, Irvine



>>
>


Re: How to set the lsm component size?

2016-09-12 Thread Taewoo Kim
Thanks to Sattam, here is the revised version. Feel free to revise this. I
will upload a patch set after some revision is done.

*storage.memorycomponent.numpages*

The number of pages to allocate for a memory component. (Default = 256)
This budget is shared by all the memory components of the primary index and
all its secondary indexes across all I/O devices on a node.
Note: in-memory components usually has fill factor of 75% since the pages
are 75% full and the remaining 25% is un-utilized.


*storage.memorycomponent.globalbudget*

[4GB + 100MB] The total size of memory in bytes that the sum of all open
memory components cannot exceed. (Default = 512MB)
Consider this as the buffer cache for all memory components of all indexes
in a node.
When this budget is fully used, a victim dataset will be chosen. It must be
evicted and closed to make a space for another dataset.


Best,
Taewoo

On Mon, Sep 12, 2016 at 4:10 PM, Mike Carey  wrote:

> +1
>
>
>
> On 9/12/16 3:42 PM, Taewoo Kim wrote:
>
>> It would be really helpful this conversation can be applied in the
>> description of each parameter. Currently, I think that is too short.
>>
>> Best,
>> Taewoo
>>
>> On Mon, Sep 12, 2016 at 2:19 PM, Jianfeng Jia 
>> wrote:
>>
>> Clear. Thanks.
>>>
>>> And Ian’s parameters works. I can have a on-disk components around 128M.
>>> Thanks!
>>>
>>> On Sep 12, 2016, at 12:50 PM, Sattam Alsubaiee 

>>> wrote:
>>>
 This is the total memory size given for all datasets. Think of it as the
 buffer cache for all memory components of all indexes in that machine.

>>> When
>>>
 it is exhausted, a victim dataset must be evicted and closed to have a
 space for another dataset.

 On Mon, Sep 12, 2016 at 12:29 PM, Jianfeng Jia 
 wrote:

 I was a little confused, there is another configuration:
>
> storage.memorycomponent.globalbudget ( which I set to 4G)
>
> I was thinking this is the budget that every component on one partition
>
 is
>>>
 shared. Is that the case?
>
> On Sep 12, 2016, at 12:16 PM, Sattam Alsubaiee 
>>
> wrote:
>
>> The 128M is shared by all the memory components of the primary index
>>
> and
>>>
 all its secondary indexes across all io devices on that node.
>> Also the in-memory components usually usually has fill factor of 75%
>>
> since
>
>> the pages are 75% full and the remaining 25% is un-utilized.
>>
>> The page size that you have set 128KB looks reasonable for most cases.
>>
> Your
>
>> best bet is to increase the value of storage.memorycomponent.numpage
>>
> to
>>>
 a
>
>> higher number.
>>
>> Sattam
>>
>>
>> On Mon, Sep 12, 2016 at 11:33 AM, Jianfeng Jia <
>> jianfeng@gmail.com
>> wrote:
>>
>> Dear devs,
>>>
>>> I’m using the `no-merge` compaction policy and find that the physical
>>> flushed on-disk component is smaller than I was expected.
>>>
>>> Here are my related configurations
>>>
>>> 
>>>storage.memorycomponent.pagesize
>>>128KB
>>>The page size in bytes for pages allocated to memory
>>>  components. (Default = "131072" // 128KB)
>>>
>>> 
>>>
>>> 
>>>storage.memorycomponent.numpages
>>>1024
>>>The number of pages to allocate for a memory
>>> component.
>>>  (Default = 256)
>>>
>>> 
>>>
>>> With these two settings, I’m expecting the lsm component should be
>>>
>> 128M.
>>>
 However, the flushed one is about 16M~ 20M. Do we have some
>>>
>> compression
>>>
 for
>
>> the on-disk components? If so, it will be good. Otherwise, could
>>>
>> someone
>>>
 help me to increase the component size? Thanks!
>>>
>>> Best,
>>>
>>> Jianfeng Jia
>>> PhD Candidate of Computer Science
>>> University of California, Irvine
>>>
>>>
>>>
>
> Best,
>
> Jianfeng Jia
> PhD Candidate of Computer Science
> University of California, Irvine
>
>
>
>>>
>>> Best,
>>>
>>> Jianfeng Jia
>>> PhD Candidate of Computer Science
>>> University of California, Irvine
>>>
>>>
>>>
>


Re: How to set the lsm component size?

2016-09-12 Thread Mike Carey

+1


On 9/12/16 3:42 PM, Taewoo Kim wrote:

It would be really helpful this conversation can be applied in the
description of each parameter. Currently, I think that is too short.

Best,
Taewoo

On Mon, Sep 12, 2016 at 2:19 PM, Jianfeng Jia 
wrote:


Clear. Thanks.

And Ian’s parameters works. I can have a on-disk components around 128M.
Thanks!


On Sep 12, 2016, at 12:50 PM, Sattam Alsubaiee 

wrote:

This is the total memory size given for all datasets. Think of it as the
buffer cache for all memory components of all indexes in that machine.

When

it is exhausted, a victim dataset must be evicted and closed to have a
space for another dataset.

On Mon, Sep 12, 2016 at 12:29 PM, Jianfeng Jia 
wrote:


I was a little confused, there is another configuration:

storage.memorycomponent.globalbudget ( which I set to 4G)

I was thinking this is the budget that every component on one partition

is

shared. Is that the case?


On Sep 12, 2016, at 12:16 PM, Sattam Alsubaiee 

wrote:

The 128M is shared by all the memory components of the primary index

and

all its secondary indexes across all io devices on that node.
Also the in-memory components usually usually has fill factor of 75%

since

the pages are 75% full and the remaining 25% is un-utilized.

The page size that you have set 128KB looks reasonable for most cases.

Your

best bet is to increase the value of storage.memorycomponent.numpage

to

a

higher number.

Sattam


On Mon, Sep 12, 2016 at 11:33 AM, Jianfeng Jia 
Dear devs,

I’m using the `no-merge` compaction policy and find that the physical
flushed on-disk component is smaller than I was expected.

Here are my related configurations


   storage.memorycomponent.pagesize
   128KB
   The page size in bytes for pages allocated to memory
 components. (Default = "131072" // 128KB)
   



   storage.memorycomponent.numpages
   1024
   The number of pages to allocate for a memory component.
 (Default = 256)
   


With these two settings, I’m expecting the lsm component should be

128M.

However, the flushed one is about 16M~ 20M. Do we have some

compression

for

the on-disk components? If so, it will be good. Otherwise, could

someone

help me to increase the component size? Thanks!

Best,

Jianfeng Jia
PhD Candidate of Computer Science
University of California, Irvine





Best,

Jianfeng Jia
PhD Candidate of Computer Science
University of California, Irvine





Best,

Jianfeng Jia
PhD Candidate of Computer Science
University of California, Irvine






Re: How to set the lsm component size?

2016-09-12 Thread Taewoo Kim
It would be really helpful this conversation can be applied in the
description of each parameter. Currently, I think that is too short.

Best,
Taewoo

On Mon, Sep 12, 2016 at 2:19 PM, Jianfeng Jia 
wrote:

> Clear. Thanks.
>
> And Ian’s parameters works. I can have a on-disk components around 128M.
> Thanks!
>
> > On Sep 12, 2016, at 12:50 PM, Sattam Alsubaiee 
> wrote:
> >
> > This is the total memory size given for all datasets. Think of it as the
> > buffer cache for all memory components of all indexes in that machine.
> When
> > it is exhausted, a victim dataset must be evicted and closed to have a
> > space for another dataset.
> >
> > On Mon, Sep 12, 2016 at 12:29 PM, Jianfeng Jia 
> > wrote:
> >
> >> I was a little confused, there is another configuration:
> >>
> >> storage.memorycomponent.globalbudget ( which I set to 4G)
> >>
> >> I was thinking this is the budget that every component on one partition
> is
> >> shared. Is that the case?
> >>
> >>> On Sep 12, 2016, at 12:16 PM, Sattam Alsubaiee 
> >> wrote:
> >>>
> >>> The 128M is shared by all the memory components of the primary index
> and
> >>> all its secondary indexes across all io devices on that node.
> >>> Also the in-memory components usually usually has fill factor of 75%
> >> since
> >>> the pages are 75% full and the remaining 25% is un-utilized.
> >>>
> >>> The page size that you have set 128KB looks reasonable for most cases.
> >> Your
> >>> best bet is to increase the value of storage.memorycomponent.numpage
> to
> >> a
> >>> higher number.
> >>>
> >>> Sattam
> >>>
> >>>
> >>> On Mon, Sep 12, 2016 at 11:33 AM, Jianfeng Jia  >
> >>> wrote:
> >>>
>  Dear devs,
> 
>  I’m using the `no-merge` compaction policy and find that the physical
>  flushed on-disk component is smaller than I was expected.
> 
>  Here are my related configurations
> 
>  
>    storage.memorycomponent.pagesize
>    128KB
>    The page size in bytes for pages allocated to memory
>  components. (Default = "131072" // 128KB)
>    
>  
> 
>  
>    storage.memorycomponent.numpages
>    1024
>    The number of pages to allocate for a memory component.
>  (Default = 256)
>    
>  
> 
>  With these two settings, I’m expecting the lsm component should be
> 128M.
>  However, the flushed one is about 16M~ 20M. Do we have some
> compression
> >> for
>  the on-disk components? If so, it will be good. Otherwise, could
> someone
>  help me to increase the component size? Thanks!
> 
>  Best,
> 
>  Jianfeng Jia
>  PhD Candidate of Computer Science
>  University of California, Irvine
> 
> 
> >>
> >>
> >>
> >> Best,
> >>
> >> Jianfeng Jia
> >> PhD Candidate of Computer Science
> >> University of California, Irvine
> >>
> >>
>
>
>
> Best,
>
> Jianfeng Jia
> PhD Candidate of Computer Science
> University of California, Irvine
>
>


Re: How to set the lsm component size?

2016-09-12 Thread Jianfeng Jia
Clear. Thanks.

And Ian’s parameters works. I can have a on-disk components around 128M. Thanks!

> On Sep 12, 2016, at 12:50 PM, Sattam Alsubaiee  wrote:
> 
> This is the total memory size given for all datasets. Think of it as the
> buffer cache for all memory components of all indexes in that machine. When
> it is exhausted, a victim dataset must be evicted and closed to have a
> space for another dataset.
> 
> On Mon, Sep 12, 2016 at 12:29 PM, Jianfeng Jia 
> wrote:
> 
>> I was a little confused, there is another configuration:
>> 
>> storage.memorycomponent.globalbudget ( which I set to 4G)
>> 
>> I was thinking this is the budget that every component on one partition is
>> shared. Is that the case?
>> 
>>> On Sep 12, 2016, at 12:16 PM, Sattam Alsubaiee 
>> wrote:
>>> 
>>> The 128M is shared by all the memory components of the primary index and
>>> all its secondary indexes across all io devices on that node.
>>> Also the in-memory components usually usually has fill factor of 75%
>> since
>>> the pages are 75% full and the remaining 25% is un-utilized.
>>> 
>>> The page size that you have set 128KB looks reasonable for most cases.
>> Your
>>> best bet is to increase the value of storage.memorycomponent.numpage to
>> a
>>> higher number.
>>> 
>>> Sattam
>>> 
>>> 
>>> On Mon, Sep 12, 2016 at 11:33 AM, Jianfeng Jia 
>>> wrote:
>>> 
 Dear devs,
 
 I’m using the `no-merge` compaction policy and find that the physical
 flushed on-disk component is smaller than I was expected.
 
 Here are my related configurations
 
 
   storage.memorycomponent.pagesize
   128KB
   The page size in bytes for pages allocated to memory
 components. (Default = "131072" // 128KB)
   
 
 
 
   storage.memorycomponent.numpages
   1024
   The number of pages to allocate for a memory component.
 (Default = 256)
   
 
 
 With these two settings, I’m expecting the lsm component should be 128M.
 However, the flushed one is about 16M~ 20M. Do we have some compression
>> for
 the on-disk components? If so, it will be good. Otherwise, could someone
 help me to increase the component size? Thanks!
 
 Best,
 
 Jianfeng Jia
 PhD Candidate of Computer Science
 University of California, Irvine
 
 
>> 
>> 
>> 
>> Best,
>> 
>> Jianfeng Jia
>> PhD Candidate of Computer Science
>> University of California, Irvine
>> 
>> 



Best,

Jianfeng Jia
PhD Candidate of Computer Science
University of California, Irvine



Re: How to set the lsm component size?

2016-09-12 Thread Sattam Alsubaiee
This is the total memory size given for all datasets. Think of it as the
buffer cache for all memory components of all indexes in that machine. When
it is exhausted, a victim dataset must be evicted and closed to have a
space for another dataset.

On Mon, Sep 12, 2016 at 12:29 PM, Jianfeng Jia 
wrote:

> I was a little confused, there is another configuration:
>
> storage.memorycomponent.globalbudget ( which I set to 4G)
>
> I was thinking this is the budget that every component on one partition is
> shared. Is that the case?
>
> > On Sep 12, 2016, at 12:16 PM, Sattam Alsubaiee 
> wrote:
> >
> > The 128M is shared by all the memory components of the primary index and
> > all its secondary indexes across all io devices on that node.
> > Also the in-memory components usually usually has fill factor of 75%
> since
> > the pages are 75% full and the remaining 25% is un-utilized.
> >
> > The page size that you have set 128KB looks reasonable for most cases.
> Your
> > best bet is to increase the value of storage.memorycomponent.numpage to
> a
> > higher number.
> >
> > Sattam
> >
> >
> > On Mon, Sep 12, 2016 at 11:33 AM, Jianfeng Jia 
> > wrote:
> >
> >> Dear devs,
> >>
> >> I’m using the `no-merge` compaction policy and find that the physical
> >> flushed on-disk component is smaller than I was expected.
> >>
> >> Here are my related configurations
> >>
> >>  
> >>storage.memorycomponent.pagesize
> >>128KB
> >>The page size in bytes for pages allocated to memory
> >>  components. (Default = "131072" // 128KB)
> >>
> >>  
> >>
> >>  
> >>storage.memorycomponent.numpages
> >>1024
> >>The number of pages to allocate for a memory component.
> >>  (Default = 256)
> >>
> >>  
> >>
> >> With these two settings, I’m expecting the lsm component should be 128M.
> >> However, the flushed one is about 16M~ 20M. Do we have some compression
> for
> >> the on-disk components? If so, it will be good. Otherwise, could someone
> >> help me to increase the component size? Thanks!
> >>
> >> Best,
> >>
> >> Jianfeng Jia
> >> PhD Candidate of Computer Science
> >> University of California, Irvine
> >>
> >>
>
>
>
> Best,
>
> Jianfeng Jia
> PhD Candidate of Computer Science
> University of California, Irvine
>
>


Re: How to set the lsm component size?

2016-09-12 Thread Jianfeng Jia
I was a little confused, there is another configuration:

storage.memorycomponent.globalbudget ( which I set to 4G) 

I was thinking this is the budget that every component on one partition is 
shared. Is that the case?

> On Sep 12, 2016, at 12:16 PM, Sattam Alsubaiee  wrote:
> 
> The 128M is shared by all the memory components of the primary index and
> all its secondary indexes across all io devices on that node.
> Also the in-memory components usually usually has fill factor of 75% since
> the pages are 75% full and the remaining 25% is un-utilized.
> 
> The page size that you have set 128KB looks reasonable for most cases. Your
> best bet is to increase the value of storage.memorycomponent.numpage to a
> higher number.
> 
> Sattam
> 
> 
> On Mon, Sep 12, 2016 at 11:33 AM, Jianfeng Jia 
> wrote:
> 
>> Dear devs,
>> 
>> I’m using the `no-merge` compaction policy and find that the physical
>> flushed on-disk component is smaller than I was expected.
>> 
>> Here are my related configurations
>> 
>>  
>>storage.memorycomponent.pagesize
>>128KB
>>The page size in bytes for pages allocated to memory
>>  components. (Default = "131072" // 128KB)
>>
>>  
>> 
>>  
>>storage.memorycomponent.numpages
>>1024
>>The number of pages to allocate for a memory component.
>>  (Default = 256)
>>
>>  
>> 
>> With these two settings, I’m expecting the lsm component should be 128M.
>> However, the flushed one is about 16M~ 20M. Do we have some compression for
>> the on-disk components? If so, it will be good. Otherwise, could someone
>> help me to increase the component size? Thanks!
>> 
>> Best,
>> 
>> Jianfeng Jia
>> PhD Candidate of Computer Science
>> University of California, Irvine
>> 
>> 



Best,

Jianfeng Jia
PhD Candidate of Computer Science
University of California, Irvine



Re: How to set the lsm component size?

2016-09-12 Thread Sattam Alsubaiee
This parameter is related to the maximum allowed size for a disk component.
Any disk component that is larger than this, won't be merged.

On Mon, Sep 12, 2016 at 11:47 AM, Ian Maxon  wrote:

> It is a parameter to the compaction policy, which is an option during
> dataset creation. The syntax is 'using compaction policy prefix
> ((max-mergeable-component-size=" ( size in b)  ",
> ("max-tolerance-component-
> count"="3"))'
>
> On Mon, Sep 12, 2016 at 11:33 AM, Jianfeng Jia 
> wrote:
>
> > Dear devs,
> >
> > I’m using the `no-merge` compaction policy and find that the physical
> > flushed on-disk component is smaller than I was expected.
> >
> > Here are my related configurations
> >
> >   
> > storage.memorycomponent.pagesize
> > 128KB
> > The page size in bytes for pages allocated to memory
> >   components. (Default = "131072" // 128KB)
> > 
> >   
> >
> >   
> > storage.memorycomponent.numpages
> > 1024
> > The number of pages to allocate for a memory component.
> >   (Default = 256)
> > 
> >   
> >
> > With these two settings, I’m expecting the lsm component should be 128M.
> > However, the flushed one is about 16M~ 20M. Do we have some compression
> for
> > the on-disk components? If so, it will be good. Otherwise, could someone
> > help me to increase the component size? Thanks!
> >
> > Best,
> >
> > Jianfeng Jia
> > PhD Candidate of Computer Science
> > University of California, Irvine
> >
> >
>


Re: How to set the lsm component size?

2016-09-12 Thread Sattam Alsubaiee
The 128M is shared by all the memory components of the primary index and
all its secondary indexes across all io devices on that node.
Also the in-memory components usually usually has fill factor of 75% since
the pages are 75% full and the remaining 25% is un-utilized.

The page size that you have set 128KB looks reasonable for most cases. Your
best bet is to increase the value of storage.memorycomponent.numpage to a
higher number.

Sattam


On Mon, Sep 12, 2016 at 11:33 AM, Jianfeng Jia 
wrote:

> Dear devs,
>
> I’m using the `no-merge` compaction policy and find that the physical
> flushed on-disk component is smaller than I was expected.
>
> Here are my related configurations
>
>   
> storage.memorycomponent.pagesize
> 128KB
> The page size in bytes for pages allocated to memory
>   components. (Default = "131072" // 128KB)
> 
>   
>
>   
> storage.memorycomponent.numpages
> 1024
> The number of pages to allocate for a memory component.
>   (Default = 256)
> 
>   
>
> With these two settings, I’m expecting the lsm component should be 128M.
> However, the flushed one is about 16M~ 20M. Do we have some compression for
> the on-disk components? If so, it will be good. Otherwise, could someone
> help me to increase the component size? Thanks!
>
> Best,
>
> Jianfeng Jia
> PhD Candidate of Computer Science
> University of California, Irvine
>
>


Re: How to set the lsm component size?

2016-09-12 Thread Ian Maxon
It is a parameter to the compaction policy, which is an option during
dataset creation. The syntax is 'using compaction policy prefix
((max-mergeable-component-size=" ( size in b)  ", ("max-tolerance-component-
count"="3"))'

On Mon, Sep 12, 2016 at 11:33 AM, Jianfeng Jia 
wrote:

> Dear devs,
>
> I’m using the `no-merge` compaction policy and find that the physical
> flushed on-disk component is smaller than I was expected.
>
> Here are my related configurations
>
>   
> storage.memorycomponent.pagesize
> 128KB
> The page size in bytes for pages allocated to memory
>   components. (Default = "131072" // 128KB)
> 
>   
>
>   
> storage.memorycomponent.numpages
> 1024
> The number of pages to allocate for a memory component.
>   (Default = 256)
> 
>   
>
> With these two settings, I’m expecting the lsm component should be 128M.
> However, the flushed one is about 16M~ 20M. Do we have some compression for
> the on-disk components? If so, it will be good. Otherwise, could someone
> help me to increase the component size? Thanks!
>
> Best,
>
> Jianfeng Jia
> PhD Candidate of Computer Science
> University of California, Irvine
>
>