[jira] [Commented] (CASSANDRA-7882) Memtable slab allocation should scale logarithmically to improve occupancy rate
[ https://issues.apache.org/jira/browse/CASSANDRA-7882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14239891#comment-14239891 ] Benedict commented on CASSANDRA-7882: - Yes, I don't think that's a problem. > Memtable slab allocation should scale logarithmically to improve occupancy > rate > --- > > Key: CASSANDRA-7882 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7882 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Jay Patel >Assignee: Benedict > Labels: performance > Fix For: 2.1.3 > > Attachments: trunk-7882.txt > > > CASSANDRA-5935 allows option to disable region-based allocation for on-heap > memtables but there is no option to disable it for off-heap memtables > (memtable_allocation_type: offheap_objects). > Disabling region-based allocation will allow us to pack more tables in the > schema since minimum of 1MB region won't be allocated per table. Downside can > be more fragmentation which should be controllable by using better allocator > like JEMalloc. > How about below option in yaml?: > memtable_allocation_type: unslabbed_offheap_objects > Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7882) Memtable slab allocation should scale logarithmically to improve occupancy rate
[ https://issues.apache.org/jira/browse/CASSANDRA-7882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14239871#comment-14239871 ] Jay Patel commented on CASSANDRA-7882: -- Ah, CASSANDRA-7883 is for 3.0. Can we have it merge to 2.1.3? > Memtable slab allocation should scale logarithmically to improve occupancy > rate > --- > > Key: CASSANDRA-7882 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7882 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Jay Patel >Assignee: Benedict > Labels: performance > Fix For: 2.1.3 > > Attachments: trunk-7882.txt > > > CASSANDRA-5935 allows option to disable region-based allocation for on-heap > memtables but there is no option to disable it for off-heap memtables > (memtable_allocation_type: offheap_objects). > Disabling region-based allocation will allow us to pack more tables in the > schema since minimum of 1MB region won't be allocated per table. Downside can > be more fragmentation which should be controllable by using better allocator > like JEMalloc. > How about below option in yaml?: > memtable_allocation_type: unslabbed_offheap_objects > Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7882) Memtable slab allocation should scale logarithmically to improve occupancy rate
[ https://issues.apache.org/jira/browse/CASSANDRA-7882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14239863#comment-14239863 ] Jay Patel commented on CASSANDRA-7882: -- Thanks [~benedict]. Changes look good except I see it uses unsafe.allocateMemory directly instead of IAllocator interface (CASSANDRA-7883). We're planning to run load test with this final patch from next week. > Memtable slab allocation should scale logarithmically to improve occupancy > rate > --- > > Key: CASSANDRA-7882 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7882 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Jay Patel >Assignee: Benedict > Labels: performance > Fix For: 2.1.3 > > Attachments: trunk-7882.txt > > > CASSANDRA-5935 allows option to disable region-based allocation for on-heap > memtables but there is no option to disable it for off-heap memtables > (memtable_allocation_type: offheap_objects). > Disabling region-based allocation will allow us to pack more tables in the > schema since minimum of 1MB region won't be allocated per table. Downside can > be more fragmentation which should be controllable by using better allocator > like JEMalloc. > How about below option in yaml?: > memtable_allocation_type: unslabbed_offheap_objects > Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7882) Memtable slab allocation should scale logarithmically to improve occupancy rate
[ https://issues.apache.org/jira/browse/CASSANDRA-7882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14236526#comment-14236526 ] Jonathan Ellis commented on CASSANDRA-7882: --- Since this only affects NativeAllocator, which is kind of experimental in 2.1 anyway, I'm okay with committing for 2.1.3. > Memtable slab allocation should scale logarithmically to improve occupancy > rate > --- > > Key: CASSANDRA-7882 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7882 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Jay Patel >Assignee: Benedict > Labels: performance > Fix For: 2.1.3 > > Attachments: trunk-7882.txt > > > CASSANDRA-5935 allows option to disable region-based allocation for on-heap > memtables but there is no option to disable it for off-heap memtables > (memtable_allocation_type: offheap_objects). > Disabling region-based allocation will allow us to pack more tables in the > schema since minimum of 1MB region won't be allocated per table. Downside can > be more fragmentation which should be controllable by using better allocator > like JEMalloc. > How about below option in yaml?: > memtable_allocation_type: unslabbed_offheap_objects > Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7882) Memtable slab allocation should scale logarithmically to improve occupancy rate
[ https://issues.apache.org/jira/browse/CASSANDRA-7882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14233170#comment-14233170 ] Benedict commented on CASSANDRA-7882: - I've posted a variant of the patch [here|https://github.com/belliottsmith/cassandra/tree/7882-nativeallocator] There are a few changes, a couple unrelated just cleaning up the class: # removed the unslabbed and regionCount variables, as they weren't used for anything important # removed the nextRegionSize variable: it wasn't being maintained atomically, but just as importantly it's messy to do it separately: #* instead of setting a full region to null, we swap it straight to a new region, using the prior region to determine the size of the new region #* we ensure the new region size is at least large enough to hold the allocation we're inserting # we cap the size of each "race allocated" queue to 8 entries, as this should permit plenty of leeway for avoiding heavy competition thrashing the allocator, but not so much that we have a lot of primarily unused memory There is one issue, though, which is if this should make it into 2.1, or wait until 3.0. I'm pretty comfortable either way, but my gut feeling is others will prefer it wait until 3.0. [~jbellis], what's your view? > Memtable slab allocation should scale logarithmically to improve occupancy > rate > --- > > Key: CASSANDRA-7882 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7882 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Jay Patel >Assignee: Jay Patel > Labels: performance > Fix For: 2.1.3 > > Attachments: trunk-7882.txt > > > CASSANDRA-5935 allows option to disable region-based allocation for on-heap > memtables but there is no option to disable it for off-heap memtables > (memtable_allocation_type: offheap_objects). > Disabling region-based allocation will allow us to pack more tables in the > schema since minimum of 1MB region won't be allocated per table. Downside can > be more fragmentation which should be controllable by using better allocator > like JEMalloc. > How about below option in yaml?: > memtable_allocation_type: unslabbed_offheap_objects > Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7882) Memtable slab allocation should scale logarithmically to improve occupancy rate
[ https://issues.apache.org/jira/browse/CASSANDRA-7882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14231436#comment-14231436 ] Jay Patel commented on CASSANDRA-7882: -- Awesome. Thanks Benedict! Let me know if you face any issue while applying patch. > Memtable slab allocation should scale logarithmically to improve occupancy > rate > --- > > Key: CASSANDRA-7882 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7882 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Jay Patel >Assignee: Jay Patel > Labels: performance > Fix For: 2.1.3 > > Attachments: trunk-7882.txt > > > CASSANDRA-5935 allows option to disable region-based allocation for on-heap > memtables but there is no option to disable it for off-heap memtables > (memtable_allocation_type: offheap_objects). > Disabling region-based allocation will allow us to pack more tables in the > schema since minimum of 1MB region won't be allocated per table. Downside can > be more fragmentation which should be controllable by using better allocator > like JEMalloc. > How about below option in yaml?: > memtable_allocation_type: unslabbed_offheap_objects > Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7882) Memtable slab allocation should scale logarithmically to improve occupancy rate
[ https://issues.apache.org/jira/browse/CASSANDRA-7882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14231374#comment-14231374 ] Benedict commented on CASSANDRA-7882: - Hi Jay, I've been away for the past two months, so sorry this got left by the wayside in the meantime. I'll get around to reviewing it shortly. > Memtable slab allocation should scale logarithmically to improve occupancy > rate > --- > > Key: CASSANDRA-7882 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7882 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Jay Patel >Assignee: Jay Patel > Labels: performance > Fix For: 2.1.3 > > Attachments: trunk-7882.txt > > > CASSANDRA-5935 allows option to disable region-based allocation for on-heap > memtables but there is no option to disable it for off-heap memtables > (memtable_allocation_type: offheap_objects). > Disabling region-based allocation will allow us to pack more tables in the > schema since minimum of 1MB region won't be allocated per table. Downside can > be more fragmentation which should be controllable by using better allocator > like JEMalloc. > How about below option in yaml?: > memtable_allocation_type: unslabbed_offheap_objects > Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7882) Memtable slab allocation should scale logarithmically to improve occupancy rate
[ https://issues.apache.org/jira/browse/CASSANDRA-7882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14166426#comment-14166426 ] Jay Patel commented on CASSANDRA-7882: -- [~benedict], we've done initial testing with 3K tables, running for a day & looks good so far. Should we keep the start region size configurable, instead of hard-coded 8kb? This will allow fine tuning specific use case, for instance - If payload is always around 400 to 500kb, user can have start size as 512kb instead of scaling from 8kb. Actually, we've one use case like that. Let me know your thoughts or I'll submit a final patch with the todos completed. > Memtable slab allocation should scale logarithmically to improve occupancy > rate > --- > > Key: CASSANDRA-7882 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7882 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Jay Patel >Assignee: Jay Patel > Labels: performance > Fix For: 2.1.1 > > Attachments: trunk-7882.txt > > > CASSANDRA-5935 allows option to disable region-based allocation for on-heap > memtables but there is no option to disable it for off-heap memtables > (memtable_allocation_type: offheap_objects). > Disabling region-based allocation will allow us to pack more tables in the > schema since minimum of 1MB region won't be allocated per table. Downside can > be more fragmentation which should be controllable by using better allocator > like JEMalloc. > How about below option in yaml?: > memtable_allocation_type: unslabbed_offheap_objects > Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7882) Memtable slab allocation should scale logarithmically to improve occupancy rate
[
https://issues.apache.org/jira/browse/CASSANDRA-7882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146132#comment-14146132
]
Jay Patel commented on CASSANDRA-7882:
--
Hey Benedict,
I've attached the first cut. Pls. help review.
Below are some code changes and design choices/trade-offs.
1) Wait-free region scaling and allocations:
Instead of one global queue of 1 MB race allocated regions, there’re are now
set of global queues, one for each region size (8K, 16K, ..1MB). All queues are
global (not per memtable) so memtables across all the tables can reuse the race
allotted regions. Race allocated regions will never be cleaned during memtable
flushes.
Thread who wins in the race of setting new region as the current region, also
scales the region size (if it’s not already at the max). This avoids any need
for extra synchronization for atomic region scaling.
2) Region size per memtable:
Moved region size per memtable instead of global. As per the code, each
memtable creates it’s own NativeAllocator object. So, I think keeping a region
size as a member variable of NativeAllocator makes the region size per
memtable. Pls. let me know if that is not the case & I’ll take care of it.
I don’t think below can be the issue, but want to share in case you see any
problems:
1)
In the race of allocating & setting the current region, in the extreme case
there is a slight chance of allocating next region with the same size (instead
of 2x). Consider the below case:
Thread1: allocates 16K region but has not yet reach to CAS for the current
region
Thread 2: allocates 16K, do CAS for the current region. Current region gets
filled up and set back to null, by allocate() method.
Thread 1: reaches the CAS instruction. Now, this will set current region to 16K
region, instead of 32K.
This sounds very corner case. Btw, even if this happens, there is absolutely
zero harm & next allocation will be directly of 64K to catch up since we never
miss scaling the region size.
To further guard against this, you'll see a below check in the code just before
CAS:
if(region.capacity == regionSize * SCALE_FACTOR || region.capacity ==
MAX_REGION_SIZE)
2)
Unslab allocation will happen only after we hit 1 MB region size. Until then,
huge payload can quickly grow the region size & allocate new region. I think
this behavior is good but slight side effect is that we may end up with few
partially filled (or not filled?) regions before we scale up to the correct
size given the payload. One option I though of is to have unslabbed allocation
threshold and count for each region size, in addition to 1MB region size. For
instance, with 8K region, anything beyond 4K will be unslabbed and if we see
1000 (threshold) of unslab allocations, we increase the region size. But, not
too excited about this since anyway flush may happen before that to reset the
region size. I don’t see much issue with this, but let me know..
below line will print how region allocation works if you want to test. I did
quick test with 1 to 100 static tables and payload size from 100 to 2KB for
now. In a week or two, planning to try out with 10s of thousands of tables
including longevity tests.
logger.info("{} size region allocated in {}", regionSize, this);
This takes care of only off-heap objects. For other slab allocator (on-heap?),
not sure if it makes sense to do region scaling.
TODO: Convert multiplication to shiffting for optimization. Change logger.info
to logger.trace or even remove it? Refactoring.
Let me know your thoughts, and I’ll finish TODO & attach a new patch.
> Memtable slab allocation should scale logarithmically to improve occupancy
> rate
> ---
>
> Key: CASSANDRA-7882
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7882
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Jay Patel
>Assignee: Jay Patel
> Labels: performance
> Fix For: 2.1.1
>
> Attachments: trunk-7882.txt
>
>
> CASSANDRA-5935 allows option to disable region-based allocation for on-heap
> memtables but there is no option to disable it for off-heap memtables
> (memtable_allocation_type: offheap_objects).
> Disabling region-based allocation will allow us to pack more tables in the
> schema since minimum of 1MB region won't be allocated per table. Downside can
> be more fragmentation which should be controllable by using better allocator
> like JEMalloc.
> How about below option in yaml?:
> memtable_allocation_type: unslabbed_offheap_objects
> Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7882) Memtable slab allocation should scale logarithmically to improve occupancy rate
[ https://issues.apache.org/jira/browse/CASSANDRA-7882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14140114#comment-14140114 ] Benedict commented on CASSANDRA-7882: - The region size should be per memtable. Once the memtable is discarded we never use it again, so there should be no scaling back of size. > Memtable slab allocation should scale logarithmically to improve occupancy > rate > --- > > Key: CASSANDRA-7882 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7882 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Jay Patel >Assignee: Jay Patel > Labels: performance > Fix For: 2.1.1 > > > CASSANDRA-5935 allows option to disable region-based allocation for on-heap > memtables but there is no option to disable it for off-heap memtables > (memtable_allocation_type: offheap_objects). > Disabling region-based allocation will allow us to pack more tables in the > schema since minimum of 1MB region won't be allocated per table. Downside can > be more fragmentation which should be controllable by using better allocator > like JEMalloc. > How about below option in yaml?: > memtable_allocation_type: unslabbed_offheap_objects > Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7882) Memtable slab allocation should scale logarithmically to improve occupancy rate
[ https://issues.apache.org/jira/browse/CASSANDRA-7882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14139606#comment-14139606 ] Jay Patel commented on CASSANDRA-7882: -- Hey Benedict, I've a first cut on this. Need to do some testing before attaching a patch. Should we scale back region size? If so, when? For now, I'm scaling back the region size logarithmically in the setDiscard() method when all regions are deallocated. > Memtable slab allocation should scale logarithmically to improve occupancy > rate > --- > > Key: CASSANDRA-7882 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7882 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Jay Patel >Assignee: Jay Patel > Labels: performance > Fix For: 2.1.1 > > > CASSANDRA-5935 allows option to disable region-based allocation for on-heap > memtables but there is no option to disable it for off-heap memtables > (memtable_allocation_type: offheap_objects). > Disabling region-based allocation will allow us to pack more tables in the > schema since minimum of 1MB region won't be allocated per table. Downside can > be more fragmentation which should be controllable by using better allocator > like JEMalloc. > How about below option in yaml?: > memtable_allocation_type: unslabbed_offheap_objects > Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7882) Memtable slab allocation should scale logarithmically to improve occupancy rate
[ https://issues.apache.org/jira/browse/CASSANDRA-7882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14139404#comment-14139404 ] Benedict commented on CASSANDRA-7882: - [~pateljay3001], any progress or would you like us to have a crack at it? > Memtable slab allocation should scale logarithmically to improve occupancy > rate > --- > > Key: CASSANDRA-7882 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7882 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Jay Patel >Assignee: Jay Patel > Labels: performance > Fix For: 2.1.1 > > > CASSANDRA-5935 allows option to disable region-based allocation for on-heap > memtables but there is no option to disable it for off-heap memtables > (memtable_allocation_type: offheap_objects). > Disabling region-based allocation will allow us to pack more tables in the > schema since minimum of 1MB region won't be allocated per table. Downside can > be more fragmentation which should be controllable by using better allocator > like JEMalloc. > How about below option in yaml?: > memtable_allocation_type: unslabbed_offheap_objects > Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7882) Memtable slab allocation should scale logarithmically to improve occupancy rate
[ https://issues.apache.org/jira/browse/CASSANDRA-7882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14122734#comment-14122734 ] Jay Patel commented on CASSANDRA-7882: -- Sure. Pls. assign it to me. I think better to have a on/off option initially, and then we can make it on by default if it is promising for all kind of use cases, not just high # of tables. > Memtable slab allocation should scale logarithmically to improve occupancy > rate > --- > > Key: CASSANDRA-7882 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7882 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Jay Patel >Assignee: Jay Patel > Labels: performance > Fix For: 2.1.1 > > > CASSANDRA-5935 allows option to disable region-based allocation for on-heap > memtables but there is no option to disable it for off-heap memtables > (memtable_allocation_type: offheap_objects). > Disabling region-based allocation will allow us to pack more tables in the > schema since minimum of 1MB region won't be allocated per table. Downside can > be more fragmentation which should be controllable by using better allocator > like JEMalloc. > How about below option in yaml?: > memtable_allocation_type: unslabbed_offheap_objects > Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
