[
https://issues.apache.org/jira/browse/CASSANDRA-10496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14995840#comment-14995840
]
Gabriel Wicke commented on CASSANDRA-10496:
-------------------------------------------
A related idea I have been thinking about recently is using the *median* of
timestamps for the time window assignment. By making the assignment robust
against outliers, this would make it safe to use DTCS with read repair enabled.
At the same time, it would preserve the desirable behavior of grouping old /
large SSTables together when switching from other compaction strategies.
Implementing this would require additionally calculating an approximation of
the median for each SSTable. There are various approximation algorithms that
could be used, for example
http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/Remedian.pdf.
> Make DTCS split partitions based on time during compaction
> ----------------------------------------------------------
>
> Key: CASSANDRA-10496
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10496
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Marcus Eriksson
> Labels: dtcs
> Fix For: 3.x
>
>
> To avoid getting old data in new time windows with DTCS (or related, like
> [TWCS|CASSANDRA-9666]), we need to split out old data into its own sstable
> during compaction.
> My initial idea is to just create two sstables, when we create the compaction
> task we state the start and end times for the window, and any data older than
> the window will be put in its own sstable.
> By creating a single sstable with old data, we will incrementally get the
> windows correct - say we have an sstable with these timestamps:
> {{[100, 99, 98, 97, 75, 50, 10]}}
> and we are compacting in window {{[100, 80]}} - we would create two sstables:
> {{[100, 99, 98, 97]}}, {{[75, 50, 10]}}, and the first window is now
> 'correct'. The next compaction would compact in window {{[80, 60]}} and
> create sstables {{[75]}}, {{[50, 10]}} etc.
> We will probably also want to base the windows on the newest data in the
> sstables so that we actually have older data than the window.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
