[
https://issues.apache.org/jira/browse/MESOS-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997998#comment-13997998
]
Kevin Sweeney commented on MESOS-184:
-------------------------------------
Any update on this?
> Log has a space leak
> --------------------
>
> Key: MESOS-184
> URL: https://issues.apache.org/jira/browse/MESOS-184
> Project: Mesos
> Issue Type: Bug
> Components: c++ api, replicated log
> Affects Versions: 0.9.0
> Reporter: John Sirois
> Assignee: Jie Yu
> Priority: Minor
> Labels: twitter
>
> In short, the access pattern of the Log of the underlying LevelDB storage is
> such that background compactions are ineffective and a long running Log will
> have a space leak on disk even in the presence of otherwise apparently
> sufficient Log::Writer::truncate calls.
> It seems the right thing to do is to issue a DB::CompactRange(NULL,
> Slice(truncateToKey)) after a replica learns a Action::TRUNCATE Record. The
> cost here is a synchronous compaction stall on every truncate so maybe this
> should be a configuration option or even an explicit api.
> ===
> Snip of email explanation:
> I spent some time understanding what was going on here and our use pattern of
> leveldb does in fact defeat the backround compaction algorithm.
> The docs are here: http://leveldb.googlecode.com/svn/trunk/doc/impl.html in
> the 'Compactions' section, but in short the gist is compaction operates on an
> uncompacted file from a level (1 file) + all files overlapping its key range
> in the next level. Since we write sequential keys with no randomness at all,
> by definition the only overlap we ever can get is in level 0 which is the
> only level that leveldb allows for overlap in sstables in the 1st place.
> That leaves the question of why no compaction on open. Looking there:
> http://code.google.com/p/leveldb/source/browse/db/db_impl.cc#1376
> I see a call to MaybeScheduleCompaction, but following that trail, that just
> leads to
> http://code.google.com/p/leveldb/source/browse/db/version_set.cc?spec=svnbc1ee4d25e09b04e074db330a41f54ef4af0e31b&r=36a5f8ed7f9fb3373236d5eace4f5fea369856ee#1156
> which implements the compaction strategy I tried to summarize above, and
> thus background compactions for out case are limited to level0 -> level 1
> compactions and lefel1 and higher never compact automatically.
> This seems born out by the LOG files. For example, from smf1-prod - restarts
> after your manual compaction fix in bold:
> [jsirois@smf1-ajb-35-sr1 ~]$ grep Compacting
> /var/lib/mesos/scheduler_db/mesos_log/LOG.old
> 2012/04/13-00:24:20.356673 44c1e940 Compacting 3@0 + 4@1 files
> 2012/04/13-00:24:20.490113 44c1e940 Compacting 5@1 + 281@2 files
> 2012/04/13-00:24:25.824995 44c1e940 Compacting 1@1 + 0@2 files
> 2012/04/13-00:24:26.008857 44c1e940 Compacting 1@2 + 0@3 files
> 2012/04/13-00:24:26.196877 44c1e940 Compacting 1@2 + 0@3 files
> 2012/04/13-00:24:26.312465 44c1e940 Compacting 1@2 + 0@3 files
> 2012/04/13-00:24:26.429817 44c1e940 Compacting 1@2 + 0@3 files
> 2012/04/13-00:24:26.533483 44c1e940 Compacting 1@2 + 0@3 files
> 2012/04/13-00:24:26.631044 44c1e940 Compacting 1@2 + 0@3 files
> 2012/04/13-00:24:26.733702 44c1e940 Compacting 1@2 + 0@3 files
> 2012/04/13-00:24:26.832787 44c1e940 Compacting 1@2 + 0@3 files
> 2012/04/13-00:24:26.949864 44c1e940 Compacting 1@2 + 0@3 files
> 2012/04/13-00:24:27.052502 44c1e940 Compacting 1@2 + 0@3 files
> 2012/04/13-00:24:27.164623 44c1e940 Compacting 1@2 + 0@3 files
> 2012/04/13-00:24:27.275621 44c1e940 Compacting 1@2 + 0@3 files
> 2012/04/13-00:24:27.376748 44c1e940 Compacting 1@2 + 0@3 files
> 2012/04/13-00:24:27.477728 44c1e940 Compacting 1@2 + 0@3 files
> 2012/04/13-00:24:27.611332 44c1e940 Compacting 1@2 + 0@3 files
> 2012/04/13-00:24:28.050275 44c1e940 Compacting 50@2 + 242@3 files
> 2012/04/13-00:24:32.455665 44c1e940 Compacting 1@2 + 0@3 files
> 2012/04/13-00:24:32.538566 44c1e940 Compacting 1@3 + 0@4 files
> 2012/04/13-00:24:32.819205 44c1e940 Compacting 1@3 + 0@4 files
> 2012/04/13-00:24:33.052064 44c1e940 Compacting 1@3 + 0@4 files
> 2012/04/13-00:24:33.198850 44c1e940 Compacting 1@3 + 0@4 files
> 2012/04/13-00:24:33.350893 44c1e940 Compacting 1@3 + 0@4 files
> 2012/04/13-00:24:33.521784 44c1e940 Compacting 1@3 + 0@4 files
> 2012/04/13-00:24:33.693531 44c1e940 Compacting 1@3 + 0@4 files
> 2012/04/13-00:24:33.847151 44c1e940 Compacting 1@3 + 0@4 files
> 2012/04/13-00:24:34.034277 44c1e940 Compacting 1@3 + 0@4 files
> 2012/04/13-00:24:34.225582 44c1e940 Compacting 1@3 + 0@4 files
> 2012/04/13-00:24:34.390228 44c1e940 Compacting 1@3 + 0@4 files
> 2012/04/13-00:24:34.554127 44c1e940 Compacting 1@3 + 0@4 files
> 2012/04/13-00:24:34.715242 44c1e940 Compacting 1@3 + 0@4 files
> 2012/04/13-00:24:34.852110 44c1e940 Compacting 1@3 + 0@4 files
> 2012/04/13-00:24:35.046899 44c1e940 Compacting 68@3 + 331@4 files
> 2012/04/13-00:25:02.582758 44c1e940 Compacting 433@3 + 2159@4 files
> 2012/04/13-00:26:39.827096 44c1e940 Compacting 1@3 + 0@4 files
> 2012/04/13-00:26:39.992623 44c1e940 Compacting 72@4 + 354@5 files
> 2012/04/13-00:27:13.024120 44c1e940 Compacting 9@4 + 51@5 files
> 2012/04/13-00:27:18.007566 44c1e940 Compacting 9@4 + 48@5 files
> 2012/04/13-00:27:23.026351 44c1e940 Compacting 8@4 + 41@5 files
> 2012/04/13-00:27:28.408619 44c1e940 Compacting 6@4 + 33@5 files
> 2012/04/13-00:27:32.522630 44c1e940 Compacting 6@4 + 32@5 files
> 2012/04/13-00:27:36.719610 44c1e940 Compacting 6@4 + 31@5 files
> 2012/04/13-00:27:41.277302 44c1e940 Compacting 6@4 + 33@5 files
> 2012/04/13-00:27:44.928451 44c1e940 Compacting 6@4 + 32@5 files
> 2012/04/13-00:27:48.168874 44c1e940 Compacting 6@4 + 34@5 files
> 2012/04/13-00:27:52.718402 44c1e940 Compacting 6@4 + 32@5 files
> 2012/04/13-00:27:55.665107 44c1e940 Compacting 6@4 + 33@5 files
> 2012/04/13-00:27:59.381808 44c1e940 Compacting 6@4 + 34@5 files
> 2012/04/13-00:28:03.592802 44c1e940 Compacting 6@4 + 33@5 files
> 2012/04/13-00:28:07.179032 44c1e940 Compacting 664@4 + 3330@5 files
> 2012/04/13-00:29:58.239662 44c1e940 Compacting 101@4 + 500@5 files
> 2012/04/13-00:30:22.333750 44c1e940 Compacting 1@4 + 0@5 files
> 2012/04/13-00:45:28.851715 44c1e940 Compacting 4@0 + 1@1 files
> 2012/04/13-01:00:31.152105 44c1e940 Compacting 4@0 + 3@1 files
> 2012/04/13-01:10:33.167940 44c1e940 Compacting 4@0 + 3@1 files
> 2012/04/13-01:25:35.113416 44c1e940 Compacting 4@0 + 3@1 files
> 2012/04/13-01:35:37.621499 44c1e940 Compacting 4@0 + 2@1 files
> [jsirois@smf1-ajb-35-sr1 ~]$ grep Compacting
> /var/lib/mesos/scheduler_db/mesos_log/LOG
> 2012/04/13-01:44:32.533694 44c27940 Compacting 2@0 + 3@1 files
> 2012/04/13-01:44:32.586958 44c27940 Compacting 2@1 + 6@2 files
> 2012/04/13-01:44:32.739514 44c27940 Compacting 1@2 + 0@3 files
> 2012/04/13-01:44:32.768764 44c27940 Compacting 1@2 + 0@3 files
> 2012/04/13-01:44:32.843866 44c27940 Compacting 1@3 + 0@4 files
> 2012/04/13-01:44:32.973304 44c27940 Compacting 1@3 + 0@4 files
> 2012/04/13-01:44:33.009686 44c27940 Compacting 1@4 + 2@5 files
> 2012/04/13-01:44:33.074056 44c27940 Compacting 1@4 + 0@5 files
> 2012/04/13-02:01:42.947456 44c27940 Compacting 4@0 + 1@1 files
> 2012/04/13-02:16:45.326088 44c27940 Compacting 4@0 + 4@1 files
> 2012/04/13-02:31:48.360851 44c27940 Compacting 4@0 + 1@1 files
> 2012/04/13-02:41:50.055622 44c27940 Compacting 4@0 + 3@1 files
> 2012/04/13-02:51:51.889148 44c27940 Compacting 4@0 + 4@1 files
> 2012/04/13-03:01:54.345784 44c27940 Compacting 4@0 + 4@1 files
> 2012/04/13-03:11:55.987774 44c27940 Compacting 4@0 + 4@1 files
> 2012/04/13-03:21:57.701121 44c27940 Compacting 4@0 + 4@1 files
> 2012/04/13-03:31:59.373435 44c27940 Compacting 4@0 + 4@1 files
> 2012/04/13-03:42:01.047061 44c27940 Compacting 4@0 + 4@1 files
> 2012/04/13-03:52:03.088683 44c27940 Compacting 4@0 + 4@1 files
> 2012/04/13-04:02:05.181165 44c27940 Compacting 4@0 + 4@1 files
> 2012/04/13-04:12:06.757773 44c27940 Compacting 4@0 + 4@1 files
> 2012/04/13-04:22:08.598259 44c27940 Compacting 4@0 + 4@1 files
> 2012/04/13-04:32:10.882913 44c27940 Compacting 4@0 + 4@1 files
> 2012/04/13-04:42:12.602192 44c27940 Compacting 4@0 + 4@1 files
> 2012/04/13-04:52:14.779705 44c27940 Compacting 4@0 + 4@1 files
> 2012/04/13-05:02:16.621063 44c27940 Compacting 4@0 + 4@1 files
> 2012/04/13-05:12:18.608767 44c27940 Compacting 4@0 + 4@1 files
> 2012/04/13-05:22:20.453201 44c27940 Compacting 4@0 + 4@1 files
> 2012/04/13-05:32:22.215804 44c27940 Compacting 4@0 + 4@1 files
> 2012/04/13-05:42:23.882423 44c27940 Compacting 4@0 + 4@1 files
> 2012/04/13-05:52:25.553032 44c27940 Compacting 4@0 + 4@1 files
> With the format: [number of sstables compacted]@[level #] this says all
> levels are compacted on startup now, but once running we only see level0 ->
> level1 compactions and this accounts for the observed space leak.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
