[
https://issues.apache.org/jira/browse/COUCHDB-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085523#comment-13085523
]
Paul Joseph Davis commented on COUCHDB-1153:
--------------------------------------------
My thoughts on the loop were based on my day dreaming that it's entirely
possible that there's going to be feature requests to handle multiple
simultaneous compactions. I tend to have better luck reacting to messages to
maintain the state of a set of long running process directly from the
gen_server rather than have this middleman process looping around accepting
messages. Also, the more I look at this compact_loop the more things I see
wrong with it:
* You have a "Pid = spawn_link/1, MonRef = erlang:monitor(process, Pid)"
sequence for the parallel
view compactor. One of these is redundant. You want a link if you want
the compactor_loop to
exit when the view compactor crashes, or you want the monitor if you just
want to know when it dies.
* When you wait for the view compaction process to end there's no timeout.
That means that the
compactor loop could never move depending on whether the view compactor
process exits or not.
* You never flush monitor messages. This means the compact_loop process
mailbox
will slowly fill with messages over time causing hard to track memory
leaks.
* Views don't seem to be checked to see if they need to be compacted if
their database doesn't
need to be.
* View compaction holds open a reference to the database its compacting
views for. What happens if
views haven't finished compacting before the main database compaction
gets swapped out?
I'd prefer to either have os_mon in an app file or started as an app when the
VM boots. If we're going to talk about moving towards being more OTP compliant
we should be trying to avoid adding more non-OTP bits when possible.
The important part to trigger the couch_server issues you need to have a lot of
active databases as well as a lot of load so that try_close_lru turns into a
table scan of that ets table. Adam rewrote couch_server quite a long time ago
to replace this so that requests for open databases turned into a single ets
lookup on a public table which helped quite a bit. Though it introduces the
possibility of a race condition when opening a database that's just about to be
shut. Since then other things have been fixed and couch_server has become a
bottleneck again. I looked at it the other day and the only thing I came up
with would require some non-trivial changes to the close semantics of databases.
I think the general approach here is quite good and I'm quite fine with leaving
room for improvement. On the flip side, we need to avoid just pushing features
into trunk without considering how we might be asked to improve them or what
sort of maintenance cost they'll incur.
> Database and view index compaction daemon
> -----------------------------------------
>
> Key: COUCHDB-1153
> URL: https://issues.apache.org/jira/browse/COUCHDB-1153
> Project: CouchDB
> Issue Type: New Feature
> Environment: trunk
> Reporter: Filipe Manana
> Assignee: Filipe Manana
> Priority: Minor
> Labels: compaction
>
> I've recently written an Erlang process to automatically compact databases
> and they're views based on some configurable parameters. These parameters can
> be global or per database and are: minimum database fragmentation, minimum
> view fragmentation, allowed period and "strict_window" (whether an ongoing
> compaction should be canceled if it doesn't finish within the allowed
> period). These fragmentation values are based on the recently added
> "data_size" parameter to the database and view group information URIs
> (COUCHDB-1132).
> I've documented the .ini configuration, as a comment in default.ini, which I
> paste here:
> [compaction_daemon]
> ; The delay, in seconds, between each check for which database and view
> indexes
> ; need to be compacted.
> check_interval = 60
> ; If a database or view index file is smaller then this value (in bytes),
> ; compaction will not happen. Very small files always have a very high
> ; fragmentation therefore it's not worth to compact them.
> min_file_size = 131072
> [compactions]
> ; List of compaction rules for the compaction daemon.
> ; The daemon compacts databases and they're respective view groups when all
> the
> ; condition parameters are satisfied. Configuration can be per database or
> ; global, and it has the following format:
> ;
> ; database_name = parameter=value [, parameter=value]*
> ; _default = parameter=value [, parameter=value]*
> ;
> ; Possible parameters:
> ;
> ; * db_fragmentation - If the ratio (as an integer percentage), of the amount
> ; of old data (and its supporting metadata) over the
> database
> ; file size is equal to or greater then this value, this
> ; database compaction condition is satisfied.
> ; This value is computed as:
> ;
> ; (file_size - data_size) / file_size * 100
> ;
> ; The data_size and file_size values can be obtained when
> ; querying a database's information URI (GET /dbname/).
> ;
> ; * view_fragmentation - If the ratio (as an integer percentage), of the
> amount
> ; of old data (and its supporting metadata) over the
> view
> ; index (view group) file size is equal to or greater
> then
> ; this value, then this view index compaction
> condition is
> ; satisfied. This value is computed as:
> ;
> ; (file_size - data_size) / file_size * 100
> ;
> ; The data_size and file_size values can be obtained
> when
> ; querying a view group's information URI
> ; (GET /dbname/_design/groupname/_info).
> ;
> ; * period - The period for which a database (and its view groups) compaction
> ; is allowed. This value must obey the following format:
> ;
> ; HH:MM - HH:MM (HH in [0..23], MM in [0..59])
> ;
> ; * strict_window - If a compaction is still running after the end of the
> allowed
> ; period, it will be canceled if this parameter is set to
> "yes".
> ; It defaults to "no" and it's meaningful only if the
> *period*
> ; parameter is also specified.
> ;
> ; * parallel_view_compaction - If set to "yes", the database and its views are
> ; compacted in parallel. This is only useful on
> ; certain setups, like for example when the
> database
> ; and view index directories point to different
> ; disks. It defaults to "no".
> ;
> ; Before a compaction is triggered, an estimation of how much free disk space
> is
> ; needed is computed. This estimation corresponds to 2 times the data size of
> ; the database or view index. When there's not enough free disk space to
> compact
> ; a particular database or view index, a warning message is logged.
> ;
> ; Examples:
> ;
> ; 1) foo = db_fragmentation = 70%, view_fragmentation = 60%
> ; The `foo` database is compacted if its fragmentation is 70% or more.
> ; Any view index of this database is compacted only if its fragmentation
> ; is 60% or more.
> ;
> ; 2) foo = db_fragmentation = 70%, view_fragmentation = 60%, period =
> 00:00-04:00
> ; Similar to the preceding example but a compaction (database or view
> index)
> ; is only triggered if the current time is between midnight and 4 AM.
> ;
> ; 3) foo = db_fragmentation = 70%, view_fragmentation = 60%, period =
> 00:00-04:00, strict_window = yes
> ; Similar to the preceding example - a compaction (database or view index)
> ; is only triggered if the current time is between midnight and 4 AM. If at
> ; 4 AM the database or one of its views is still compacting, the compaction
> ; process will be canceled.
> ;
> ;_default = db_fragmentation = 70%, view_fragmentation = 60%, period = 23:00
> - 04:00
> (from https://github.com/fdmanana/couchdb/compare/compaction_daemon#L0R195)
> The full patch is mostly a new module but also does some minimal changes and
> a small refactoring to the view compaction code, not changing the current
> behaviour.
> Patch is at:
> https://github.com/fdmanana/couchdb/compare/compaction_daemon.patch
> By default the daemon is idle, without any configuration enabled. I'm open to
> suggestions on additional parameters and a better configuration system.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
