keith-turner opened a new issue #2199:
URL: https://github.com/apache/accumulo/issues/2199
Currently in the 2.1.0 branch when a user compaction is initiated each
tablet selects a set of files for the user compaction. Then 1 or more
compactions can be scheduled to compaction those files. Any new files that
arrive for the tablet are not added to the set of selected files. The set of
selected files is reserved for user compactions and will not be available to
routine system compactions.
For the case where files are reserved for a long time because user
compactions are queued this can prevent system compactions from lowering a
tablets number of files as new files arrive. For example assume a tablet has
files of size [(F1,100M),(F2,10M)] reserved for a user compaction. Then
assume that two new files arrive [(F3,10M),(F4,10M)]. With a compaction ratio
of 3 when the files [F2,F3,F4] could be compacted by a system compaction.
However, file F2 being reserved for a user compaction will prevent this.
Selecting and reserving a set of files must happen for user compactions so
they can complete. However reserving could be delayed until the time when the
first user compaction runs. This could be accomplished by having the following
per tablet states related to user compactions.
* *NONE* - When there is no user compaction happening or one has completed
for this tabler.
* *SELECTING* - When a tablet is in the process of examining files to
determine which ones to include in a user compaction.
* *SELECTED* - When a tablet has selected a set of files to compact for a
user compaction, but no compactions have occurred.
* *RESERVED* - When a tablet has selected a set of files to compact for a
user compaction and one or more user compactions has started.
The NONE, SELECTING, and SELECTED states exist in the current code. In the
current code SELECTED implies the files are also reserved. With the new
RESERVED state it allows the following behavior.
* A system compaction can run against files in the selected state. If it
does then two things should happen. First the tablet should move to the
SELECTING state. Second any queued user compactions should be canceled (this
can be done lazily when they try to run by checking if the selected set changed
while queued).
* Once a tablet has moved to the reserved state, then no system compactions
should be able to start against the reserved set of files. The first user
compaction job that starts running will move the tablet from SELECTED to
RESERVED, if the selected set had not changed since the compaction job was
created.
These state transitions can be made/checked atomically before a compactions
starts, possibly ignoring a compaction job if things changed since it was
queued.
One drawback of this change is that it could lead to starvation of user
compactions. If new files are continually arriving and causing system
compactions, those could possibly lead to a user compaction never running. The
current way files are reserved ASAP does not have this problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
