[
https://issues.apache.org/jira/browse/ACCUMULO-1052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580772#comment-13580772
]
Adam Fuchs commented on ACCUMULO-1052:
--------------------------------------
There are really only two reasons we have merging minor compactions -- the
first is that HDFS craps out when there are too many open files (like on a read
of a tablet with lots and lots of files). The second is that reading from a
tablet with too many files can be slow. Merging minor compactions have two
effects. The first is to cap the number of files to prevent the aforementioned
crapping out. The second is to throttle long-term ingest so that it's tied to
disk I/O capacity. The problem with using merging minor compactions as a
throttling mechanism is that it actually generates more disk I/O load by making
extra copies of entries (doing N^2 work instead of NlogN in the worst case). We
could probably use a better mechanism for throttling. Meanwhile, an alternative
solution might be to add more major compaction threads and reduce the number of
minor compaction threads.
When major compactions are keeping up, the smallest file should be really
small. With a 100GB tablet and a major compaction ratio of 3.0, the 15th
largest file should be no more than 257MB and the 20th largest file should be
no bigger than about 34MB. With a max files limit of 20, minor compactions
should take on the order of 5 seconds in that scenario. This might be a reason
to bump up the max files threshold from the default of 15. Here's a chart that
shows a few file size stats, with major compaction ratio 3, assuming major
compactions are keeping up:
||#Files||Min(Max File Size/Min File Size)||Min(Tabet Size/Min File Size)||
|1|1|1|
|2|1|2|
|3|1|3|
|4|1.5|4.5|
|5|2.25|6.75|
|6|3.375|10.125|
|7|5.0625|15.1875|
|8|7.59375|22.78125|
|9|11.390625|34.171875|
|10|17.0859375|51.2578125|
|11|25.62890625|76.88671875|
|12|38.44335938|115.3300781|
|13|57.66503906|172.9951172|
|14|86.49755859|259.4926758|
|15|129.7463379|389.2390137|
|16|194.6195068|583.8585205|
|17|291.9292603|875.7877808|
|18|437.8938904|1313.681671|
|19|656.8408356|1970.522507|
|20|985.2612534|2955.78376|
|21|1477.89188|4433.67564|
|22|2216.83782|6650.51346|
|23|3325.25673|9975.77019|
|24|4987.885095|14963.65529|
|25|7481.827643|22445.48293|
|26|11222.74146|33668.22439|
|27|16834.1122|50502.33659|
|28|25251.16829|75753.50488|
|29|37876.75244|113630.2573|
|30|56815.12866|170445.386|
> Minor compactions not finishing before master kills tabletserver can very
> large number of files per tablet
> ----------------------------------------------------------------------------------------------------------
>
> Key: ACCUMULO-1052
> URL: https://issues.apache.org/jira/browse/ACCUMULO-1052
> Project: Accumulo
> Issue Type: Improvement
> Components: master, tserver
> Affects Versions: 1.4.2
> Environment: Large, write-heavy cluster
> Reporter: Josh Elser
> Assignee: Eric Newton
>
> On a cluster that is being saturated with heavy ingest, a tserver is observed
> attempting to perform a minor compaction for a tablet with multiple WALs.
> Because of this, commits to this tablet end up being held.
> After churning on the minc for some time, the master's hold-time limit for
> tservers is exceeded, however the minc didn't finish. The tserver is forcibly
> killed, the tablet is migrated, recovery occurs on the new tserver and the
> problem repeats.
> Some of the minor compactions must finish, as the number of files for that
> tablet continue to grow, but major compactions must not have time to finish
> since the number of files grow unbounded.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
