[
https://issues.apache.org/jira/browse/ACCUMULO-4187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240017#comment-15240017
]
ASF GitHub Bot commented on ACCUMULO-4187:
------------------------------------------
Github user keith-turner commented on the pull request:
https://github.com/apache/accumulo/pull/90#issuecomment-209647745
I played around with this branch locally. I created a table with
10,000,000 entries using test_ingest using the following commands.
```
./bin/accumulo shell -u root -p secret -e "createtable test_ingest"
./bin/accumulo org.apache.accumulo.test.TestIngest -u root -p secret
--timestamp 1 --size 50 --random 56 --rows 10000000 --start 0 --cols 1
--instance instance16
```
I set the rate limit to 5M and forced a compaction. I saw the following in
the tserver logs.
```
Compaction 2<< 10,000,000 read | 10,000,000 written | 122,925 entries/sec |
81.350 secs | 431,758,096 bytes | 5307413.596 byte/sec
```
Then I split the table into 8 tablets and forced a compaction to test the
rate limit for multiple threads. I had the default of 3 compaction threads. I
saw the following in the logs for this test.
```
Compaction 2;row_0003749;row_00025 1,249,000 read | 1,249,000 written |
41,866 entries/sec | 29.833 secs | 53,926,291 bytes | 1807605.370 byte/sec
Compaction 2;row_00025;row_000125 1,250,000 read | 1,250,000 written |
41,899 entries/sec | 29.833 secs | 53,970,229 bytes | 1809078.168 byte/sec
Compaction 2;row_000125< 1,250,000 read | 1,250,000 written | 41,783
entries/sec | 29.916 secs | 53,969,343 bytes | 1804029.382 byte/sec
Compaction 2;row_000625;row_0005 1,250,000 read | 1,250,000 written |
42,134 entries/sec | 29.667 secs | 53,970,847 bytes | 1819221.593 byte/sec
Compaction 2;row_0005;row_0003749 1,251,000 read | 1,251,000 written |
42,109 entries/sec | 29.708 secs | 54,012,874 bytes | 1818125.555 byte/sec
Compaction 2;row_00075;row_000625 1,250,000 read | 1,250,000 written |
41,881 entries/sec | 29.846 secs | 53,969,549 bytes | 1808267.406 byte/sec
Compaction 2;row_000875;row_00075 1,250,000 read | 1,250,000 written |
63,909 entries/sec | 19.559 secs | 53,969,511 bytes | 2759318.523 byte/sec
Compaction 2<;row_000875 1,250,000 read | 1,250,000 written | 63,798
entries/sec | 19.593 secs | 53,969,987 bytes | 2754554.535 byte/sec
```
> Rate limiting of major compactions
> ----------------------------------
>
> Key: ACCUMULO-4187
> URL: https://issues.apache.org/jira/browse/ACCUMULO-4187
> Project: Accumulo
> Issue Type: Improvement
> Components: core, tserver
> Reporter: Shawn Walker
> Assignee: Shawn Walker
> Priority: Minor
> Fix For: 1.8.0
>
>
> In discussing [ACCUMULO-4166] with Keith Turner, we decided that the
> underlying issue is that major compactions can overwhelm a tablet server,
> rendering it nearly unresponsive.
> To address this, we should take a cue from Apache Cassandra and restrict how
> quickly we perform major compactions. Rate limiting reads and writes
> involved in major compactions will directly affect the IO load caused by
> major compactions, and should also indirectly affect the CPU load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
