[
https://issues.apache.org/jira/browse/ACCUMULO-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032412#comment-14032412
]
Eric Newton commented on ACCUMULO-2905:
---------------------------------------
The {{LargestFirstMemoryManager}} is only selecting one tablet for compaction,
which, after many tablets are loaded, does not reduce memory usage by enough to
provide space for additional ingest. The result is that writes have to wait
for the next MinC with short hold times. Several tablets should be selected to
reduce memory usage enough (10%?) to allow for writes.
> Seeing short hold times and no MinC's
> -------------------------------------
>
> Key: ACCUMULO-2905
> URL: https://issues.apache.org/jira/browse/ACCUMULO-2905
> Project: Accumulo
> Issue Type: Bug
> Components: tserver
> Environment: continuous ingest test
> Reporter: Eric Newton
> Assignee: Eric Newton
> Priority: Minor
> Attachments: SlowToMinC.png
>
>
> Started continuous ingest on a single node cluster to investigate oscillating
> ingest rates. Set the split size to 200M, and fired up 3 ingesters. I
> increase the major/minor compactors to 24/12 respectively.
> The native map was set to 4G.
> When the table got to 256 tablets, ingest was constantly showing hold times.
> MinC would run, but memory would constantly fill before new MinCs would
> start. Here's some output from the log:
> {noformat}
> $ tail -F ../../../logs/tserver_host.debug.log | egrep 'Up|Got|held|MinC \['
> 2014-06-13 14:51:56,553 [tablet.DatafileManager] TABLET_HIST: 2;5d104;5cf07
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00003kc/F0000qtk.rf
> 2014-06-13 14:51:56,695 [tablet.DatafileManager] TABLET_HIST: 2;11cb1;11ab
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000eww/F0000qtl.rf
> 2014-06-13 14:51:56,861 [tablet.DatafileManager] TABLET_HIST: 2;0e4a3;0e2a3
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000ery/F0000qtm.rf
> 2014-06-13 14:51:56,927 [tablet.DatafileManager] TABLET_HIST: 2;44907;44504
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008q9/F0000qtn.rf
> 2014-06-13 14:51:57,019 [tablet.DatafileManager] TABLET_HIST: 2;4a0fa;49cfd
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00001lg/F0000qto.rf
> 2014-06-13 14:51:57,535 [tablet.DatafileManager] TABLET_HIST: 2;4c0ff;4bd03
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00000ut/F0000qtq.rf
> 2014-06-13 14:51:58,443 [tablet.DatafileManager] TABLET_HIST: 2;770cf;76ed2f
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00003lv/F0000qts.rf
> 2014-06-13 14:51:58,448 [tserver.TabletServerResourceManager] DEBUG: Commits
> held for 2.18 secs
> 2014-06-13 14:51:58,449 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:33363
> 225 in 1.669s, at=[0 0 0.00 1] ft=1.668s(pt=1.667s lt=0.001s ct=0.000s)
> 2014-06-13 14:51:58,790 [tablet.DatafileManager] TABLET_HIST: 2;4ad04b;4a902
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000f05/F0000qtt.rf
> 2014-06-13 14:51:59,113 [tablet.DatafileManager] TABLET_HIST: 2;23af3;238f4
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000ei6/F0000qtw.rf
> 2014-06-13 14:52:00,545 [tablet.DatafileManager] TABLET_HIST: 2;4d906e;4d508
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008s0/F0000qtx.rf
> 2014-06-13 14:52:00,599 [tablet.DatafileManager] TABLET_HIST: 2;1bcb4f;1b8b6
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000f9v/F0000qty.rf
> 2014-06-13 14:52:00,640 [tablet.DatafileManager] TABLET_HIST: 2;1d8be;1d4c1
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008j7/F0000qtz.rf
> 2014-06-13 14:52:00,648 [tablet.DatafileManager] TABLET_HIST: 2;40ce3;408de
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000etz/F0000qu0.rf
> 2014-06-13 14:52:01,040 [tablet.DatafileManager] TABLET_HIST: 2;2c13;2bf32
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00000uc/F0000qu5.rf
> 2014-06-13 14:52:01,939 [tablet.DatafileManager] TABLET_HIST: 2;5fedd;5fce3
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000fir/F0000qu7.rf
> 2014-06-13 14:52:02,063 [tablet.DatafileManager] TABLET_HIST: 2;26d0a;26907
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000eum/F0000qu8.rf
> 2014-06-13 14:52:02,067 [tserver.TabletServerResourceManager] DEBUG: Commits
> held for 3.19 secs
> 2014-06-13 14:52:02,069 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:33363
> 288 in 2.617s, at=[0 0 0.00 1] ft=2.616s(pt=2.614s lt=0.001s ct=0.001s)
> 2014-06-13 14:52:02,604 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:58940
> 279,538 in 9.656s, at=[0 1 0.00 899] ft=9.274s(pt=7.516s lt=1.099s ct=0.659s)
> 2014-06-13 14:52:02,700 [tablet.DatafileManager] TABLET_HIST: 2;758e6;756e8
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008py/F0000qub.rf
> 2014-06-13 14:52:02,765 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:58937
> 279,538 in 6.735s, at=[0 1 0.00 899] ft=6.446s(pt=4.446s lt=1.359s ct=0.641s)
> 2014-06-13 14:52:02,821 [tablet.DatafileManager] TABLET_HIST: 2;53b18;53918
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000kk7/F0000quc.rf
> 2014-06-13 14:52:03,503 [tablet.DatafileManager] TABLET_HIST: 2;116af;114af
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000jnw/F0000qud.rf
> 2014-06-13 14:52:03,554 [tablet.DatafileManager] TABLET_HIST: 2;76ed2f;76cd7
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000ju3/F0000que.rf
> 2014-06-13 14:52:04,197 [tablet.DatafileManager] TABLET_HIST: 2;7da4e;7d851
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000igh/F0000quf.rf
> 2014-06-13 14:52:04,603 [tablet.DatafileManager] TABLET_HIST: 2;38d03;388fe
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000evr/F0000quh.rf
> 2014-06-13 14:52:05,119 [tablet.DatafileManager] TABLET_HIST: 2;156d1f;154d1e
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000jqx/F0000quj.rf
> 2014-06-13 14:52:05,577 [tablet.DatafileManager] TABLET_HIST: 2;3fada;3f8db
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000pre/F0000qul.rf
> 2014-06-13 14:52:05,582 [tserver.TabletServerResourceManager] DEBUG: Commits
> held for 3.29 secs
> 2014-06-13 14:52:05,583 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:33363
> 252 in 2.512s, at=[0 0 0.00 1] ft=2.511s(pt=2.510s lt=0.001s ct=0.000s)
> 2014-06-13 14:52:06,206 [tablet.DatafileManager] TABLET_HIST: 2;4550e;4510b
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000ex0/F0000qum.rf
> 2014-06-13 14:52:06,360 [tablet.DatafileManager] TABLET_HIST: 2;7fe06;7fc0b
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000hyp/F0000qun.rf
> 2014-06-13 14:52:06,642 [tablet.DatafileManager] TABLET_HIST: 2;238f4;236f4
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008i1/F0000quq.rf
> 2014-06-13 14:52:08,626 [tablet.DatafileManager] TABLET_HIST: 2;5931a;59119
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000fy8/F0000qus.rf
> 2014-06-13 14:52:08,682 [tablet.DatafileManager] TABLET_HIST: 2;4a902;4a5
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008kd/F0000quu.rf
> 2014-06-13 14:52:08,740 [tablet.DatafileManager] TABLET_HIST: 2;1dcbe;1d8be
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000etq/F0000quv.rf
> 2014-06-13 14:52:08,748 [tablet.DatafileManager] TABLET_HIST: 2;23ef2;23cf2
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000ei9/F0000qut.rf
> 2014-06-13 14:52:09,215 [tablet.DatafileManager] TABLET_HIST: 2;0fa97;0f899
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000fgk/F0000quy.rf
> 2014-06-13 14:52:09,623 [tablet.DatafileManager] TABLET_HIST: 2;0101f;00e1df
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00003ka/F0000qv0.rf
> 2014-06-13 14:52:09,626 [tserver.TabletServerResourceManager] DEBUG: Commits
> held for 3.81 secs
> 2014-06-13 14:52:09,628 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:33363
> 250 in 3.043s, at=[0 0 0.00 1] ft=3.042s(pt=3.041s lt=0.000s ct=0.001s)
> 2014-06-13 14:52:10,256 [tablet.DatafileManager] TABLET_HIST: 2;29d21;2991f
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000f6f/F0000qv2.rf
> 2014-06-13 14:52:10,388 [tablet.DatafileManager] TABLET_HIST: 2;6f6f3;6f4f4
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000pr5/F0000qv4.rf
> 2014-06-13 14:52:10,879 [tablet.DatafileManager] TABLET_HIST: 2;16cc5;168c8
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000fb5/F0000qv5.rf
> 2014-06-13 14:52:11,379 [tablet.DatafileManager] TABLET_HIST: 2;4b104;4ad04b
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00003lg/F0000qv7.rf
> 2014-06-13 14:52:11,879 [tablet.DatafileManager] TABLET_HIST: 2;6f8f;6f6f3
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008mo/F0000qv9.rf
> 2014-06-13 14:52:12,395 [tablet.DatafileManager] TABLET_HIST: 2;3f4e1;3f2e3e
> MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000fb1/F0000qva.rf
> {noformat}
> Perhaps the server should be more aggressive with MinC's to get better ingest
> performance?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
