We're not on 1.4 yet, unfortunately. Are there any config params I can tweak to manipulate the compressor pool?
On Thu, Nov 29, 2012 at 1:49 PM, Keith Turner <[email protected]> wrote: > > > On Thu, Nov 29, 2012 at 12:20 PM, Anthony Fox <[email protected]>wrote: > >> Compacting down to a single file is not feasible - there's about 70G in >> 255 tablets across 15 tablet servers. Is there another way to tune the >> compressor pool or another mechanism to verify that this is the issue? > > > I suppose another way to test this would be to run a lot of concurrent > scans, but not enough to kill the tserver. Then get a heap dump of the > tserver and see if it contains a lot of 128k or 256k (can not remember > exact size) byte arrays that are referenced by the compressor pool. > > >> >> >> On Thu, Nov 29, 2012 at 12:09 PM, Keith Turner <[email protected]> wrote: >> >>> >>> >>> On Thu, Nov 29, 2012 at 11:14 AM, Anthony Fox <[email protected]>wrote: >>> >>>> I am experiencing some issues running multiple parallel scans against >>>> Accumulo. Running single scans works just fine but when I ramp up the >>>> number of simultaneous clients, my tablet servers die due to running out of >>>> heap space. I've tried raising max heap to 4G which should be more than >>>> enough but I still see this error. I've tried with >>>> table.cache.block.enable=false >>>> table.cache.index.enable=false, and table.scan.cache.enable=false and >>>> all combinations of caching enabled as well. >>>> >>>> My scans involve a custom intersecting iterator that maintains no more >>>> state than the top key and value. The scans also do a bit of aggregation >>>> on column qualifiers but the result is small and the number of returned >>>> entries is only in the dozens. The size of each returned value is only >>>> around 500 bytes. >>>> >>>> Any ideas why this may be happening or where to look for further info? >>>> >>> >>> One know issues is hadoops compressor pool. If you have a tablet with 8 >>> files and you query 10 terms, you will allocate 80 decompressors. Each >>> decompressor uses 128K. If you have 10 concurrent queries, 10 terms, and >>> 10 files then you will allocate 1000 decompressors. These decompressors >>> come from a pool that never shrinks. So if you allocate 1000 at the same >>> time, they will stay around. >>> >>> Try compacting your table down to one file and rerun your query just to >>> see if that helps. If it does, then thats an important clue. >>> >>> >>> >>>> >>>> Thanks, >>>> Anthony >>>> >>> >>> >> >
