RE: Efficient Tablet Merging [SEC=UNOFFICIAL]

2013-10-03 Thread Dickson, Matt MR
UNOFFICIAL Hi Eric, We have gone with the second more conservative option. We changed our split threshold to 10GB and then we ran a merge over a week worth of tablets which has resulted in one tablet with a massive number of files. We then ran a query over that range and it is returning an

Re: Efficient Tablet Merging [SEC=UNOFFICIAL]

2013-10-03 Thread Adam Fuchs
Never underestimate the power of ascii art! Adam On Oct 2, 2013 11:28 PM, Eric Newton eric.new...@gmail.com wrote: I'll use ASCII graphics to demonstrate the size of a tablet. Small: [] Medium: [ ] Large: [ ] Think of it like this... if you are running age-off... you probably have lots

Re: Efficient Tablet Merging [SEC=UNOFFICIAL]

2013-10-03 Thread Eric Newton
You should have a major compaction running if your tablet has too many files. If you don't, something is wrong. It does take some time to re-write 10G of data. If many merges occurred on a single tablet server, you may have these many-file tablets on the same server, and there are not enough

RE: Efficient Tablet Merging [SEC=UNOFFICIAL]

2013-10-03 Thread Dickson, Matt MR
UNOFFICIAL We have restarted the tablet servers that contain tablets with high volumes of files and did not see any majc's run. Some more details are: On 3 of our nodes we have 10-15 times the number of entries that are on the other nodes. When I view the tablets for one of these nodes there

Re: Efficient Tablet Merging [SEC=UNOFFICIAL]

2013-10-03 Thread Eric Newton
Great details... but I need to sleep. I'll dig in more tomorrow. Sorry! On Thu, Oct 3, 2013 at 11:20 PM, Dickson, Matt MR matt.dick...@defence.gov.au wrote: UNOFFICIAL Hi Eric, Our answers are in blue. Just a note that we do have the write ahead log disabled for ingest performance. We