Yes, I tried manually trigger major compact on the problematic region, and it is the same that the region did not get any deleted rows removed, therefore the size is kept at 90+G. And I verified that if do a split, then the major compaction will work as expected, i.e. into two 10+G regions.
Thanks Tian-Ying On Fri, May 27, 2016 at 2:54 PM, Frank Luo <[email protected]> wrote: > What if you manually trigger major-compact on that particular region? Does > it run and the delete markers removed? > > -----Original Message----- > From: Tianying Chang [mailto:[email protected]] > Sent: Friday, May 27, 2016 4:33 PM > To: [email protected] > Subject: Major compaction cannot remove deleted rows until the region is > split. Strange! > > Hi, > > We saw a very strange case in one of our production cluster. A couple > regions cannot get their deleted rows or delete marker removed even after > major compaction. However when the region triggered split (we set 100G for > auto split), the deletion worked. The 100G region becomes two 10G daughter > regions, and all the delete marker are gone. > > Also, the same region in the slave cluster (through replication) have > normal size at about 20+G. > > BTW, the delete marker in the regions are mostly deleteFamily if it > matters. > > This is really weird. Anyone has any clue for this strange behavior? > > Thanks > Tian-Ying > > A snippet of the HFile generated by the major compaction: > > : \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@ > \x10\x00?PF/d:/1459808114380/DeleteFamily/vlen=0/ts=2292870047 > V: > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@ > \x10\x00?PF/d:/1459808114011/DeleteFamily/vlen=0/ts=2292869794 > V: > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@ > \x10\x00?PF/d:/1459805381104/DeleteFamily/vlen=0/ts=2291072240 > V: > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@ > \x10\x00?PF/d:/1459805380673/DeleteFamily/vlen=0/ts=2291071997 > V: > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@ > \x10\x00?PF/d:/1459802643449/DeleteFamily/vlen=0/ts=2290248886 > V: > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@ > \x10\x00?PF/d:/1459802643246/DeleteFamily/vlen=0/ts=2290248786 > V: > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@ > \x10\x00?PF/d:/1459799913003/DeleteFamily/vlen=0/ts=2289446916 > V: > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@ > \x10\x00?PF/d:/1459797181831/DeleteFamily/vlen=0/ts=2288670451 > V: > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@ > \x10\x00?PF/d:/1459794447388/DeleteFamily/vlen=0/ts=2287911443 > V: > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@ > \x10\x00?PF/d:/1459791708803/DeleteFamily/vlen=0/ts=2287213792 > V: > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@ > \x10\x00?PF/d:/1459788978387/DeleteFamily/vlen=0/ts=2286488738 > V: > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@ > \x10\x00?PF/d:/1459786243642/DeleteFamily/vlen=0/ts=2285778927 > V: > Join us at Merkle’s 2016 annual Performance Marketer Executive Summit – > June 7 – 9 in Memphis, TN > <http://www2.merkleinc.com/l/47252/2016-04-26/3lbfdc> > > Download the latest installment of our annual Marketing Imperatives, > “Winning with People-Based Marketing”< > http://www2.merkleinc.com/l/47252/2016-04-26/3lbfd1> > This email and any attachments transmitted with it are intended for use by > the intended recipient(s) only. If you have received this email in error, > please notify the sender immediately and then delete it. If you are not the > intended recipient, you must not keep, use, disclose, copy or distribute > this email without the author’s prior permission. We take precautions to > minimize the risk of transmitting software viruses, but we advise you to > perform your own virus checks on any attachment to this message. We cannot > accept liability for any loss or damage caused by software viruses. The > information contained in this communication may be confidential and may be > subject to the attorney-client privilege. >
