Also the hfile version setting and hbase version on your masters, presuming you are using the mob compaction chore in the master.
On Sat, Aug 17, 2019, 09:59 Sean Busbey <[email protected]> wrote: > That sounds like some of your hfiles are being written as hfile v2 instead > of v3. Can you check versions and configs on all your region servers? > > On Sat, Aug 17, 2019, 08:42 Omri Cohen <[email protected]> wrote: > >> Sorry, accidentally pressed "send" before i finished the massage. >> >> Hello, >> >> We recently encountered a problem in our production hbase cluster (CDH >> deployment, version 5.13.1). >> >> We have a cluster with a high mob percentage ( > 50% of objects are MOB, >> the threshold is the default 102400 bytes). The cluster has been active for >> about a year. Recently, our clients started to receive the mob reference >> instead of the mob data when trying to access certain rows (in most tables >> about 10% of the rows are bad, in one table it is about 50% of rows). >> >> When we investigated the HFiles, we saw that the "bad" cells are missing >> two tags that exist in the "good" rows, it looks something like that: >> >> K: {rowkey of a "good" row} {Column family, column qualifier}... >> vlen=76/seqid=... V: \x00\x04\xFB.{MOB file name} T[0]: T[1]: {table name} >> K: {rowkey of a "bad" row} {Column family, column qualifier}... >> vlen=76/seqid=... V: \x00\x13\x1A{MOB file name} >> >> The "good" row returns the expected data when queried, while the "bad" >> row returned the mob file reference instead of the data. This happened when >> we queried from the REST server, the native java client, and the hbase >> shell. >> >> When we examined the mob files, we saw that all the data was there. >> >> * Has anyone encountered a similar situation? >> * Is it possible to manually add the missing mob reference tags to >> the "bad" rows? >> >> No changes where made to the table in recently. We never encountered this >> issue before. >> >> We would appreciate any help on this issue. >> ________________________________ >> From: Omri Cohen >> Sent: Saturday, August 17, 2019 4:28 PM >> To: [email protected] <[email protected]> >> Subject: Mob reference tags go missing >> >> Hello, >> >> We recently encountered a problem in our production hbase cluster. >> We have a cluster with a high mob percentage ( > 50% of objects are MOB, >> the threshold is the default 102400 bytes). The cluster has been active for >> about a year. Recently, our clients started to receive the mob reference >> instead of the mob data when trying to access certain rows (in most tables >> about 10% of the rows are bad, in one table it is about 50% of rows). >> >> When we investigated the HFiles, we saw that the "bad" cells are missing >> two tags that exist in the "good" rows, it looks something like that: >> >
