Adam, Do you have GC logs? Can you see if those missing RFiles were removed by the GC process? That could indicate you somehow got old metadata info replayed. Also, the rfiles increment so compare the current rfile names in the srv.dir directory vs what is in the metadata table. Are the existing files after files in the metadata. Finally, pick a few of the missing files and grep all your master and tserver logs to see if you can learn anything. This sounds ungood.
Mike On Fri, May 11, 2018 at 6:06 PM Christopher <[email protected]> wrote: > This is strange. I've only ever seen this when HDFS has reported problems, > such as missing blocks, or another obvious failure. What is your durability > settings (were WALs turned on)? > > On Fri, May 11, 2018 at 12:45 PM Adam J. Shook <[email protected]> > wrote: > >> Hello all, >> >> On one of our clusters, there are a good number of missing RFiles from >> HDFS, however HDFS is not/has not reported any missing blocks. We were >> experiencing issues with HDFS; some flapping DataNode processes that needed >> more heap. >> >> I don't anticipate I can do much besides create a bunch of empty RFiles >> (open to suggestions). My question is, Is it possible that Accumulo could >> have written the metadata for these RFiles but failed to write it to HDFS? >> In which case it would have been re-tried later and the data was persisted >> to a different RFile? Or is it an 'RFile is in Accumulo metadata if and >> only if it is in HDFS' situation? >> >> Accumulo 1.8.1 on HDFS 2.6.0. >> >> Thank you, >> --Adam >> >
