Alexander, I can tell you what's happening but I don't know why.
When you do a Put in HBase (which is what Phoenix UPSERTs do underneath), it gets committed to a mutable in-memory buffer called the MemStore. Periodically, the MemStore is flushed to a physical HDFS file called an HFile. The rule for HFiles is that all of the data inside them is sorted by row key. That means that the code to flush the memstore to an HFile has a built in sanity check that makes sure that each Cell flushed to an HBase HFile is monotonically increasing. That sanity check is failing here for some reason. You mention that this happens constantly with increased write load -- that's likely because more write load causes more MemStore flushes. Since you're using a global index, which stores the index data in a separate table (and hence different regions, each of which has a different MemStore / set of HFiles), and the error's happening to the base table, I'd be surprised if Phoenix indexing is related. What coprocessors (Phoenix and otherwise) are loaded on the table? Geoffrey On Thu, Aug 15, 2019 at 12:49 PM Alexander Batyrshin <0x62...@gmail.com> wrote: > > > On 15 Aug 2019, at 21:27, Josh Elser <els...@apache.org> wrote: > > > > Short-answer, it's possible that something around secondary indexing in > Phoenix causes this but not possible to definitively say in a vaccuum. > > > As I see region-server crashes on main table (not index) memstore flush. > How can I help to provide more information? > Maybe I should increase logging level for some specific java class of > Phoenix or HBase?
