noslowerdna commented on PR #8485: URL: https://github.com/apache/hbase/pull/8485#issuecomment-5178706959
> For unit tests, you might want to check out TestSeekOptimizations. The existing cases seem to compare lazy seeks against eager. They never set a filter though. Sounds good from here. I will see today what corresponding updates can be made there. > Regarding the new configuration property, I don't think we should add a flag to re-introduce a correctness bug unless the performance hit is really unacceptable, and we don't know that yet. But I doubt it is. Agreed. > I also find it hard to come up with a realistic scenario that would actually suffer from the regression, since it depends on several factors at once: ... Right, for that reason my team's consensus opinion is that the risk of this correction being problematic for any real-world application is extremely small. > Out of curiosity: How did you run into this issue? Do you have a production workload that was affected by the bug? If so, what does that workload look like? Yes, we did. The issue was encountered in a dev environment with an existing legacy data processing workload being migrated from HBase 1.x to 2.x. It has a custom multi-segment row key binary serialization format with a RowFilter for scan efficiency (CompareOperator.EQUAL + ByteArrayComparable impl). We use the DataInputStream class to decode its segments. Reading the stream for a row ended unexpectedly when a `0x00` terminal byte was not found for a string segment resulting in an EOFException thrown. A generic IOException catch block wrapped this in a RuntimeException, and a batch processing workflow failed to start. At first we were confused how a row key could have become corrupted in such a way when encoded from its natural structured form, then noticed the value was a region start boundary. That led us to this HBase JIRA opened a few years ago. For now, our temporary mitigation is catching any Exception (and returning 1 to skip the row) as analysis showed several different possibilities if decoding arbitrary bytes. While the code is certainly more robust now with that guardrail in place, any issue where the encoding of real row keys is malformed would effectively cause quiet (apart from region server error logging) downstream data loss rather than failing fast which is what we would want. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
