Another option which I do not see mentioned yet is to extract the relevant common proto and source files from the ‘hbase’ repository into a new repository (‘hbase-storage’?), from which we would release artifacts to be consumed by both hbase and hbase-operator-tools. This maintains D.R.Y. through refactoring although it may down the road cause some complexity in coordinating evolution among the three (if not more) repositories and releases produced from them. This is like Josh’s Option 1 but without duplication.
Regarding the option 2 issue… If it would help we can drop SFT into branch-2.5 along with the log4j2 changes and release 2.5.0 afterward. We are taking the opportunity of this minor increment to accelerate log4j1 retirement, which is why it’s still waiting (but not for long). We can use the same opportunity to release SFT even if we designate it as an experimental feature if that would simplify some other logistics. For what it’s worth. > On Feb 15, 2022, at 7:44 AM, Josh Elser <els...@apache.org> wrote: > > I was talking with Szabolcs prior to him sending this one, and it's a tricky > issue for sure. > > To date, we've solved any HBase API issues by copying code into HBCK2 e.g. > HBCKMetaTableAccessor which copies parts of MetaTableAccessor, or we push the > logic down server-side to the HBase Master and invoke it over the Hbck RPC > interface. > > I definitely want to avoid HBase version specific builds of the > operator-tools, so that is not an option in my mind for 2.x. The discussions > we had (that I remember) around HBCK2 were limited in scope to HBase 2.x. > > Option 1: we copy the necessary proto files from HBase into the > operator-tools and try to remember that, if we make any change to the > serialization of the storefile list files, we have to copy that change to > HBCK2. Brittle on the surface but effective. > > Option 2: We bump HBCK2 to hbase-2.6.0-SNAPSHOT. Problematic until we make an > HBase 2.6.0[-alpha] release. We should already have wire compat between all > of HBase 2.x which makes that a non-issue. > > Option 3: We create an HBCK3 targeted for HBase 3.x. I'm not convinced we > need to do that (hbck for hbase 3.x would be just like hbck for hbase 2.x). > This would also not solve the problem for the SFT feature in hbase 2.6. > > I think option 3 is a no-go. I am leaning towards option 1 at this point. > Hopefully my thought process is helpful for others to weigh in. > > >> On 2/14/22 11:31 AM, Szabolcs Bukros wrote: >> Hi Folks! >> While working on adding tools to handle potential FileBased >> StoreFileTracker issues to HBCK2 (HBASE-26624 >> <https://issues.apache.org/jira/browse/HBASE-26624>) I ran into multiple >> problems I'm unsure how to solve. >> First of all the tools would rely on files not yet available in any of the >> released hbase artifacts. I tried to solve this without changing the hbase >> dependency version to keep HBCK2 as hbase version independent as possible, >> but none of the solutions I have found looked acceptable: >> - Pushing the logic to the hbase side (as far as I can tell) is not >> feasible because it has to be able to repair meta which is easier when >> hbase is down and the tool should be able to run without a working hbase. >> - The files tracking the store content are serialized proto objects so >> while replicating those files in the operator tools is possible, it would >> not be pretty. >> Bumping operator tools to use hbase 2.6.0-SNAPSHOT (branch-2 has the SFT >> changes) would mean that now we need that or a newer version to build the >> project and a version check to avoid runtime problems with the new tools, >> but otherwise this looks rather painless and backwards compatible. I know >> operator tools tries to avoid having a hbase-specific release, but having >> 2.6 as a min version to build against might be acceptable. >> While looking into this I also checked what needs to be done to make >> operator tools work with hbase 3.0.0-alpha-3-SNAPSHOT. Most of the changes >> are backwards compatible but not all of them and the ones that aren't would >> make a big chunk of Fsck unusable with older hbases. For me that looks >> acceptable since this is a major version change, but that would mean I can >> not rely on a potential HBCK3 to fix SFT issues, I would also need a >> solution for HBCK2. >> I tried to look for plans/direction regarding the new 1.3 operator tools >> but could not find any. >> Do you think it would be possible to bump the hbase version it uses to >> 2.6.0-SNAPSHOT? >> Do you think it would make sense to start working on a hbase3 compatible >> branch or is it too early? >> NOTE: >> I'm aware hbase does not publish SNAPSHOT builds for years, but I do not >> know how the internal build system works and if these artifacts would be >> available for internal builds or not. I also do not know if necessary could >> they be made available.
