I like the idea of including the store file tracking in 2.5.0 to unblock the HBCK development efforts.
Unfortunately, I was not following its development that much. Can it cause any issues if 2.5.0 has the feature but later an incompatible change is needed for SFT? Can it be marked as a beta feature where we are free to modify interfaces? Peter On Tue, Feb 15, 2022 at 11:07 PM Andrew Purtell <andrew.purt...@gmail.com> wrote: > Another option which I do not see mentioned yet is to extract the relevant > common proto and source files from the ‘hbase’ repository into a new > repository (‘hbase-storage’?), from which we would release artifacts to be > consumed by both hbase and hbase-operator-tools. This maintains D.R.Y. > through refactoring although it may down the road cause some complexity in > coordinating evolution among the three (if not more) repositories and > releases produced from them. This is like Josh’s Option 1 but without > duplication. > > Regarding the option 2 issue… If it would help we can drop SFT into > branch-2.5 along with the log4j2 changes and release 2.5.0 afterward. We > are taking the opportunity of this minor increment to accelerate log4j1 > retirement, which is why it’s still waiting (but not for long). We can use > the same opportunity to release SFT even if we designate it as an > experimental feature if that would simplify some other logistics. For what > it’s worth. > > > On Feb 15, 2022, at 7:44 AM, Josh Elser <els...@apache.org> wrote: > > > > I was talking with Szabolcs prior to him sending this one, and it's a > tricky issue for sure. > > > > To date, we've solved any HBase API issues by copying code into HBCK2 > e.g. HBCKMetaTableAccessor which copies parts of MetaTableAccessor, or we > push the logic down server-side to the HBase Master and invoke it over the > Hbck RPC interface. > > > > I definitely want to avoid HBase version specific builds of the > operator-tools, so that is not an option in my mind for 2.x. The > discussions we had (that I remember) around HBCK2 were limited in scope to > HBase 2.x. > > > > Option 1: we copy the necessary proto files from HBase into the > operator-tools and try to remember that, if we make any change to the > serialization of the storefile list files, we have to copy that change to > HBCK2. Brittle on the surface but effective. > > > > Option 2: We bump HBCK2 to hbase-2.6.0-SNAPSHOT. Problematic until we > make an HBase 2.6.0[-alpha] release. We should already have wire compat > between all of HBase 2.x which makes that a non-issue. > > > > Option 3: We create an HBCK3 targeted for HBase 3.x. I'm not convinced > we need to do that (hbck for hbase 3.x would be just like hbck for hbase > 2.x). This would also not solve the problem for the SFT feature in hbase > 2.6. > > > > I think option 3 is a no-go. I am leaning towards option 1 at this > point. Hopefully my thought process is helpful for others to weigh in. > > > > > >> On 2/14/22 11:31 AM, Szabolcs Bukros wrote: > >> Hi Folks! > >> While working on adding tools to handle potential FileBased > >> StoreFileTracker issues to HBCK2 (HBASE-26624 > >> <https://issues.apache.org/jira/browse/HBASE-26624>) I ran into > multiple > >> problems I'm unsure how to solve. > >> First of all the tools would rely on files not yet available in any of > the > >> released hbase artifacts. I tried to solve this without changing the > hbase > >> dependency version to keep HBCK2 as hbase version independent as > possible, > >> but none of the solutions I have found looked acceptable: > >> - Pushing the logic to the hbase side (as far as I can tell) is not > >> feasible because it has to be able to repair meta which is easier when > >> hbase is down and the tool should be able to run without a working > hbase. > >> - The files tracking the store content are serialized proto objects so > >> while replicating those files in the operator tools is possible, it > would > >> not be pretty. > >> Bumping operator tools to use hbase 2.6.0-SNAPSHOT (branch-2 has the SFT > >> changes) would mean that now we need that or a newer version to build > the > >> project and a version check to avoid runtime problems with the new > tools, > >> but otherwise this looks rather painless and backwards compatible. I > know > >> operator tools tries to avoid having a hbase-specific release, but > having > >> 2.6 as a min version to build against might be acceptable. > >> While looking into this I also checked what needs to be done to make > >> operator tools work with hbase 3.0.0-alpha-3-SNAPSHOT. Most of the > changes > >> are backwards compatible but not all of them and the ones that aren't > would > >> make a big chunk of Fsck unusable with older hbases. For me that looks > >> acceptable since this is a major version change, but that would mean I > can > >> not rely on a potential HBCK3 to fix SFT issues, I would also need a > >> solution for HBCK2. > >> I tried to look for plans/direction regarding the new 1.3 operator tools > >> but could not find any. > >> Do you think it would be possible to bump the hbase version it uses to > >> 2.6.0-SNAPSHOT? > >> Do you think it would make sense to start working on a hbase3 compatible > >> branch or is it too early? > >> NOTE: > >> I'm aware hbase does not publish SNAPSHOT builds for years, but I do not > >> know how the internal build system works and if these artifacts would be > >> available for internal builds or not. I also do not know if necessary > could > >> they be made available. >
