This has landed in branch-2. Your assumption on the configuration toggle is correct: we should expect no changes to the output of these jobs (and thus no downstream impact in other jobs). The new directory structure is toggled via hbase.hfileoutputformat.tablename.namespace.inclusive, which I've documented in the release note.
Thank you both for your guidance here, and big thanks to Mallikarjun for his hard work on the backport and fixes. We've filed two follow-ons: https://issues.apache.org/jira/browse/HBASE-27584 - document the availability of experimental support as of 2.6.0 https://issues.apache.org/jira/browse/HBASE-27582 - errorprone cleanup in hbase-backup, which we noticed during the backport Members of my team have also pushed some improvements to backups in master recently, which I'll also backport shortly. Mallikarjun has some long open PRs for backups enhancements, which I hope to take a look at soon. On Wed, Jan 18, 2023 at 1:35 AM Andrew Purtell <apurt...@apache.org> wrote: > I assume supporting either directory structure with a configuration toggle > means existing code can coexist (in other jobs) with the backup feature, > and so it will be fine to port this into a code line that will release a > new minor. Sounds like a plan. +1 > > > On Tue, Jan 17, 2023 at 5:21 PM Bryan Beaudreault > <bbeaudrea...@hubspot.com.invalid> wrote: > > > Hey all, > > > > I reviewed the entirety of the backport PR. It's almost all net-new code, > > which matches identically with what was already reviewed and approved in > > the original implementation. There are some minor changes to > > HFileOutputFormat2 and WALPlayer (and related tests). > > > > Of the changes to those 2 files, there was only 1 incompatible change -- > a > > change to the directory structure of the output path for > > HFileOutputFormat2. This change is necessary to differentiate output > files > > for tables in different namespaces. However, in discussing with > Mallikarjun > > we are going to wrap this change in a config to preserve backwards > > compatibility. In addition to use by backup/restore, the config would > > provide users of HFileOutputFormat2 to have a path towards upgrading to > 3.0 > > in the future without breaking their downstream jobs which depend on the > > current structure. > > > > Let me know if there are any other concerns. Otherwise once the > > compatibility fix is in place and I give it a final review, I will merge > it > > to branch-2 (with a release note that it is experimental) and update our > > refguide. > > > > On Wed, Oct 5, 2022 at 8:25 PM Andrew Purtell <apurt...@apache.org> > wrote: > > > > > Agreed that the open tasks are not essential before considering a > > backport > > > for (near term) release. We have often released backported features > from > > > the main branch in new minors with documentation -- release notes and > > > updates to the online book, typically -- describing them as > > "experimental", > > > until something causes the community to reconsider that designation. I > > > assume this would happen in this case too? > > > > > > On Mon, Oct 3, 2022 at 11:03 AM Bryan Beaudreault > > > <bbeaudrea...@hubspot.com.invalid> wrote: > > > > > > > Hi again all, > > > > > > > > We have a relatively full featured backup solution in master branch. > It > > > > looks like the original development had intended to be included in > > > branch-2 > > > > [1], but did not make the deadline for 2.0.0 release and was removed > > [2]. > > > > Later the idea of backporting was forgotten, potentially with some of > > the > > > > main devs moving onto other projects. > > > > > > > > In the interim, one company, Flipkart (Mallikarjun works there), took > > it > > > > upon themselves to backport the feature to their own fork. They've > been > > > > running that backport in production for some time now. Mallikarjun > has > > > been > > > > trying to contribute some improvements, but has lacked committer > > support. > > > > > > > > At my company, we're considering redesigning our backup/restore > > solution > > > > which has been relatively static since originally built back in 2014 > > and > > > is > > > > showing its age. While investigating options, I reached out to > > > Mallikarjun > > > > and he was graciously willing to provide a backport PR [3]. The > > backport > > > > applied cleanly with small conflicts in one file. > > > > > > > > There were a few blockers listed in the original thread in [1] and > from > > > > what I can tell, they are all done. There is a remaining "Phase 4" > > > umbrella > > > > [4] with all of the issues looking like nice-to-haves. Most could > just > > be > > > > tackled based on community interest. > > > > > > > > I think a big reason why there is no committer support and relatively > > > > little uptake on this feature is because it has for years been stuck > on > > > > master, when pretty much everyone runs a 2.x release. So no one is > > using > > > it > > > > or has the ability to test it out, outside flipkart who backported it > > > > themselves. > > > > > > > > We are currently evaluating the backport PR in our own fork to see if > > the > > > > solution as-is can be a good foundation for what we're trying to > > > > accomplish. If the evaluation works out and we decide to move > forward, > > > I'd > > > > like to work with Mallikarjun to get his backport committed. > > > > > > > > Before doing that, I wanted to check if any other devs have concerns > > > about > > > > this initiative given the size of the patch and history of the > project. > > > Let > > > > me know what you think. > > > > > > > > [1] https://lists.apache.org/thread/k9dx12rj58r1qbrxqb0s1306lhhg3grn > <https://lists.apache.org/thread/k9dx12rj58r1qbrxqb0s1306lhhg3grn> > > > <https://lists.apache.org/thread/k9dx12rj58r1qbrxqb0s1306lhhg3grn > <https://lists.apache.org/thread/k9dx12rj58r1qbrxqb0s1306lhhg3grn> > > > > > > [2] https://issues.apache.org/jira/browse/HBASE-19407 > <https://issues.apache.org/jira/browse/HBASE-19407> > > > <https://issues.apache.org/jira/browse/HBASE-19407 > <https://issues.apache.org/jira/browse/HBASE-19407> > > > > > > [3] https://github.com/apache/hbase/pull/4770 > <https://github.com/apache/hbase/pull/4770> > > > <https://github.com/apache/hbase/pull/4770 > <https://github.com/apache/hbase/pull/4770> > > > > > > [4] https://issues.apache.org/jira/browse/HBASE-17362 > <https://issues.apache.org/jira/browse/HBASE-17362> > > > <https://issues.apache.org/jira/browse/HBASE-17362 > <https://issues.apache.org/jira/browse/HBASE-17362> > > > > > > > > > > > > > > > -- > > > Best regards, > > > Andrew > > > > > > Unrest, ignorance distilled, nihilistic imbeciles - > > > It's what we’ve earned > > > Welcome, apocalypse, what’s taken you so long? > > > Bring us the fitting end that we’ve been counting on > > > - A23, Welcome, Apocalypse > > > > > > > > -- > Best regards, > Andrew > > Unrest, ignorance distilled, nihilistic imbeciles - > It's what we’ve earned > Welcome, apocalypse, what’s taken you so long? > Bring us the fitting end that we’ve been counting on > - A23, Welcome, Apocalypse >
