Does Hadoop have a marker interface that lets an application know its FileSystem instances can support hsync/hflush? Ideally all we should need to do is test with instanceof for that marker and use reflection (in the worst case) to get a handle to the hsync or hflush method, and then call it. This approach should be taken wherever we have a requirement to use a special WAL specific API provided by the underlying FileSystem, so we can abstract it sufficiently to not require a direct dependency on Ozone or S3A or any non HDFS filesystem.
On Wed, Mar 15, 2023 at 12:31 PM Tak Lon (Stephen) Wu <tak...@apache.org> wrote: > Hi team, > > Recently, Wei-Chiu and I have been discussing about if HBase can use > Ozone as another storage as WAL (see the hsync and hflush JIRAs [1]) > and HFile, for HFile it’s pluggable by configuring the file system to > use Ozone File System (Ozone) > > But we found that the WAL it’s a bit different, especially > RecoverLeaseFSUtils#recoverFileLease [2], it has one check about if > the file system is an instance of HDFS, and thus WAL recovery to > execute file lease recovery from RS crashes. Here, if we would like to > add Ozone, it does not matter by importing as a direct dependency to > perform similar lease recovery or via reflection by class name in > plaintext String, we still need to somehow introduce Ozone to be > another supported file system. (we can discuss how we can implement > better as well) > > We also found other places e.g. FSUtils and HFileSystem have used > DistributedFileSystem, but it should be able to move them into either > hbase-asyncfs or a new FS related component to separate the use of > different supported file systems. > > So, we’re wondering if anyone would have any objections to adding > Ozone as a dependency to hbase-asyncfs? or if you have a better idea > how this could be added without adding Ozone as dependency, please > feel free to comment on this thread. > > > [1] Ozone is working on support for hsync and hflush, > https://issues.apache.org/jira/browse/HDDS-7593, > https://issues.apache.org/jira/browse/HDDS-4353 > [2] RecoverLeaseFSUtils#recoverFileLease, > > https://github.com/apache/hbase/blob/master/hbase-asyncfs/src/main/java/org/apache/hadoop/hbase/util/RecoverLeaseFSUtils.java#L53-L63 > > Thanks, > Stephen > -- Best regards, Andrew Unrest, ignorance distilled, nihilistic imbeciles - It's what we’ve earned Welcome, apocalypse, what’s taken you so long? Bring us the fitting end that we’ve been counting on - A23, Welcome, Apocalypse
