hsync/hflush, or any input/output stream APIs for that matter, can be probed using StreamCapabilities.hasCapabilitiy() API.
lease recovery isn't (DistributedFIleSystem.recoverLease()). safe mode check isn't. There are a number of HDFS specific APIs that HBase uses. I'm all for abstracting out FS implementation details. But it would be an overkill to try and add every single FS specific APIs to the generic FileSystem API interface. One idea that Stephen had was to add a RecoverableFileSystem interface in Hadoop which adds lease recovery capability, and then HDFS or Ozone can implement this interface. In a less ideal world, I imagine we could have one hbase module for utilities that does FS specific tasks. That way it is future proofing. On Wed, Mar 15, 2023 at 12:50 PM Andrew Purtell <apurt...@apache.org> wrote: > Does Hadoop have a marker interface that lets an application know its > FileSystem instances can support hsync/hflush? Ideally all we should need > to do is test with instanceof for that marker and use reflection (in the > worst case) to get a handle to the hsync or hflush method, and then call > it. This approach should be taken wherever we have a requirement to use a > special WAL specific API provided by the underlying FileSystem, so we can > abstract it sufficiently to not require a direct dependency on Ozone or S3A > or any non HDFS filesystem. > > On Wed, Mar 15, 2023 at 12:31 PM Tak Lon (Stephen) Wu <tak...@apache.org> > wrote: > > > Hi team, > > > > Recently, Wei-Chiu and I have been discussing about if HBase can use > > Ozone as another storage as WAL (see the hsync and hflush JIRAs [1]) > > and HFile, for HFile it’s pluggable by configuring the file system to > > use Ozone File System (Ozone) > > > > But we found that the WAL it’s a bit different, especially > > RecoverLeaseFSUtils#recoverFileLease [2], it has one check about if > > the file system is an instance of HDFS, and thus WAL recovery to > > execute file lease recovery from RS crashes. Here, if we would like to > > add Ozone, it does not matter by importing as a direct dependency to > > perform similar lease recovery or via reflection by class name in > > plaintext String, we still need to somehow introduce Ozone to be > > another supported file system. (we can discuss how we can implement > > better as well) > > > > We also found other places e.g. FSUtils and HFileSystem have used > > DistributedFileSystem, but it should be able to move them into either > > hbase-asyncfs or a new FS related component to separate the use of > > different supported file systems. > > > > So, we’re wondering if anyone would have any objections to adding > > Ozone as a dependency to hbase-asyncfs? or if you have a better idea > > how this could be added without adding Ozone as dependency, please > > feel free to comment on this thread. > > > > > > [1] Ozone is working on support for hsync and hflush, > > https://issues.apache.org/jira/browse/HDDS-7593, > > https://issues.apache.org/jira/browse/HDDS-4353 > > [2] RecoverLeaseFSUtils#recoverFileLease, > > > > > https://github.com/apache/hbase/blob/master/hbase-asyncfs/src/main/java/org/apache/hadoop/hbase/util/RecoverLeaseFSUtils.java#L53-L63 > > > > Thanks, > > Stephen > > > > > -- > Best regards, > Andrew > > Unrest, ignorance distilled, nihilistic imbeciles - > It's what we’ve earned > Welcome, apocalypse, what’s taken you so long? > Bring us the fitting end that we’ve been counting on > - A23, Welcome, Apocalypse >
