So we can test StreamCapabilities in code, in worst case by wrapping some probe code during startup with try-catch and examining the exception.
On Wed, Mar 15, 2023 at 1:09 PM Viraj Jasani <vjas...@apache.org> wrote: > As of today, both WAL impl (fshlog and asyncfs) throw > StreamLacksCapabilityException if the FS Data OutputStream probe fails for > Hflush/Hsync: > > StreamLacksCapabilityException(StreamCapabilities.HFLUSH) > and > StreamLacksCapabilityException(StreamCapabilities.HSYNC) > > > On Wed, Mar 15, 2023 at 12:51 PM Andrew Purtell <apurt...@apache.org> > wrote: > > > Does Hadoop have a marker interface that lets an application know its > > FileSystem instances can support hsync/hflush? Ideally all we should need > > to do is test with instanceof for that marker and use reflection (in the > > worst case) to get a handle to the hsync or hflush method, and then call > > it. This approach should be taken wherever we have a requirement to use a > > special WAL specific API provided by the underlying FileSystem, so we can > > abstract it sufficiently to not require a direct dependency on Ozone or > S3A > > or any non HDFS filesystem. > > > > On Wed, Mar 15, 2023 at 12:31 PM Tak Lon (Stephen) Wu <tak...@apache.org > > > > wrote: > > > > > Hi team, > > > > > > Recently, Wei-Chiu and I have been discussing about if HBase can use > > > Ozone as another storage as WAL (see the hsync and hflush JIRAs [1]) > > > and HFile, for HFile it’s pluggable by configuring the file system to > > > use Ozone File System (Ozone) > > > > > > But we found that the WAL it’s a bit different, especially > > > RecoverLeaseFSUtils#recoverFileLease [2], it has one check about if > > > the file system is an instance of HDFS, and thus WAL recovery to > > > execute file lease recovery from RS crashes. Here, if we would like to > > > add Ozone, it does not matter by importing as a direct dependency to > > > perform similar lease recovery or via reflection by class name in > > > plaintext String, we still need to somehow introduce Ozone to be > > > another supported file system. (we can discuss how we can implement > > > better as well) > > > > > > We also found other places e.g. FSUtils and HFileSystem have used > > > DistributedFileSystem, but it should be able to move them into either > > > hbase-asyncfs or a new FS related component to separate the use of > > > different supported file systems. > > > > > > So, we’re wondering if anyone would have any objections to adding > > > Ozone as a dependency to hbase-asyncfs? or if you have a better idea > > > how this could be added without adding Ozone as dependency, please > > > feel free to comment on this thread. > > > > > > > > > [1] Ozone is working on support for hsync and hflush, > > > https://issues.apache.org/jira/browse/HDDS-7593, > > > https://issues.apache.org/jira/browse/HDDS-4353 > > > [2] RecoverLeaseFSUtils#recoverFileLease, > > > > > > > > > https://github.com/apache/hbase/blob/master/hbase-asyncfs/src/main/java/org/apache/hadoop/hbase/util/RecoverLeaseFSUtils.java#L53-L63 > > > > > > Thanks, > > > Stephen >
