GitHub user danny0405 added a comment to the discussion: Add gvfs as a storage scheme in Hudi
Supporting GVFS sounds reasonable, but I think we should validate a few storage semantics before treating an entry in `StorageSchemes` as sufficient. GVFS can resolve to different underlying filesystems, including HDFS and S3-compatible storage. `StorageSchemes` also determines transactional-write and atomic-create capabilities, so could you clarify which backing storage you are using and what values you propose for those flags? A single set of flags needs to be justified across the supported backends. There is also a separate scheme-dependent path in [`HoodieStorage.needCreateTempFile()`](https://github.com/apache/hudi/blob/master/hudi-io/src/main/java/org/apache/hudi/storage/HoodieStorage.java): it currently enables temporary-file-then-rename publication only for `hdfs`, `viewfs`, and `file`. Adding `gvfs` to the enum alone would leave this returning false even for an HDFS-backed fileset. We should verify how partially written immutable/timeline files are prevented from becoming visible in that case. It would help to include integration coverage for GVFS over both HDFS and S3-compatible storage (MinIO would be useful for the latter), covering commits, failed-write recovery/rollback, and Merge-on-Read updates if included in the supported scope. Please also document any locking or concurrency limitations. For context, MinIO does not need its own scheme entry: Hudi accesses it through the existing `s3a` scheme with a configured endpoint. GVFS exposes a distinct `gvfs` scheme, which is why it encounters this check. GitHub link: https://github.com/apache/hudi/discussions/19961#discussioncomment-18459278 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
