GitHub user architlatkar27 added a comment to the discussion: Add gvfs as a
storage scheme in Hudi
Hi @danny0405 thanks for your inputs here
I went through some of the code myself and appreciate the complexity with gvfs.
Let me try to answer some of the questions you have posed.
> could you clarify which backing storage you are using and what values you
> propose for those flags?
We were using gvfs on top of hdfs filesystem. We are also planning to register
and consume S3, Azure, Gcs catalogs as filesets in the future and we may need
some solution there as well.
In a patch that i tried out i had set `GVFS("gvfs", null, null, null)` and it
had worked for us. But admittedly we are trying out some simple stuff as of now
so it might not cause much issues. I understand that this might not exactly be
the right configuration for S3, ABFS etc.
The complexity also comes due to S3, Abfs, Gcs and Hdfs using different
configurations.
I kept everything null above since that felt like the lowest common
denominator.
> We should verify how partially written immutable/timeline files are prevented
> from becoming visible in that case.
I will try to reproduce something like this. Do we have integration tests that
can cover such a scenario?
GitHub link:
https://github.com/apache/hudi/discussions/19961#discussioncomment-18475316
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
