Clcanny commented on PR #5712: URL: https://github.com/apache/hudi/pull/5712#issuecomment-1153042140
> Can you help me understand why we need this. for any new storage scheme, typically we ask the user to test out hudi end to end once and then add it explicitly to supported storage schemes. But w/ this if we keep it loose, we never know if a particular storage scheme works or not. Thanks for your review. There are two reasons to keep schemes loose: 1. Name conflict. `hdfs://`, `file://` and `s3://` are famous schemes, no other filesystem won't reuse names of these filesystems. But for a scheme of a not so that famous filesystem, name can be conflict. For example, vendor a uses `xfs://` for a new filesystem, and it is append-able; vendor b uses `xfs://` for a new filesystem, and it is not append-able. 2. Let user specify file system properties. For example, append-able is `true` for hdfs which doesn't support erasure coding; but for some vendor-specific hdfs which support erasure coding, append-able is `false`. For now, there is only one property on filesystem(append-able), but there may be a few properties on filesystem(e.g. truncate-able) in future. In my opinion, let users decide to use which file systems with hudi is a good idea. Hudi should provide end-to-end tests. And it is users's job to run end-to-end tests when integrating new file system with hudi. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
