CTTY commented on code in PR #9462: URL: https://github.com/apache/hudi/pull/9462#discussion_r1510428461
########## rfc/rfc-60/rfc-60.md: ########## @@ -196,13 +195,75 @@ for metadata table to be populated. 4. If there is an error reading from Metadata table, we will not fall back listing from file system. -5. In case of metadata table getting corrupted or lost, we need to have a solution here to reconstruct metadata table -from the files which distributed using federated storage. We will likely have to implement a file system listing -logic, that can get all the partition to files mapping by listing all the prefixes under the `Table Storage Path`. -Following the folder structure of adding table name/partitions under the prefix will help in getting the listing and -identifying the table/partition they belong to. +### Integration +This section mainly describes how storage strategy is integrated with other components and how read/write +would look like from Hudi side with object storage layout. + +We propose integrating the storage strategy at the filesystem level, specifically within `HoodieWrapperFileSystem`. +This way, only file read/write operations undergo path conversion and we can limit the usage of +storage strategy to only filesystem level so other upper-level components don't need to be aware of physical paths. + +This also mandates that `HoodieWrapperFileSystem` is the filesystem of choice for all upper-level Hudi components. +Getting filesystem from `Path` or such won't be allowed anymore as using raw filesystem may not reach +to physical locations without storage strategy. Hudi components can simply call `HoodieMetaClient#getFs` +to get `HoodieWrapperFileSystem`, and this needs to be the only allowed way for any filesystem-related operation. +The only exception is when we need to interact with metadata that's still stored under the original table path, +and we should call `HoodieMetaClient#getRawFs` in this case so `HoodieMetaClient` can still be the single entry +for getting filesystem. + + + +When conducting a read operation, Hudi would: +1. Access filesystem view, `HoodieMetadataFileSystemView` specifically +2. Scan metadata table via filesystem view to compose `HoodieMetadataPayload` +3. Call `HoodieMetadataPayload#getFileStatuses` and employ `HoodieWrapperFileSystem` to get Review Comment: Excatly -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
