anuragmantri edited a comment on issue #1617:
URL: https://github.com/apache/iceberg/issues/1617#issuecomment-880068307
Catching up on the comments from past couple of days. Thanks everyone for
providing inputs.
@flyrain - Although it will increase the scope of this design to include
bi-directional replication, we should consider covering that here.
Overall, I agree with @jackye1995's proposal. Thanks for the detailed
explanation.
> Then these 4 properties are used to get 2 absolute root paths:
>
> root path to write data: the LocationProvider would read some fields in
this list to determine the base root path, and generate absolute paths to write
data files.
> root path to write metadata: similarly, the
TableOperations.metadataFileLocation reads some fields above to determine the
base root path, and generate paths for metadata files.
In the initial design doc, my thinking was the writes mentioned above need
to change at all. However, reading the comments from @pvary above, we may
indeed have to store a `root` location somewhere for cases where we want to
read inaccessible paths. Whether we want to have explicit method to set this
root before accessing a table or have a mechanism for replicated tables to
register themselves as replicas and then use a retry logic can be discussed.
My takeaways from this are:
1. Metadata writers need to know
a) Relative path boolean
b) True root of the table
c) Metadata path if set
Additionally, they will also convert paths to relative in metadata files.
2. Data writers need to know
a) Data path if set
b) True root of the table
An update transaction will change all the above except relative path
boolean. Please correct me if my understanding is incorrect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
