[jira] [Updated] (HDDS-14943) Implement manifest list rewrite logic for path migration
[ https://issues.apache.org/jira/browse/HDDS-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Doroszlai updated HDDS-14943: Fix Version/s: 2.2.0 > Implement manifest list rewrite logic for path migration > > > Key: HDDS-14943 > URL: https://issues.apache.org/jira/browse/HDDS-14943 > Project: Apache Ozone > Issue Type: Sub-task >Reporter: Sreeja >Assignee: Sreeja >Priority: Major > Labels: pull-request-available > Fix For: 2.2.0 > > > Add logic to find the valid snapshots of the Iceberg table and rewrite > manifest lists associated with those snapshots by Iterating through each > valid snapshot. We can take help of Iceberg's RewriteTablePathUtil methods to > replace sourcePrefix -> targetPrefix for each manifest file absolute path > mentioned in the manifest list associated with the respective snapshot and > then add the rewritten manifest list file in staging location. > We should also update copy plan with staging -> final path mappings for each > manifest list file rewritten. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Updated] (HDDS-14943) Implement manifest list rewrite logic for path migration
[ https://issues.apache.org/jira/browse/HDDS-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-14943: -- Labels: pull-request-available (was: ) > Implement manifest list rewrite logic for path migration > > > Key: HDDS-14943 > URL: https://issues.apache.org/jira/browse/HDDS-14943 > Project: Apache Ozone > Issue Type: Sub-task >Reporter: Sreeja >Assignee: Sreeja >Priority: Major > Labels: pull-request-available > > Add logic to find the valid snapshots of the Iceberg table and rewrite > manifest lists associated with those snapshots by Iterating through each > valid snapshot. We can take help of Iceberg's RewriteTablePathUtil methods to > replace sourcePrefix -> targetPrefix for each manifest file absolute path > mentioned in the manifest list associated with the respective snapshot and > then add the rewritten manifest list file in staging location. > We should also update copy plan with staging -> final path mappings for each > manifest list file rewritten. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Updated] (HDDS-14943) Implement manifest list rewrite logic for path migration
[ https://issues.apache.org/jira/browse/HDDS-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreeja updated HDDS-14943: -- Description: Add logic to find the valid snapshots of the Iceberg table and rewrite manifest lists associated with those snapshots by Iterating through each valid snapshot. We can take help of Iceberg's RewriteTablePathUtil methods to replace sourcePrefix -> targetPrefix for each manifest file absolute path mentioned in the manifest list associated with the respective snapshot and then add the rewritten manifest list file in staging location. We should also update copy plan with staging -> final path mappings for each manifest list file rewritten. was: Add logic to find the valid snapshots for the table and rewrite manifest lists associated with those snapshots by Iterating through each valid snapshot. We can take help of Iceberg's RewriteTablePathUtil methods to replace sourcePrefix -> targetPrefix for each manifest file absolute path mentioned in the manifest list associated with the respective snapshot and then add the rewritten manifest list file in staging location. We should also update copy plan with staging -> final path mappings for each manifest list file rewritten. > Implement manifest list rewrite logic for path migration > > > Key: HDDS-14943 > URL: https://issues.apache.org/jira/browse/HDDS-14943 > Project: Apache Ozone > Issue Type: Sub-task >Reporter: Sreeja >Assignee: Sreeja >Priority: Major > > Add logic to find the valid snapshots of the Iceberg table and rewrite > manifest lists associated with those snapshots by Iterating through each > valid snapshot. We can take help of Iceberg's RewriteTablePathUtil methods to > replace sourcePrefix -> targetPrefix for each manifest file absolute path > mentioned in the manifest list associated with the respective snapshot and > then add the rewritten manifest list file in staging location. > We should also update copy plan with staging -> final path mappings for each > manifest list file rewritten. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Updated] (HDDS-14943) Implement manifest list rewrite logic for path migration
[ https://issues.apache.org/jira/browse/HDDS-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreeja updated HDDS-14943: -- Description: Add logic to find the valid snapshots for the table and rewrite manifest lists associated with those snapshots by Iterating through each valid snapshot. We can take help of Iceberg's RewriteTablePathUtil methods to replace sourcePrefix -> targetPrefix for each manifest file absolute path mentioned in the manifest list associated with the respective snapshot and then add the rewritten manifest list file in staging location. We should also update copy plan with staging -> final path mappings for each manifest list file rewritten. was: Add logic to find the valid snapshots for the table and rewrite manifest lists associated with those snapshots by Iterating through each valid snapshot. We can take help of Iceberg's RewriteTablePathUtil methods to replace sourcePrefix -> targetPrefix for each manifest file absolute path mentioned in the manifest list associated with the respective snapshot and then write the rewritten manifest list file in staging location. We should also update copy plan with staging -> final path mappings for each manifest list file rewritten. > Implement manifest list rewrite logic for path migration > > > Key: HDDS-14943 > URL: https://issues.apache.org/jira/browse/HDDS-14943 > Project: Apache Ozone > Issue Type: Sub-task >Reporter: Sreeja >Assignee: Sreeja >Priority: Major > > Add logic to find the valid snapshots for the table and rewrite manifest > lists associated with those snapshots by Iterating through each valid > snapshot. We can take help of Iceberg's RewriteTablePathUtil methods to > replace sourcePrefix -> targetPrefix for each manifest file absolute path > mentioned in the manifest list associated with the respective snapshot and > then add the rewritten manifest list file in staging location. > We should also update copy plan with staging -> final path mappings for each > manifest list file rewritten. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Updated] (HDDS-14943) Implement manifest list rewrite logic for path migration
[ https://issues.apache.org/jira/browse/HDDS-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreeja updated HDDS-14943: -- Description: Add logic to find the valid snapshots for the table and rewrite manifest lists associated with those snapshots by Iterating through each valid snapshot. We can take help of Iceberg's RewriteTablePathUtil methods to replace sourcePrefix -> targetPrefix for each manifest file absolute path mentioned in the manifest list associated with the respective snapshot and then write the rewritten manifest list file in staging location. We should also update copy plan with staging -> final path mappings for each manifest list file rewritten. was: Add logic to find the valid snapshots for the table and rewrite manifest lists associated with those snapshots by Iterating through each valid snapshot. We can take help of Iceberg's RewriteTablePathUtil methods to replace sourcePrefix -> targetPrefix in each and write the rewritten file in staging location. We should also update copy plan with staging -> final path mappings for each manifest list file rewritten. > Implement manifest list rewrite logic for path migration > > > Key: HDDS-14943 > URL: https://issues.apache.org/jira/browse/HDDS-14943 > Project: Apache Ozone > Issue Type: Sub-task >Reporter: Sreeja >Assignee: Sreeja >Priority: Major > > Add logic to find the valid snapshots for the table and rewrite manifest > lists associated with those snapshots by Iterating through each valid > snapshot. We can take help of Iceberg's RewriteTablePathUtil methods to > replace sourcePrefix -> targetPrefix for each manifest file absolute path > mentioned in the manifest list associated with the respective snapshot and > then write the rewritten manifest list file in staging location. > We should also update copy plan with staging -> final path mappings for each > manifest list file rewritten. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Updated] (HDDS-14943) Implement manifest list rewrite logic for path migration
[ https://issues.apache.org/jira/browse/HDDS-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreeja updated HDDS-14943: -- Description: Add logic to find the valid snapshots for the table and rewrite manifest lists associated with those snapshots by Iterating through each valid snapshot. We can take help of Iceberg's RewriteTablePathUtil methods to replace sourcePrefix -> targetPrefix in each and write the rewritten file in staging location. We should also update copy plan with staging -> final path mappings for each manifest list file rewritten. was: Add logic to find the valid snapshots for the table and rewrite manifest lists associated with those snapshots by Iterating through each valid snapshot. We take help of Iceberg's RewriteTablePathUtil methods to replace sourcePrefix -> targetPrefix in each and write the rewritten file in staging location. We should also update copy plan with staging -> final path mappings for each manifest list file rewritten. > Implement manifest list rewrite logic for path migration > > > Key: HDDS-14943 > URL: https://issues.apache.org/jira/browse/HDDS-14943 > Project: Apache Ozone > Issue Type: Sub-task >Reporter: Sreeja >Assignee: Sreeja >Priority: Major > > Add logic to find the valid snapshots for the table and rewrite manifest > lists associated with those snapshots by Iterating through each valid > snapshot. We can take help of Iceberg's RewriteTablePathUtil methods to > replace sourcePrefix -> targetPrefix in each and write the rewritten file in > staging location. > We should also update copy plan with staging -> final path mappings for each > manifest list file rewritten. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Updated] (HDDS-14943) Implement manifest list rewrite logic for path migration
[ https://issues.apache.org/jira/browse/HDDS-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreeja updated HDDS-14943: -- Description: Add logic to find the valid snapshots for the table and rewrite manifest lists associated with those snapshots by Iterating through each valid snapshot. We take help of Iceberg's RewriteTablePathUtil methods to replace sourcePrefix -> targetPrefix in each and write the rewritten file in staging location. We should also update copy plan with staging -> final path mappings for each manifest list file rewritten. was: Add logic to find the valid snapshots for the table and rewrite manifest lists associated with those snapshots by Iterating through each valid snapshot. We can use Iceberg's RewriteTablePathUtil methods to replace sourcePrefix -> targetPrefix in each and write the rewritten file in staging location. We should also update copy plan with staging -> final path mappings for each manifest list file rewritten. > Implement manifest list rewrite logic for path migration > > > Key: HDDS-14943 > URL: https://issues.apache.org/jira/browse/HDDS-14943 > Project: Apache Ozone > Issue Type: Sub-task >Reporter: Sreeja >Assignee: Sreeja >Priority: Major > > Add logic to find the valid snapshots for the table and rewrite manifest > lists associated with those snapshots by Iterating through each valid > snapshot. We take help of Iceberg's RewriteTablePathUtil methods to replace > sourcePrefix -> targetPrefix in each and write the rewritten file in staging > location. > We should also update copy plan with staging -> final path mappings for each > manifest list file rewritten. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Updated] (HDDS-14943) Implement manifest list rewrite logic for path migration
[ https://issues.apache.org/jira/browse/HDDS-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreeja updated HDDS-14943: -- Description: Add logic to find the valid snapshots for the table and rewrite manifest lists associated with those snapshots by Iterating through each valid snapshot. We can use Iceberg's RewriteTablePathUtil methods to replace sourcePrefix -> targetPrefix in each and write the rewritten file in staging location. We should also update copy plan with staging -> final path mappings for each manifest list file rewritten. was: Add logic to find the valid snapshots for the table and rewrite manifest lists associated with those snapshots by Iterating through each valid snapshot. We can use Iceberg's RewriteTablePathUtil methods to replace sourcePrefix -> targetPrefix in each and update the rewritten file in staging location. We should also update copy plan with staging -> final path mappings for each manifest list file rewritten. > Implement manifest list rewrite logic for path migration > > > Key: HDDS-14943 > URL: https://issues.apache.org/jira/browse/HDDS-14943 > Project: Apache Ozone > Issue Type: Sub-task >Reporter: Sreeja >Assignee: Sreeja >Priority: Major > > Add logic to find the valid snapshots for the table and rewrite manifest > lists associated with those snapshots by Iterating through each valid > snapshot. We can use Iceberg's RewriteTablePathUtil methods to replace > sourcePrefix -> targetPrefix in each and write the rewritten file in staging > location. > We should also update copy plan with staging -> final path mappings for each > manifest list file rewritten. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Updated] (HDDS-14943) Implement manifest list rewrite logic for path migration
[ https://issues.apache.org/jira/browse/HDDS-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreeja updated HDDS-14943: -- Description: Add logic to find the valid snapshots for the table and rewrite manifest lists associated with those snapshots by Iterating through each valid snapshot. We can use Iceberg's RewriteTablePathUtil methods to replace sourcePrefix -> targetPrefix in each and update the rewritten file in staging location. We should also update copy plan with staging -> final path mappings for each manifest list file rewritten. was: Add logic to *rewrite manifest lists for snapshots* by Iterating over valid snapshots and rewrite their associated manifest lists by replacing sourcePrefix -> targetPrefix in each and update in staging location (using iceberg's RewriteTablePathUtil) Update copy plan with *staging -> final path mappings* for each manifest list file rewritten. > Implement manifest list rewrite logic for path migration > > > Key: HDDS-14943 > URL: https://issues.apache.org/jira/browse/HDDS-14943 > Project: Apache Ozone > Issue Type: Sub-task >Reporter: Sreeja >Assignee: Sreeja >Priority: Major > > Add logic to find the valid snapshots for the table and rewrite manifest > lists associated with those snapshots by Iterating through each valid > snapshot. We can use Iceberg's RewriteTablePathUtil methods to replace > sourcePrefix -> targetPrefix in each and update the rewritten file in staging > location. > We should also update copy plan with staging -> final path mappings for each > manifest list file rewritten. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Updated] (HDDS-14943) Implement manifest list rewrite logic for path migration
[ https://issues.apache.org/jira/browse/HDDS-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreeja updated HDDS-14943: -- Description: Add logic to *rewrite manifest lists for snapshots* by Iterating over valid snapshots and rewrite their associated manifest lists by replacing sourcePrefix -> targetPrefix in each and update in staging location (using iceberg's RewriteTablePathUtil) Update copy plan with *staging -> final path mappings* for each manifest list file rewritten. was: Add logic to *rewrite manifest lists for snapshots* by Iterating over valid snapshots and rewrite their associated manifest lists by replacing sourcePrefix -> targetPrefix in each and update in staging location. Update copy plan with *staging -> final path mappings* for each manifest list file rewritten. > Implement manifest list rewrite logic for path migration > > > Key: HDDS-14943 > URL: https://issues.apache.org/jira/browse/HDDS-14943 > Project: Apache Ozone > Issue Type: Sub-task >Reporter: Sreeja >Assignee: Sreeja >Priority: Major > > Add logic to *rewrite manifest lists for snapshots* by Iterating over valid > snapshots and rewrite their associated manifest lists by replacing > sourcePrefix -> targetPrefix in each and update in staging location (using > iceberg's RewriteTablePathUtil) > Update copy plan with *staging -> final path mappings* for each manifest list > file rewritten. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Updated] (HDDS-14943) Implement manifest list rewrite logic for path migration
[ https://issues.apache.org/jira/browse/HDDS-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreeja updated HDDS-14943: -- Description: Add logic to *rewrite manifest lists for snapshots* by Iterating over valid snapshots and rewrite their associated manifest lists by replacing sourcePrefix -> targetPrefix in each and update in staging location. Update copy plan with *staging -> final path mappings* for each manifest list file rewritten. was: Add logic to *rewrite manifest lists for snapshots* by Iterating over valid snapshots and rewrite their associated manifest lists by replacing sourcePrefix -> targetPrefix in each and update in staging location. Update copy plan with *staging -> final path mappings* for each manifest list file rewritten. Output : Returns a RewriteResult containing: * *Rewritten manifests* -> updated manifest list entries across snapshots * *Copy plan* -> set of (staging path, final path) mappings for manifest lists > Implement manifest list rewrite logic for path migration > > > Key: HDDS-14943 > URL: https://issues.apache.org/jira/browse/HDDS-14943 > Project: Apache Ozone > Issue Type: Sub-task >Reporter: Sreeja >Assignee: Sreeja >Priority: Major > > Add logic to *rewrite manifest lists for snapshots* by Iterating over valid > snapshots and rewrite their associated manifest lists by replacing > sourcePrefix -> targetPrefix in each and update in staging location. > Update copy plan with *staging -> final path mappings* for each manifest list > file rewritten. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Updated] (HDDS-14943) Implement manifest list rewrite logic for path migration
[ https://issues.apache.org/jira/browse/HDDS-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreeja updated HDDS-14943: -- Description: Add logic to *rewrite manifest lists for snapshots* by Iterating over valid snapshots and rewrite their associated manifest lists by replacing sourcePrefix -> targetPrefix in each and update in staging location. Update copy plan with *staging -> final path mappings* for each manifest list file rewritten. Output : Returns a RewriteResult containing: * *Rewritten manifests* -> updated manifest list entries across snapshots * *Copy plan* -> set of (staging path, final path) mappings for manifest lists was: Add logic to *rewrite manifest lists for snapshots* by Iterating over valid snapshots and rewrite their associated manifest lists by replacing sourcePrefix -> targetPrefix in each and update in staging location. Update copy plan with *staging -> final path mappings* for each manifest list file rewritten. > Implement manifest list rewrite logic for path migration > > > Key: HDDS-14943 > URL: https://issues.apache.org/jira/browse/HDDS-14943 > Project: Apache Ozone > Issue Type: Sub-task >Reporter: Sreeja >Assignee: Sreeja >Priority: Major > > Add logic to *rewrite manifest lists for snapshots* by Iterating over valid > snapshots and rewrite their associated manifest lists by replacing > sourcePrefix -> targetPrefix in each and update in staging location. > Update copy plan with *staging -> final path mappings* for each manifest list > file rewritten. > Output : > Returns a RewriteResult containing: > * *Rewritten manifests* -> updated manifest list entries across snapshots > * *Copy plan* -> set of (staging path, final path) mappings for manifest > lists -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
