[
https://issues.apache.org/jira/browse/HUDI-7882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan updated HUDI-7882:
--------------------------------------
Description:
We have 4 major goals w/ this umbrella ticket.
a. 1.x reader should be capable of reading any of 0.14.x to 0.16.x tables for
all query types.
b. 0.16.x should be capable of reading 1.x tables for most features
c. Upgrade 0.16.x to 1.x
d. Downgrade 1.x to 0.16.0.
We wanted to support reading 1.x tables in 0.16.0 release. So, creating this
umbrella ticket to track all of them.
RFC in progress: [https://github.com/apache/hudi/pull/11514]
Changes required to be ported:
0. Creating 0.16.0 branch
0.a https://issues.apache.org/jira/browse/HUDI-7860 Completed.
1. Timeline
1.a Hoodie instant parsing should be able to read 1.x instants.
https://issues.apache.org/jira/browse/HUDI-7883 Sagar.
1.b Commit metadata parsing is able to handle both json and avro formats. Scope
might be non-trivial. https://issues.apache.org/jira/browse/HUDI-7866 Siva.
1.c HoodieDefaultTimeline able to read both timelines based on table version.
https://issues.apache.org/jira/browse/HUDI-7884 Siva.
1.d Reading LSM timeline using 0.16.0
https://issues.apache.org/jira/browse/HUDI-7890 Siva.
1.e Ensure 1.0 MDT timeline is readable by 0.16 - HUDI-7901
2. Table property changes
2.a Table property changes https://issues.apache.org/jira/browse/HUDI-7885
https://issues.apache.org/jira/browse/HUDI-7865 LJ
3. MDT table changes
3.a record positions to RLI https://issues.apache.org/jira/browse/HUDI-7877 LJ
3.b MDT payload schema changes. https://issues.apache.org/jira/browse/HUDI-7886
LJ
4. Log format changes
4.a All metadata header types porting
https://issues.apache.org/jira/browse/HUDI-7887 Jon
4.b Meaningful error for incompatible features from 1.x
https://issues.apache.org/jira/browse/HUDI-7888 Jon
5. Log file slice or grouping detection compatibility
5. Tests
5.a Tests to validate that 1.x tables can be read w/ 0.16.0
https://issues.apache.org/jira/browse/HUDI-7896 Siva and Sagar.
6 Doc changes
6.a Call out unsupported features in 0.16.0 reader when reading 1.x tables.
https://issues.apache.org/jira/browse/HUDI-7889
was:
We wanted to support reading 1.x tables in 0.16.0 release. So, creating this
umbrella ticket to track all of them.
RFC in progress: [https://github.com/apache/hudi/pull/11514]
Changes required to be ported:
0. Creating 0.16.0 branch
0.a https://issues.apache.org/jira/browse/HUDI-7860 Completed.
1. Timeline
1.a Hoodie instant parsing should be able to read 1.x instants.
https://issues.apache.org/jira/browse/HUDI-7883 Sagar.
1.b Commit metadata parsing is able to handle both json and avro formats. Scope
might be non-trivial. https://issues.apache.org/jira/browse/HUDI-7866 Siva.
1.c HoodieDefaultTimeline able to read both timelines based on table version.
https://issues.apache.org/jira/browse/HUDI-7884 Siva.
1.d Reading LSM timeline using 0.16.0
https://issues.apache.org/jira/browse/HUDI-7890 Siva.
1.e Ensure 1.0 MDT timeline is readable by 0.16 - HUDI-7901
2. Table property changes
2.a Table property changes https://issues.apache.org/jira/browse/HUDI-7885
https://issues.apache.org/jira/browse/HUDI-7865 LJ
3. MDT table changes
3.a record positions to RLI https://issues.apache.org/jira/browse/HUDI-7877 LJ
3.b MDT payload schema changes. https://issues.apache.org/jira/browse/HUDI-7886
LJ
4. Log format changes
4.a All metadata header types porting
https://issues.apache.org/jira/browse/HUDI-7887 Jon
4.b Meaningful error for incompatible features from 1.x
https://issues.apache.org/jira/browse/HUDI-7888 Jon
5. Log file slice or grouping detection compatibility
5. Tests
5.a Tests to validate that 1.x tables can be read w/ 0.16.0
https://issues.apache.org/jira/browse/HUDI-7896 Siva and Sagar.
6 Doc changes
6.a Call out unsupported features in 0.16.0 reader when reading 1.x tables.
https://issues.apache.org/jira/browse/HUDI-7889
> Umbrella ticket for 1.x tables and 0.16.x compatibility
> -------------------------------------------------------
>
> Key: HUDI-7882
> URL: https://issues.apache.org/jira/browse/HUDI-7882
> Project: Apache Hudi
> Issue Type: Improvement
> Components: reader-core
> Reporter: sivabalan narayanan
> Assignee: sivabalan narayanan
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.16.0, 1.0.0
>
>
> We have 4 major goals w/ this umbrella ticket.
> a. 1.x reader should be capable of reading any of 0.14.x to 0.16.x tables for
> all query types.
> b. 0.16.x should be capable of reading 1.x tables for most features
> c. Upgrade 0.16.x to 1.x
> d. Downgrade 1.x to 0.16.0.
>
>
> We wanted to support reading 1.x tables in 0.16.0 release. So, creating this
> umbrella ticket to track all of them.
>
> RFC in progress: [https://github.com/apache/hudi/pull/11514]
>
> Changes required to be ported:
> 0. Creating 0.16.0 branch
> 0.a https://issues.apache.org/jira/browse/HUDI-7860 Completed.
>
> 1. Timeline
> 1.a Hoodie instant parsing should be able to read 1.x instants.
> https://issues.apache.org/jira/browse/HUDI-7883 Sagar.
> 1.b Commit metadata parsing is able to handle both json and avro formats.
> Scope might be non-trivial. https://issues.apache.org/jira/browse/HUDI-7866
> Siva.
> 1.c HoodieDefaultTimeline able to read both timelines based on table version.
> https://issues.apache.org/jira/browse/HUDI-7884 Siva.
> 1.d Reading LSM timeline using 0.16.0
> https://issues.apache.org/jira/browse/HUDI-7890 Siva.
> 1.e Ensure 1.0 MDT timeline is readable by 0.16 - HUDI-7901
>
> 2. Table property changes
> 2.a Table property changes https://issues.apache.org/jira/browse/HUDI-7885
> https://issues.apache.org/jira/browse/HUDI-7865 LJ
>
> 3. MDT table changes
> 3.a record positions to RLI https://issues.apache.org/jira/browse/HUDI-7877 LJ
> 3.b MDT payload schema changes.
> https://issues.apache.org/jira/browse/HUDI-7886 LJ
>
> 4. Log format changes
> 4.a All metadata header types porting
> https://issues.apache.org/jira/browse/HUDI-7887 Jon
> 4.b Meaningful error for incompatible features from 1.x
> https://issues.apache.org/jira/browse/HUDI-7888 Jon
>
> 5. Log file slice or grouping detection compatibility
>
> 5. Tests
> 5.a Tests to validate that 1.x tables can be read w/ 0.16.0
> https://issues.apache.org/jira/browse/HUDI-7896 Siva and Sagar.
>
> 6 Doc changes
> 6.a Call out unsupported features in 0.16.0 reader when reading 1.x tables.
> https://issues.apache.org/jira/browse/HUDI-7889
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
