openinx opened a new pull request, #99: URL: https://github.com/apache/flink-table-store/pull/99
This PR is trying to provide table store the ability to accept append-only ingestion without any defined primary keys. The previous table store abstraction are built on top of primary keys, so in theory all the read & write path will need to be reconsidered or refactored, so that we can abstract the correct API which works fine for both primary keys storage and immutable logs (without primary keys). The current version is a draft PR (Actually, I'm not quite familiar with the flink-table-store project before, so I'm trying to implement this append-only abstraction to understand the API & implementation better). There are TODO issues that I didn't consider clearly in this PRs ( I think I will need the next update to address those things): 1. The append-only table's file level statistics are quite different with the primary key tables. For example, the primary key tables will generate a collection of `SstFileMeta` when calling the `writer#prepareCommit()`, and then accomplish the first stage commit in the flink's two-phrase commit. The `SstFileMeta` will include the statistics for both key fields and value fields, while in the append-only table we don't have any key fields (its statistic information should include all columns' max-min, count etc.) . So in theory, we are required to abstract the common file level statistic informations data structure for both two kinds of table; 2. The different manifests design for both two kinds of tables. 3. What's the read API abstraction for those two kinds of tables. I still don't have a clearly propose for it. Will try to update this PR for this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
