GitHub user MisterRaindrop added a comment to the discussion: [Proposal] Iceberg subsystem for datalake_fdw — design proposal
Status update, and a proposal to move this from "Under Discussion" to "Accepted". Two PRs have landed: #1842, the extension skeleton, and #1951, the Parquet format layer (Arrow round trip, field ids, an Arrow memory pool accounted to the vmem tracker, type rules). Both were reviewed in the open; #1951's review changed two design points for the better — column matching by field id, and memory accounting in that PR rather than later. What remains is tracked in #2008: twelve PRs in three waves. First, in parallel: S3 storage I/O, the gRPC contract, the agent's service framework and the fdw-side engine client. Then the agent's Iceberg operations and the Builtin catalog — the first catalog to work end to end, needing nothing but object storage — followed by the write and read paths, and one PR per external catalog provider: Polaris, Hive Metastore, Hadoop. Merge-on-read and DML last. Each PR has an issue with scope and acceptance criteria; a milestone collects them. Type questions with a real choice left (DECIMAL forms, timestamp units, name mapping) are separate issues, not blockers. Two asks: - The agent is Java and self-contained; the catalog providers (Polaris, Hive Metastore, Hadoop) are each a bounded PR with an end-to-end test to pass. If someone wants one, those are the items — and a Java reviewer for the agent PRs would be welcome. - Unless there are objections in the next week, I will mark this proposal Accepted: the design as written here, with the changes recorded in #1842 and #1951, is what is being built. GitHub link: https://github.com/apache/cloudberry/discussions/1683#discussioncomment-18461167 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
