GitHub user igor-suhorukov added a comment to the discussion: [Proposal] Apache Cloudberry as PostgreSQL 19 Extensions
@MisterRaindrop thank you for initiating a substantive discussion regarding the proposal. The three interfaces vexec uses are not specific to it, and a DataFusion engine could be their second user: [`vexec_source.h`](https://github.com/igor-suhorukov/cloudberry/blob/extension_postgresql_19/pg19/include/vexec_source.h) (PAX and `ao_column` hand over column batches in Arrow layouts), [`gp_orca_vec.h`](https://github.com/igor-suhorukov/cloudberry/blob/extension_postgresql_19/pg19/include/gp_orca_vec.h) (ORCA prices and places an engine's nodes) and [`gp_core_api.h`](https://github.com/igor-suhorukov/cloudberry/blob/extension_postgresql_19/pg19/include/gp_core_api.h) (routing batches across Motions). 1. The 24 upstream core commits fall into three groups. - General: [`new_oid_hook`](https://github.com/igor-suhorukov/postgres/commit/7e2a0be6218), [`raw_parser_hook`](https://github.com/igor-suhorukov/postgres/commit/480f3128f87), [`smgr_file_event_hook`](https://github.com/igor-suhorukov/postgres/commit/a91cbb5aaf0), and a [registry](https://github.com/igor-suhorukov/postgres/commit/21f45bdde60) of extra table access method routines with five uses: [reloptions](https://github.com/igor-suhorukov/postgres/commit/112462d155c), [the columns a scan reads](https://github.com/igor-suhorukov/postgres/commit/3a2d4fcc1c2), [a unique index's probe](https://github.com/igor-suhorukov/postgres/commit/150343312fe), [ADD COLUMN without a rewrite](https://github.com/igor-suhorukov/postgres/commit/93eb6501b97) and [BRIN over sparse block numbers](https://github.com/igor-suhorukov/postgres/commit/5ac8dbfc9ab). - A general problem, but another shape upstream: [`analyze_sample_rows_hook`](https://github.com/igor-suhorukov/postgres/commit/6f8d26d16c2), [`memory_block_alloc_hook`](https://github.com/igor-suhorukov/postgres/commit/41107a43583), [materialized view maintenance](https://github.com/igor-suhorukov/postgres/commit/a903759f70d) and [an injection point](https://github.com/igor-suhorukov/postgres/commit/4026e083a14). - Specific to an MPP engine: a transaction handed to reader processes and the order of its commit ([`XactAdoptTransactionState()`](https://github.com/igor-suhorukov/postgres/commit/60e82c61d7c), [combo command ID hooks](https://github.com/igor-suhorukov/postgres/commit/791fd7171a0), [`xact_commit_recorded_hook`](https://github.com/igor-suhorukov/postgres/commit/1853d9244f0), [`XactAbortAgainAfterEnd`](https://github.com/igor-suhorukov/postgres/commit/871f0c5102e)); [`query_lockmode_hook`](https://github.com/igor-suhorukov/postgres/commit/8c908a24f4c); [`physical_slot_restart_lsn_hook`](https://github.com/igor-suhorukov/postgres/commit/bea8a637438); [tablespace location hooks](https://github.com/igor-suhorukov/postgres/commit/9b351c13e72); [pg_checksums' marks](https://github.com/igor-suhorukov/postgres/commit/b213d194740); and Cloudberry's syntax ([`gp_segment_id`](https://github.com/igor-suhorukov/postgres/commit/f26d74cc863), [`gp_dist_random()`](https://github.com/igor-suhorukov/p ostgres/commit/aeae8a306d2), [columns hidden from `*`](https://github.com/igor-suhorukov/postgres/commit/19b27fd7687)). 2. Following `main`. The port edits no Cloudberry file, so `git merge main` cannot conflict. For the port's 244 modified copies, a new script (not pushed yet) does a three-way merge with the original at the last branch or merge point and the original in `main`. A clean merge is applied automatically; a conflict goes to an LLM and then to a reviewer. On today's `main`, 54 commits ahead, 13 copies were affected: 7 merged automatically, 1 already had the change, and 5 had conflicts, which the LLM resolved. 3. Comparison. Yes, of course I plan to. A benchmark that others can reproduce has to run in a public cloud, and I have no prepaid AWS access. Given access to an AWS account, I will run all of it: Motion-heavy joins, concurrency, spill and failure recovery. I would do it after the port passes as many of Cloudberry's tests as it can and vexec gets a little more work, so that the measurements include it. 4. Branch or repository. I would prefer an experimental branch in the project's main repository. `pg19/` compiles Cloudberry's files in place, so it has to sit inside Cloudberry's tree, and on a branch, following `main` is a plain `git merge main`. And a repository for Postgres kernel 24 hooks is also needed. GitHub link: https://github.com/apache/cloudberry/discussions/2065#discussioncomment-18906162 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
