yz271544 opened a new pull request, #1976: URL: https://github.com/apache/cloudberry/pull/1976
Add experimental privacy authorization and output integration hooks <!-- Thank you for your contribution to Apache Cloudberry (Incubating)! --> Related discussion: https://github.com/apache/cloudberry/discussions/1943 ### What does this PR do? This PR introduces the Cloudberry core integration points for an experimental, extension-based output privacy design. It allows a companion extension to apply policy-based transformations or permit authorized access immediately before values are delivered to a client, without modifying executor input tuples. This is the core integration portion, not a complete privacy protection implementation. JWT verification, session authorization, policy management, MASK/CPT transformations, and policy-version validation are implemented in the separate `cloudberry-privacy` project and are not included in this PR. The core changes are: 1. **Authorization command parsing** - Add the unreserved `AUTHORIZE` keyword and parse `AUTHORIZE PRIVACY FROM STDIN`. - Represent the command as a dedicated `CopyStmt` marker for the extension's utility hook. With the companion extension, JWT bytes are received through CopyData rather than embedded in SQL text. - Parsing this command does not itself verify a JWT or grant any privilege. 2. **Final-output hooks** - Add versioned output and endpoint-export interfaces in `utils/privacy_output.h` (experimental ABI version 2). - Invoke the field-output hook in `printtup.c` and `copyto.c` before text or binary encoding. - Distinguish SELECT, RETURNING, COPY, external/callback output, and RETRIEVE operations. - Resolve the original cursor's execution context for FETCH so the extension can check authorization on continuation. 3. **Distributed security labels** - Dispatch SECURITY LABEL statements for the `cloudberry_privacy` provider with snapshot and two-phase transaction flags. - Add the required `SecLabelStmt` binary serialization/deserialization and text-output support. - Leave other label providers' dispatch behavior unchanged. 4. **Parallel RETRIEVE integration** - Pass the planned statement to endpoint setup and let the extension export an output-metadata-only plan. - Store that metadata in separately keyed endpoint shared-memory entries and copy it into the retrieving portal. - Apply the output hook at final delivery, allowing the extension to check RETRIEVE scope, expiry, and policy versions during continuation. - Refuse RETRIEVE when an endpoint contains privacy metadata but no output handler is installed. The companion extension uses the existing `PlannedStmt.extensionContext` transport for QD-to-QE policy fingerprints and scoped capabilities. No new client protocol message is introduced. ### Type of Change - [ ] Bug fix (non-breaking change) - [ ] New feature (non-breaking change) - [x] Breaking change (fix or feature with breaking changes) - [ ] Documentation update This is an experimental feature with internal ABI and endpoint shared-memory compatibility changes. ### Breaking Changes - `PortalData` gains an output-metadata field, and `SetupEndpointExecState()` gains a planned-statement argument. - The endpoint shared-memory layout and magic value change. Mixed old/new endpoint producers and receivers are not supported. - Deploy matching core and extension builds across coordinator, segment, and retrieve backends, rebuild affected binary modules, and restart the cluster together. Rolling mixed-version compatibility is not claimed. - The hook interface is experimental and is not a stable extension ABI. No on-disk table format change is introduced. Without registered handlers, these hooks do not apply privacy transformations; this patch alone does not protect labeled data. ### Test Plan Local validation used Cloudberry 3-devel / PostgreSQL 16.9, based on `867c6a147df`, with this core patch and the companion extension. Tests used disposable installations and a one-coordinator/one-segment topology, not production data. - [ ] Unit tests added/updated in this repository - [ ] Integration tests added/updated in this repository - [x] Companion-extension algorithm, integration, isolation, and regression tests executed locally - [ ] Passed `make installcheck` - [ ] Passed `make -C src/test installcheck-cbdb-parallel` **Passed locally:** - Core and companion-extension builds. - JWT rejection cases, fragmented CopyData reception, session cleanup, and log scans for test JWT/claim/CopyFail payloads. - UTF-8 MASK and 768 CPT compatibility vectors, including round trips and invalid input domains. - Masked and authorized SELECT, direct COPY TO, query COPY TO, binary COPY, RETURNING, and ordinary cursor output. - Rejection of FETCH after clearing the cursor's session authorization. - Policy publication/revision, publication waiting on reader locks, and committed-label cache invalidation. - QD/QE policy-version mismatch rejection and prepared-plan authorization isolation. - Parallel RETRIEVE masking, independent operation scope, authorized continuation, and expired-capability rejection. - The companion extension's `basic` SQL regression through Cloudberry `pg_regress`: 1/1 passed. The full Cloudberry regression and parallel installcheck suites have **not** been run. The test scripts and expected output are in the companion project, not in this core diff; repository-local coverage is still needed for upstream integration. Reproduction from the companion project, with the patched Cloudberry source in an adjacent `cloudberry` directory: ```sh python3 tests/prepare_stage.py # Replace the path with the temporary stage printed by the command above. python3 tests/run_all.py --stage /tmp/cloudberry-privacy-build.REPLACE_ME --benchmark-rows 10000 ``` ### Impact **Performance:** The patch adds per-field hook checks and endpoint metadata serialization/storage. Standalone core overhead has not been isolated. An exploratory companion-extension run fetched 10,000 rows with two fields using a single utility backend. Median complete PQexec times over seven measured runs were: | Scenario | Median | | --- | ---: | | Extension loaded, no policy binding | 13.45 ms | | Two MASK-protected fields | 33.88 ms | | Two fields with authorized raw output | 17.56 ms | | Two CPT-protected fields | 169.28 ms | These are end-to-end prototype measurements, not p95, multi-segment benchmarks, or performance acceptance results. They are affected by cache state, run order, and machine load. The current companion CPT adapter opens and validates its key file for every value; safe caching and key-rotation semantics remain future work. **User-facing changes:** - Adds the authorization command syntax, which requires the companion extension to handle it. - With that extension enabled, supported protected outputs are transformed unless a matching capability permits raw values. - Expired or changed capabilities can interrupt output; clients must treat interrupted results as incomplete. - This does not change stored data or provide encryption at rest. Executor tuples and endpoint queues can still contain plaintext inside the trusted server boundary. **Dependencies:** - This core diff adds no third-party library dependency. - The separately built companion extension uses OpenSSL and contains a CPT compatibility adapter. - Its runtime configuration, policy APIs, and tests are not shipped by this PR. ### Checklist - [ ] Followed [contribution guide](https://cloudberry.apache.org/contribute/code) - [ ] Added/updated documentation in this repository - [x] Reviewed code for security implications during implementation - [x] This PR contains AI-assisted code generation - [ ] Requested review from [cloudberry committers](https://github.com/orgs/apache/teams/cloudberry-committers) The implementation review is not an independent security audit. Companion-project documentation exists, but core documentation and upstream test integration remain follow-up work. ### Additional Context This change is submitted as an experimental implementation for design review, not as a production-ready security boundary. The companion prototype is opt-in and supports a restricted direct-column query subset. General expression provenance, inference resistance, complete view/partition/DDL/restore coverage, durable fail-closed auditing, cluster-wide immediate revocation, protected internal-copy workflows, and fault/failover testing remain incomplete. Protected WITH HOLD cursors and several unsupported query shapes are currently rejected by the companion extension. The new privacy metadata is intended to contain only field origins, policy/version identifiers, and scoped capabilities—not raw JWTs, keys, or protected values. This does not imply that ordinary query plans or internal tuples contain no plaintext. Implementation references: - Core commit: `9a36e58b961` - Companion project: `cloudberry-privacy`, commit `a3c2831` - Companion validation report: `docs/test-results-2026-09-09.md` - Companion test entry points: `tests/prepare_stage.py` and `tests/run_all.py` <!-- Before submission, provide an accessible companion repository/commit link or attach the relevant test artifacts so reviewers can reproduce the results. --> Feedback is particularly welcome on the hook API, the provider-specific SECURITY LABEL dispatch, endpoint metadata ownership/lifetime, and the appropriate split between core and extension responsibilities. ### CI Please run the normal CI jobs for this core change. No CI skip marker is requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
