Hi everyone, I'd like to resume discussion of PIP-419. I've revised the proposal to focus on broker-side subscription message filtering and address the earlier feedback on policy ownership, batches, and acknowledgment state. The PR is open again.
Proposal PR: https://github.com/apache/pulsar/pull/24305 Revised PIP, including diagrams of the dispatch flow, mixed-batch delivery, and policy update barrier: https://github.com/AuroraTwinkle/pulsar/blob/ce99073b0567b45a14f8535ac2e6480becc0a617/pip/pip-419.md The main changes are: - An optional, restricted CEL profile evaluates string message properties. Administrators manage one durable, versioned policy per subscription through the Admin API/CLI. - Existing EntryFilter plugins retain their behavior. An internal evaluator handles per-message decisions in unencrypted batches, preserving original message IDs and using batch-index ACK masks. - Completely rejected entries are not sent. Mixed batches retain their original encoded bytes, with capable clients exposing only the selected indexes to the application. This does not provide payload isolation or proportional bandwidth savings for mixed batches. - Policy updates gate dispatch and durably persist old filter rejection ACKs, including partial-batch masks, before persisting a new policy revision. ACK holes still require bounded cursor state and backpressure. - The draft specifies bounded execution, errors that pause the affected subscription, explicit client capability negotiation, per-partition update outcomes, metrics, and upgrade/rollback procedures. - BookKeeper predicate execution is outside this proposal. Encrypted batches, chunked messages, and replicated subscriptions are excluded from the initial scope. I'd especially appreciate feedback on: 1. Is the restricted CEL profile and administrator-owned policy model appropriate for the initial feature? 2. Is the original-entry plus delivery-mask approach acceptable for mixed batches, given its bandwidth and client compatibility tradeoffs? 3. Are the proposed ACK durability barrier and bounded persistence requirements sufficient for policy changes and broker recovery? 4. Are the initial compatibility scope and non-atomic, per-partition update semantics clear and workable? This is a design discussion. Implementation and performance measurements are pending; the PIP includes an acceptance-test and benchmark plan. Feedback on the proposal or in this thread would be appreciated. Thanks, Aurora Twinkle
