This is great stuff, Viquar! ... I'd be happy to Champion this effort through the Incubator. I already have a need to build an MCP from scratch/zero, and would like to use exactly this kind of framework to build that.
We have a few existing MCP servers at the Foundation today, which might be able to leverage this framework, too. I'll poke some people to get some additional Mentors. My background: 28 years at Apache, current Director, past Chairman, initial Incubator member, current Apache RAI participant, past Infra, Subversion, HTTPD, and other projects. Cheers, -g On Tue, Aug 4, 2026 at 12:18 AM vaquar khan <[email protected]> wrote: > Hello everyone, > I would like to propose a project called Aegis MCP Governance Gateway for > the Apache Incubator, and start a discussion. It is a working reference > implementation, and I am bringing it here mainly because it needs a > community rather than a single author, which I state plainly in the > proposal. > *In short: *when an AI agent needs to operate data systems like Flink, > Kafka, Spark, or Iceberg, it usually drives each engine control plane > directly or connects to a pile of separate MCP servers, each with its own > security posture, no shared audit, and no common policy. An MCP server that > can reach a control plane is a privileged client, so shipping one safely > means shipping an access-control product. Aegis puts one governed control > plane in front of all of those engines, so every agent action runs through > a single fail-closed chain of identity, scope, policy, approval for > destructive actions, egress and request-forgery defense, rate limiting, > output redaction, and a tamper-evident audit trail. Engines plug in through > a small service provider interface and contribute no policy. > Current state, stated honestly. There is a working Java 21 build: a gateway > core, four in-depth adapters (Flink, Kafka, Spark, Iceberg), thinner > adapters for other Apache systems, and a runnable distribution. It builds > cleanly with 309 tests passing across all modules. A published > design-conformance matrix maps every design claim to Done or Roadmap. > Several capabilities ship as roadmap items and are not sold as done: > externalized high-availability state, CIMD and SPIFFE identity, a full > Cedar runtime, vault-backed credentials, and an OpenTelemetry SDK bridge. > Parts of the code were developed with AI-assisted tools under my direction > and review, which I disclose up front for IP clearance. > *Biggest risk: *The project is currently a single contributor. Growing a > diverse, multi-organization community is the main reason I am bringing it > here. I am currently in active conversations with engineers in the > ecosystem and expect to add a few of them as initial committers in the next > few days, prior to a formal vote > What I am looking for now: a Champion (an ASF Member) and mentors (three or > more IPMC members), ideally including people from the Flink, Kafka, Spark, > or Iceberg communities. The full proposal follows, and the reference > implementation is at https://github.com/vaquarkhan/aegis-mcp-gateway. I > welcome feedback on scope, name (a name search is still to be run), risks, > and anything I have missed. > Thank you, > Viquar Khan > https://www.linkedin.com/in/vaquar-khan-b695577/ > > Apache Incubator Proposal: Aegis MCP Governance Gateway > This is the proposal for entry into the Apache Incubator, written for > discussion on the Incubator general list. Aegis is a working codename; a > name search will be run and the ASF trademark process followed if the > project is accepted. The document follows the ASF Incubator proposal > template and states the maturity of the code honestly, including what is > implemented and what is deferred to the roadmap. > Field > Value > Proposal name > Aegis MCP Governance Gateway (codename) > Proposed short name > aegis (subject to name search) > Proposal date > to be set at submission > Sponsor > Apache Incubator PMC > Champion > to be identified (ASF Member) > Mentors > to be recruited (three or more IPMC members) > Initial code > Java 21 multi-module reference implementation, 36 Maven modules, 309 tests > passing > Reference repository > https://github.com/vaquarkhan/aegis-mcp-gateway (private for now, access > on > request) > License > Apache License 2.0 > Discussion list > general at incubator.apache.org > Abstract > Aegis MCP Governance Gateway is a vendor-neutral, secure-by-default Model > Context Protocol (MCP) gateway that puts one governed, auditable control > plane in front of many Apache data and streaming engines, so an AI agent > never talks to Flink, Kafka, Spark, or Iceberg directly. Every agent action > passes one fail-closed governance chain: secure-by-default tool exposure, > per-caller authentication and scope, a policy decision point, approval for > destructive operations, egress and server-side request forgery defense, > rate limiting, circuit breaking, prompt-injection screening, output > bounding and redaction, tool-catalog integrity, and a tamper-evident > hash-chained audit trail. Engines plug in through a small service provider > interface and contribute only tool definitions and backend calls. > Proposal > We propose to develop, under Apache governance, a Layer 7 policy MCP > gateway plus a family of engine adapters. A proxy forwards bytes and a > router dispatches by capability, but neither is a trust boundary. A gateway > is: it knows the tool, the principal, the budget, and the policy for every > call, and it can deny, redact, gate, and record. The gateway core is > stateless at the protocol edge so it can scale behind an ordinary load > balancer once shared governance state is externalized, and it is designed > to align with OAuth 2.1, SPIFFE, SLSA, OpenTelemetry, and the NIST AI Risk > Management Framework and ISO/IEC 42001. > The initial contribution is a working Java reference implementation: a > reusable gateway core, four in-depth engine adapters (Flink, Kafka, Spark, > Iceberg), and a further set of thinner adapters for the wider Apache data > platform. During incubation the priorities are community building, IP > clearance, and promoting the roadmap items to full implementations. > Background > The Model Context Protocol has become a common way for AI agents to reach > tools and data. As organizations connect agents to distributed Apache data > estates, an agent that needs Flink, Kafka, Spark, and Iceberg must either > drive each engine control plane directly or connect to a sprawl of > independent, mostly read-only MCP servers, each with its own security > posture, no shared audit, and no unified policy. > An MCP server that can reach a control plane (a JobManager REST API, a > Kafka AdminClient, a SQL gateway, an Iceberg catalog) is a privileged > client. Shipping one safely is effectively shipping an access-control > product: authentication, authorization, approval for destructive actions, > redaction of secrets and personal data, rate and cost limits, and forensic > audit. Many existing options lack these controls and carry > single-maintainer sustainability risk. A single governed gateway, developed > in the open under neutral governance, gives the ecosystem a shared standard > for how agents operate Apache systems safely. > Rationale > A single governed gateway resolves the problems of the decentralized > approach: > > - Security. One place enforces identity, authorization, approval, > redaction, egress, and audit, instead of each server reimplementing or > omitting them. > - Maintainability. Adding an engine is a small adapter, not a new > security model. There is one chain to review and harden, so the security > review surface stays flat as engines are added. > - Governance. One control plane for policy, budget, and provenance, with > a stable deny taxonomy so operators can alert on codes such as > POLICY_DENIED or APPROVAL_REQUIRED across every engine with one rule. > > Doing this under Apache governance keeps it neutral across vendors and > engines rather than leaving agent access to fragmented single-vendor > servers. > Initial goals > > 1. Establish the project and practice the Apache Way: transparency, > consensus, meritocracy, and community over code. > 2. Complete IP clearance for the existing code and land it under > Apache-2.0 with correct LICENSE and NOTICE files. > 3. Publish and stabilize the adapter service provider interface so > per-engine adapters are small, reviewable contributions. > 4. Promote the roadmap items to full implementations: externalized > high-availability state, CIMD and SPIFFE identity, a full Cedar runtime, > vault-backed credential exchange, and an OpenTelemetry SDK bridge. > 5. Produce the first Apache source releases and grow a diverse, > multi-organization contributor base. > > Current status > Aegis is a new codebase with a working multi-module reference > implementation. It is not yet a community. The design, threat model, > low-level design, and standards research are complete and public. The code > builds green and is tested. A published design-conformance matrix maps > every design claim to Done or Roadmap, so the maturity line is explicit. > Verified build and test state > > - 36 Maven modules: a gateway core, adapters for 33 Apache engines (four > in-depth today, Flink, Kafka, Spark, and Iceberg, and 29 thinner > integrations covering Cassandra, HBase, Hive, Pulsar, Airflow, NiFi, > Druid, > Pinot, Doris, Impala, Calcite, Hudi, Paimon, Kudu, Ignite, BookKeeper, > Ozone, Hadoop, ZooKeeper, Solr, Superset, Arrow Flight, Beam, Storm, > Flume, > ActiveMQ, CouchDB, Ranger, and Atlas), and a shaded runnable > distribution. > These are engine adapters the gateway supports, which is distinct from > external adopters. > - Builds with JDK 21 and runs on JDK 17 or newer with Maven 3.9 or > newer. > - 309 unit and integration tests across all modules, all passing, zero > failures and zero errors on a clean build. > > Implemented in 0.1.0 > > - The governance chain with ordered steps and first-denial-wins > semantics. > - Identity: shared bearer and a hashed multi-caller token file with > per-caller scope, and an OAuth 2.1 resource-server mode with JWKS > validation. Constant-time secret comparison and fail-closed startup > validation. > - Approvals: HMAC-SHA256 signed, TTL-bounded, single-use, > tool-and-scope-bound tokens with replay rejection. > - Policy: a builtin decision point, a live fail-closed OPA HTTP decision > point, and a Cedar-style deny-file or HTTP delegate. > - Per-caller outbound credential pass-through so a downstream engine can > see a per-caller Authorization value when one was bound at admission. > - Egress and server-side request forgery defense: allow list plus > unconditional deny of cloud metadata and non-routable ranges, including > numeric IP encodings and resolved addresses, with redirects disabled on > outbound clients. > - Rate limiting, circuit breaking, per-tool timeout, read-only SQL > guard, identifier validation, output bounding, and data-loss-prevention > redaction. > - Tool-catalog integrity with durable pins and a fail-closed switch, a > tamper-evident hash-chained audit, Prometheus metrics, and GenAI-shaped > span log lines. > - Two transports: stdio and authenticated Streamable HTTP with TLS, plus > health, readiness, and metrics endpoints. A CycloneDX software bill of > materials is produced at build time. > > Deferred to the roadmap (honestly labeled, not sold as implemented) > > - CIMD and SPIFFE authentication modes (they refuse to start today). > - A full Cedar policy runtime, and vault-backed or token-exchange > credential resolution. > - Externalized high-availability state (shared nonce, rate, and audit > stores). Until then, run a single replica; multi-replica replay > protection > and rate limits require the shared store. > - The MCP Tasks extension for long-running jobs, an OpenTelemetry SDK > bridge, a full Merkle and signature-based reconciliation proof, and true > semantic routing and caching. > > These are tracked in the roadmap and the design-conformance matrix that > ship with the code. > Meritocracy, community, core developers, alignment > The project will operate meritocratically from day one, with technical > decisions on the dev list and committership earned through contribution. > The community is currently centered on the initial author, Viquar Khan, who > has around 23 years in software, has worked in the open since 2013, and has > contributed to Apache Spark, Apache Iceberg, Apache Kafka, and other > open-source projects. The author also maintains MCP-related open-source > packages that have together been downloaded on the order of half a million > times, so there is an existing user base to draw contributors from. Growing > a committer base across the Flink, Kafka, Spark, and Iceberg communities > and across employers is the primary goal of incubation. Aegis consumes > public engine interfaces only and does not fork or modify any engine core. > Known risks > Orphaned products > Low to moderate. Agentic access to data systems is a growing need and the > governance gap is real, so demand is unlikely to vanish. The initial author > has maintained open-source projects since 2013 rather than abandoning them, > and the MCP-related packages already have a real user base, which lowers > the abandonment risk. Mitigation: build a multi-organization community, > keep the core small and standards aligned, and keep adapters independent so > the project survives the departure of any single contributor. > Inexperience with open source > Low. This is not the situation here. The initial author has about 23 years > in software, has worked in open source since 2013, and has contributed to > Apache Spark, Apache Iceberg, Apache Kafka, and other projects, so the > norms of public development, code review, and releases are familiar. > Mentors will still help with the ASF-specific mechanics of voting, board > reports, and IP clearance, which are process details rather than a gap in > open-source experience. > Homogenous developers > This is the real risk and we state it plainly: the podling has one > committer today. However, we are in active discussions with potential > co-committers and expect to formally add them to this proposal in the > coming days. What further reduces this risk is that the author is already > embedded in the exact communities this project draws from (Spark, Iceberg, > Kafka) and has an existing user base for the MCP packages, so there is a > concrete pool to recruit from rather than a cold start. Mitigation: recruit > committers from more than one organization before graduation, shape the > adapter contract so per-engine contributions are natural, and use > good-first-issue and adapter-sized work to onboard newcomers. Growing the > committer base is the main reason we are here. > Reliance on salaried developers > Low current reliance; the work to date is largely independent. The project > will attract contributors with varied motivations and employers so no > single company can stall it. > Relationships with other Apache products > Aegis consumes the public interfaces of Flink, Kafka, Spark, Iceberg, and > other Apache data systems and does not fork or modify them. We will > coordinate on the relevant dev lists and will not duplicate or compete with > an engine that ships its own MCP surface; where an engine has one, Aegis > composes with it. Aegis is not proposed for any engine core. > Excessive fascination with the Apache brand > The project seeks Apache governance for its community model and neutrality, > not for the brand. Until and unless accepted, it uses a neutral name and > does not use the Apache marks or the feather. The breadth of engine > adapters reflects a common governance surface, not an attempt to claim the > ecosystem; only four adapters are in-depth today and the rest are > explicitly thin. > Documentation > Public documentation exists and will be maintained in the repository: an > end-to-end design with architecture decision records, a low-level design > with architecture, sequence, and state diagrams, a threat and governance > model, a design-conformance matrix, standards research, and operations and > adapter guides. > Initial source > The initial source is the Java multi-module gateway described above, > currently in a private repository at > https://github.com/vaquarkhan/aegis-mcp-gateway under the initial author. > Access is available on request to anyone reviewing this proposal, and the > author will make it public before the vote if the community prefers to read > the code first. It includes the gateway core, the four in-depth adapters, > the thinner adapters, the shaded distribution, runnable examples, and the > documentation set. On acceptance it will be donated under a Software Grant > and made public. > Source and intellectual property submission plan > The initial author will submit the code under a Software Grant Agreement > and complete IP clearance early in incubation, with ICLAs from every > committer. Every source file will carry the standard Apache-2.0 header with > correct LICENSE and NOTICE files; any premature ASF-ownership wording will > be corrected to the standard header at donation time. > Transparency on tooling: parts of the codebase were developed with > AI-assisted coding tools under the author's direction and review. The > author asserts authorship and the right to contribute the work, and will > confirm provenance and authorship during IP clearance so the grant is > clean. We raise this proactively so the IPMC can review it rather than > discover it. > External dependencies > Runtime dependencies are Apache-2.0 or compatible, including the MCP Java > SDK, a servlet container for the HTTP transport, SLF4J, Logback, and Nimbus > for JOSE and JWT, with JUnit for tests. Per-engine adapters add that engine > to the official client, for example the Kafka clients library. A full > dependency review and a CycloneDX software bill of materials are produced. > No GPL or ASF category X dependency is introduced into the core. > Cryptography > Aegis uses standard, library-based cryptography: HMAC-SHA256 for approval > tokens, TLS for the HTTP transport, and JSON Web Token validation for > OAuth. Signature-based integrity is an optional gate. The project will > comply with ASF export notification requirements. > Required resources > Mailing lists > > - [email protected] > - [email protected] > - [email protected] > > Source control > > - A Git repository under the ASF (gitbox with a GitHub mirror). > > Issue tracking > > - ASF Jira or GitHub issues per current ASF practice. > > Other > > - Website and wiki space, and continuous integration through ASF > infrastructure. > > Initial committers > > - Viquar Khan *(Note: We are in active conversations with potential > contributors and expect to add a few additional initial committers here > in > the next few days before moving to a formal vote.)* > > - To be recruited before and during incubation: committers from the > Flink, Kafka, Spark, and Iceberg communities and from additional > organizations. Adapter-sized work is designed to make these > contributions > natural. > > Affiliations > To be completed as committers are confirmed. Diversity of affiliation is an > explicit goal; the current single-affiliation status is the main risk being > addressed. > Sponsors > Champion > To be identified. An ASF Member Champion will help finalize this proposal, > recruit mentors, and shepherd the discussion and vote on the Incubator > general list. > Nominated mentors > Three or more IPMC members to be recruited, ideally including participants > from the Flink, Kafka, and Iceberg communities. > Sponsoring entity > The Apache Incubator PMC. > Roadmap during incubation > Phase > Focus > Bootstrap > IP clearance, name search, infra, initial committers, first dev-list > decisions > 0.2 > Externalized HA state, vault-backed credentials, OAuth multi-issuer, OTel > SDK bridge, supply-chain CI (enforcer, CVE scan, signing) > 0.3 > CIMD and SPIFFE identity, full Cedar runtime, MCP Tasks, signature-based > reconciliation proof, semantic routing and caching > Graduation readiness > Diverse community, cadence of Apache releases, healthy governance > References > > - Apache Incubator proposal guide: > https://incubator.apache.org/guides/proposal.html > - Apache Incubator Cookbook: https://incubator.apache.org/cookbook/ > - The Apache Way: https://www.apache.org/theapacheway/ > - Model Context Protocol: https://modelcontextprotocol.io/ >
