GitHub user nevzheng created a discussion: Proposal: Adopt OpenAPI as the External Contract for Gravitino API V2
## Two birds with one stone ### Abstract I recommend adopting OpenAPI as the authoritative external contract for Gravitino API V2 and integrating that contract into the server’s development, testing, and release lifecycle. This addresses two related problems: - Our [error-handling strategy](https://github.com/apache/gravitino/discussions/11982), where caller errors, dependency failures, and Gravitino defects frequently collapse into generic HTTP 500 responses. - Our [OpenAPI contract and validation gap](https://github.com/apache/gravitino/issues/11985), where the existing description is not authoritative or reliable enough for code generation and compatibility guarantees. A specification-first V2 would give us a stable external contract, enable non-Java integrations and generated clients, improve error semantics, and support conformance testing, property-based testing, API fuzzing, and automated regression detection. ## Context Hi everyone, Last week I spent considerable time investigating our error-handling strategy and OpenAPI description. I initially approached them as separate problems, but I now believe we can solve them through the same initiative. Gravitino already has useful OpenAPI and Java DTO work to build upon. However, we do not yet treat OpenAPI as the authoritative external contract or continuously verify that the running server conforms to it. My recommended approach is: > Use OpenAPI as the authoritative external contract for Gravitino API V2 and > integrate it directly into how we design, implement, test, and release the > server. ## A specification-first V2 V2 should be designed specification-first: 1. Define and review the external behavior in OpenAPI. 2. Implement or modify the server to conform. 3. Generate clients from the specification. 4. Validate the running server against the contract. 5. Detect regressions and breaking changes continuously. By “V2,” I mean the next trustworthy and reliable version of the Gravitino external API—not OpenAPI Specification 2.0. The V2 contract should use an appropriate current OpenAPI 3.x version and follow established API design practices. V2 should be a clean external contract. This gives us room to improve: - Resource and operation design. - Request and response representations. - HTTP status semantics. - Authentication and authorization behavior. - Pagination and filtering. - Idempotency and retry behavior. - Public error responses. - Naming and compatibility conventions. V1 and V2 can coexist while V2 is developed. We can publish the implemented V2 surface as beta, allowing users to generate clients, build integrations, and give us feedback before the API is declared stable. ## OpenAPI should define the external boundary Our Java DTOs remain useful internal and Java-client models. I am not proposing that we remove them. I recommend that we stop treating Java DTO shapes as the definition of the external wire protocol. The OpenAPI contract should independently define: - Resources and operations. - Wire representations. - Required and optional fields. - Validation constraints and nullability. - HTTP statuses, media types, and headers. - Authentication requirements. - Public error semantics. The server can use a translation layer between the OpenAPI-defined external representations and its internal DTO or domain models. This allows internal Java code to evolve without automatically changing the public protocol. It also prevents Java-specific types, exception classes, inheritance, and serialization choices from leaking into integrations written in other languages. > Java DTOs may implement the API, but they should not define it. ## Error handling as part of the contract Error handling is observable API behavior, not merely an implementation detail. Clients need to distinguish: - Invalid requests. - Authentication and authorization failures. - Missing or conflicting resources. - Unavailable or failing dependencies. - Timeouts and retryable conditions. - Genuine Gravitino defects. V2 should define stable HTTP semantics and a consistent public error model. That model should provide: - Stable, machine-readable error reasons. - Correct HTTP statuses. - Failure-domain attribution. - Explicit retry semantics. - Safe, actionable public messages. - Request identifiers for correlation with internal diagnostics. Stack traces and internal causes should remain in server logs rather than becoming part of the public protocol. The error taxonomy defines what failures mean. OpenAPI makes those semantics public and machine-verifiable. Server-side classification preserves the meaning from the catalog adapter to the HTTP response, and conformance testing verifies that the implementation follows the contract. OpenAPI will therefore become an important pillar of our effort to eliminate undifferentiated HTTP 500 responses. ## Why this matters ### Adoption and interoperability Much of the modern data ecosystem is not written in Java. [Apache DataFusion](https://datafusion.apache.org/), for example, is written in Rust. Other consumers use Python, Go, TypeScript, C++, and languages we may not support directly. These users cannot easily reuse our Java DTOs. Without a reliable language-neutral contract, every integration must reverse-engineer payloads, maintain handwritten models, or infer behavior from server implementation details. A code-generation-clean OpenAPI contract allows users to: - Generate typed clients in their preferred language. - Use standard mock servers and validators. - Build integrations without studying Java source code. - Validate compatibility before deployment. - Depend on documented error and retry behavior. This reduces integration friction and expands Gravitino’s potential ecosystem beyond the JVM. For initial validation, I propose generating Java, Rust, and Python clients. The precise long-term client-support strategy can evolve as we learn from real usage. ### Trust through verifiable behavior A specification alone is a promise. A specification backed by automated conformance, generated-client checks, compatibility analysis, and regression testing provides evidence that the implementation honors that promise. Users and downstream projects should be able to verify that: - Requests and responses have stable semantics. - The running server behaves as documented. - Failures are predictable and actionable. - Compatibility is checked before release. - Regressions are detected before deployment. This establishes confidence through measurable behavior rather than documentation alone. ### Reliability, security, and maintainability A stable contract enables: - Safe retry and backoff policies. - Circuit breakers and correct alert routing. - Useful operational runbooks. - Better attribution between Gravitino and its dependencies. - Consistent authentication and authorization tests. - Automated checks against internal information leakage. - Safer API evolution. - Faster diagnosis of integration and production failures. It also gives contributors a clear target. Server and client development can proceed against an agreed contract, while CI detects drift before it reaches users. ## An executable contract The V2 OpenAPI specification should be part of our testing and release infrastructure, not only a source for rendered documentation. A practical pipeline can include: 1. Lint and bundle the specification. 2. Generate and compile representative clients. 3. Run the server and validate its responses against the contract. 4. Execute representative end-to-end customer journeys. 5. Run property-based and schema-guided fuzz tests. 6. Detect and review breaking changes. End-to-end tests cover known workflows, but they can exercise only scenarios we write manually. Tools such as [Schemathesis](https://schemathesis.readthedocs.io/en/stable/) can generate valid, invalid, boundary, and adversarial requests directly from OpenAPI. This lets us test properties such as: - Malformed input does not produce an unexpected 500. - Responses match their documented schemas. - Only documented statuses and media types are returned. - Boundary values behave consistently. - Authentication failures do not expose internal details. - Equivalent failures produce consistent error semantics. CI can begin in advisory mode while we expose and resolve existing issues. As the specification and server reach conformance, we can progressively make the checks blocking. ## Incremental adoption This should be implemented incrementally, use case by use case. I propose beginning with catalog and table workflows because they represent central Gravitino behavior and exercise meaningful success, validation, dependency, and error cases. A practical sequence is: 1. Establish the V2 OpenAPI structure and design rules. 2. Define the shared success and error representations. 3. Specify the catalog and table use cases. 4. Implement the external-to-internal translation layer. 5. Generate initial Java, Rust, and Python clients. 6. Add conformance, property, and customer-journey tests. 7. Publish the implemented surface as V2 beta. 8. Gather feedback and expand V2 incrementally. 9. Promote CI checks from advisory to blocking as V2 matures. 10. Declare V2 stable when its supported surface consistently passes the contract gate. This approach lets users engage early while allowing the API to grow through concrete, tested use cases. ## Future proposal: Antithesis One future direction I am particularly excited about is integrating [Antithesis](https://antithesis.com/docs/) into the Gravitino testing stack. A formal V2 contract would give us explicit properties and invariants to test under dependency failures, network faults, concurrency, retries, and unusual operation sequences. Antithesis could exercise the complete system in a deterministic simulation environment and make discovered failures reproducible. This is not required to begin V2. I intend to develop it as a separate future proposal once the contract and initial conformance infrastructure are established. ## Feedback requested from maintainers I would appreciate feedback on this recommended direction: 1. Adopt OpenAPI as the authoritative external contract for V2. 2. Design V2 specification-first as a clean external API. 3. Keep Java DTOs internally, with translation at the external boundary. 4. Make the public error contract part of the initial V2 work. 5. Begin incrementally with catalog and table use cases. 6. Publish the implemented V2 surface as beta for early feedback. 7. Introduce CI checks progressively, moving from advisory to blocking. 8. Use generated Java, Rust, and Python clients as initial contract validation. If there are viable alternatives to OpenAPI that better serve these goals, I would like to understand and compare them. ## References - [OpenAPI Specification](https://spec.openapis.org/oas/v3.0.4.html) - [Zalando RESTful API Guidelines](https://opensource.zalando.com/restful-api-guidelines/) - [Microsoft Azure REST API Guidelines](https://github.com/microsoft/api-guidelines/blob/vNext/azure/Guidelines.md) - [Google AIP-193: Errors](https://google.aip.dev/193) - [Google AIP-194: Automatic retries](https://google.aip.dev/194) - [RFC 9110: HTTP Semantics](https://www.rfc-editor.org/rfc/rfc9110.html) - [Schemathesis](https://schemathesis.readthedocs.io/en/stable/) - [OWASP API Security Top 10](https://owasp.org/API-Security/editions/2023/en/0x00-header/) GitHub link: https://github.com/apache/gravitino/discussions/11999 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
