This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new e321ce6652 Update api endpoint documentation for contributors (#42216)
e321ce6652 is described below
commit e321ce6652eaed6dd077f1ee4088b8dd728c81ed
Author: Pierre Jeambrun <[email protected]>
AuthorDate: Mon Sep 16 23:22:23 2024 +0800
Update api endpoint documentation for contributors (#42216)
---
contributing-docs/17_adding_api_endpoints.rst | 147 +++++++++++---------------
docs/spelling_wordlist.txt | 1 +
2 files changed, 60 insertions(+), 88 deletions(-)
diff --git a/contributing-docs/17_adding_api_endpoints.rst
b/contributing-docs/17_adding_api_endpoints.rst
index a146294740..2a9ef9af7b 100644
--- a/contributing-docs/17_adding_api_endpoints.rst
+++ b/contributing-docs/17_adding_api_endpoints.rst
@@ -18,77 +18,63 @@
Adding a New API Endpoint in Apache Airflow
===========================================
-This documentation outlines the steps required to add a new API endpoint in
Apache Airflow. It includes defining the endpoint in the OpenAPI specification,
implementing the logic, running pre-commit checks, and documenting the changes.
+This documentation outlines the steps required to add a new API endpoint in
Apache Airflow. It includes implementing the logic, running pre-commit checks,
and documenting the changes.
**The outline for this document in GitHub is available at top-right corner
button (with 3-dots and 3 lines).**
-Step 1: Define the Endpoint in ``v1.yaml``
-----------------------------------------
-1. Navigate to the ``v1.yaml`` file, which contains the OpenAPI specifications.
-2. Add a new path for your endpoint, specifying the URL path, HTTP method
(GET, POST, etc.), and a brief summary.
-3. Define the parameters required for the endpoint, including types,
required/optional status, and default values.
-4. Describe the responses, including status codes, content types, and schema
details.
-Example:
+Introduction
+------------
+
+The source code for the RestAPI is located under ``api_fastapi``. Endpoints
are located under ``api_fastapi/views`` and contains different types of
endpoints. The main two are ``public`` and ``ui``.
+Public endpoints are part of the public API, standardized, well documented and
most importantly backward compatible. UI endpoints are custom endpoints made
for the frontend that do not respect backward compatibility i.e they can be
adapted at any time for UI needs.
+When adding an endpoint you should try as much as possible to make it reusable
by the community, have a stable design in mind, standardized and therefore part
of the public API. If this is not possible because the data types are too
specific or subject to frequent change
+then adding it to the UI endpoints is more suitable.
+
-.. code-block:: yaml
-
- paths:
- /example/endpoint:
- get:
- summary: Example API endpoint
- description: This endpoint provides an example response.
- parameters:
- - name: example_param
- in: query
- required: true
- schema:
- type: string
- responses:
- "200":
- description: Successful response
- content:
- application/json:
- schema:
- type: object
- properties:
- message:
- type: string
- "400":
- $ref: "#/components/responses/BadRequest"
- "404":
- $ref: "#/components/responses/NotFound"
-
-
-Step 2: Implement the Endpoint Logic
+Step 1: Implement the Endpoint Logic
------------------------------------
-1. In the appropriate Python file, implement the endpoint's logic.
-2. Ensure proper parameter handling and validation.
-3. Implement the core logic, such as data retrieval or processing.
-4. Add error handling for potential issues like missing parameters or invalid
data.
+1. Considering the details above decide whether your endpoints should be part
of the ``public`` or ``ui`` interface.
+1. Navigate to the appropriate views directory in ``api_fastapi/views``.
+2. Register a new route for your endpoint with the appropriate HTTP method,
query params, permissions, body type, etc.
+3. Specify the appropriate Pydantic type in the return type annotation.
Example:
.. code-block:: python
- @security.requires_access_dag("GET", DagAccessEntity.TASK_LOGS)
- @provide_session
- @unify_bucket_name_and_key
- @provide_bucket_name
- def get_example(
- *,
- example_param: str,
- session: Session = NEW_SESSION,
- ) -> APIResponse:
- # Implementation details here
- pass
+ @dags_router.get("/dags") # permissions go in the dependencies parameter
here
+ async def get_dags(
+ *,
+ limit: int = 100,
+ offset: int = 0,
+ tags: Annotated[list[str] | None, Query()] = None,
+ dag_id_pattern: str | None = None,
+ only_active: bool = True,
+ paused: bool | None = None,
+ order_by: str = "dag_id",
+ session: Annotated[Session, Depends(get_session)],
+ ) -> DAGCollectionResponse:
+ pass
+
+
+Step 2: Add tests for your Endpoints
+------------------------------------
+1. Verify manually with a local API that the endpoint behaves as expected.
+2. Go to the test folder and initialize new tests.
+3. Implements extensives tests to validate query param, permissions, error
handling etc.
+
+Step 3: Documentation
+---------------------
+Documentation is built automatically by FastAPI and our pre-commit hooks.
Verify by going to ``/docs`` that the documentation is clear and appears as
expected (body and return types, query params, validation)
-Step 3: Run Pre-commit Hooks
+
+Step 4: Run Pre-commit Hooks
-----------------------------
1. Ensure all code meets the project's quality standards by running pre-commit
hooks.
2. Pre-commit hooks include static code checks, formatting, and other
validations.
-3. One specific pre-commit hook to note is the ``update-common-sql-api-stubs``
hook. This hook automatically updates the common SQL API stubs whenever it
recognizes changes in the API. This ensures that any modifications to the API
are accurately reflected in the stubs, maintaining consistency between the
implementation and documentation.
+3. Persisted openapi spec is located in ``v1-generated.yaml`` and a hook will
take care of updating it based on your new endpoint, you just need to add and
commit the change.
4. Run the following command to execute all pre-commit hooks:
.. code-block:: bash
@@ -96,44 +82,29 @@ Step 3: Run Pre-commit Hooks
pre-commit run --all-files
-Optional: Adding Schemas
------------------------------
-In some cases, you may need to define additional schemas for new data
structures. For example, if you are adding an endpoint that involves new data
objects or collections, you may define a schema in a Python file. Here's an
example:
+Optional: Adding Pydantic Model
+-------------------------------
+In some cases, you may need to define additional models for new data
structures. For example, if you are adding an endpoint that involves new data
objects or collections, you may define a model in a Python file. The model will
be used to validate and serialize/deserialize objects. Here's an example:
.. code-block:: python
- class TaskLogPageSchema(Schema):
- """Schema for task log pagination details."""
-
- total_pages = fields.Int(description="Total number of pages for task
logs.")
- current_page = fields.Int(description="Current page number.")
- page_size = fields.Int(description="Number of logs per page.")
-
-These schemas are defined to structure and validate the data handled by the
API. Once defined, you can include these schemas in the OpenAPI specification
file (e.g., v1.yaml) and reference them in the API endpoint definitions.
-
-For example, in v1.yaml, you might add:
+ class DAGModelResponse(BaseModel):
+ """DAG serializer for responses."""
-.. code-block:: yaml
+ dag_id: str
+ dag_display_name: str
+ is_paused: bool
+ is_active: bool
+ last_parsed_time: datetime | None
- components:
- schemas:
- TaskLogPage:
- type: object
- properties:
- total_pages:
- type: integer
- description: Total number of pages for task logs.
- current_page:
- type: integer
- description: Current page number.
- page_size:
- type: integer
- description: Number of logs per page.
+These models are defined to structure and validate the data handled by the
API. Once defined, these models will automatically be added to the OpenAPI spec
file as long as they are actually used by one endpoint.
-Including schemas helps in automatically generating API documentation and
ensures consistent data structures across the API.
+After adding or modifying Pydantic models, make sure to run the pre-commit
hooks again to update any generated files.
-After adding or modifying schemas, make sure to run the pre-commit hooks again
to update any generated files.
+Situational: Legacy Endpoint Migration
+--------------------------------------
+When migrating legacy endpoint to the new FastAPI API:
-Note on the New FastAPI API
----------------------------
-As part of the AIP-84 you may be adding new endpoints to the FastAPI API by
migrating some legacy endpoints. In case you are doing so don't forget to mark
the legacy endpoint (Flask one) with the ``@mark_fastapi_migration_done``
decorator. This will help maintainers keep track of the endpoints remaining for
the migration and those already migrated.
+1. Implement a feature complete endpoint in comparison to the legacy one or
explain why this is not possible in your context.
+2. Make sure to have a good test coverage by copying over the legacy test
cases to the new endpoint. This will guarantee an isofunctional new endpoint.
+3. Mark the legacy endpoint with the ``@mark_fastapi_migration_done``
decorator. This will help maintainers keep track of the endpoints remaining for
the migration and those already migrated.
diff --git a/docs/spelling_wordlist.txt b/docs/spelling_wordlist.txt
index c7a7d1221c..0cdae6ba51 100644
--- a/docs/spelling_wordlist.txt
+++ b/docs/spelling_wordlist.txt
@@ -1797,6 +1797,7 @@ verticaql
Vevo
videointelligence
VideoIntelligenceServiceClient
+views
virtualenv
virtualenvs
vm
