Lee-W commented on code in PR #74694:
URL: https://github.com/apache/airflow/pull/74694#discussion_r4269318102
##########
providers/common/ai/docs/stability.rst:
##########
@@ -99,6 +102,110 @@ Pydantic AI toolsets, but the Pydantic AI class they
inherit from can change.
timeout under the default ``on_approval_timeout="fail"``, fails the
task, except on
``@task.llm_branch``, where it skips the downstream tasks unless
``fail_on_reject=True``.
+ * - ``decision_policy`` on ``@task.llm`` and ``@task.llm_branch``, and
+ ``min_confidence`` on
+ :class:`~airflow.providers.common.ai.policies.decision.BranchOption`
+ - Without a ``min_confidence`` the operator behaves as it does without a
policy. With
+ one, it returns (or branches on) the answer itself only when the least
confident
+ field that reported a confidence is at or above the bar. Under the bar,
+ ``on_uncertain="review"`` (the default; needs Airflow 3.1+) opens the
same human
+ review as ``require_approval``, and ``"fail"`` fails the task with
+ ``LowConfidenceError``. A model that reports no confidence counts as
under the bar.
+ ``require_approval=True`` always asks a person. On ``@task.llm_branch``
a
+ ``BranchOption.min_confidence`` replaces the policy's bar for that
branch, and with
+ ``allow_multiple_branches=True`` the strictest bar among the picked
branches applies.
+ The operator pushes a ``decision`` XCom recording the confidence and
the bar that
+ applied. ``@task.llm_sql``, ``@task.llm_file_analysis`` and
+ ``@task.llm_schema_compare`` reject a policy with a bar when they are
created.
+ * - Per-tool approval on ``AgentOperator`` (``tool_approval_timeout``,
+ ``on_tool_approval_timeout`` and ``tool_approval_assigned_users``)
+ - Needs Airflow 3.3 or later; on earlier versions a tool marked for
approval fails the
+ task. A call to a tool marked with Pydantic AI's approval API pauses
the task before
+ the tool runs and asks on the Required Actions page, showing the tool
name and its
+ arguments with secrets masked. **Approve** runs the call; **Reject**
tells the agent
+ the call was denied, with the reviewer's reason, and the run carries
on. A task
+ instance asks at most once per Dag run, retries and clears included; a
second
+ request fails the task with ``ToolApprovalAlreadyRequestedError``.
+ ``tool_approval_timeout`` (default ``None``, wait for as long as it
takes) bounds the
+ pause. ``on_tool_approval_timeout`` is ``"fail"`` (default) or
``"deny"``, which needs
+ a timeout; a timeout never approves. ``tool_approval_assigned_users``
limits who may
+ decide. Beside ``durable=True``, ``enable_hitl_review=True``, a
``CodeMode``
+ capability or a ``SandboxToolset`` that provisions its own sandbox, a
tool marked for
+ approval fails the task, except one called from code mode's
``run_code``, which does
+ not run and is reported to the model.
+ * -
:class:`~airflow.providers.common.ai.toolsets.object_storage.ObjectStorageToolset`
+ - Exposes ``list_files``, ``get_file_info`` and ``read_file``, each named
+ ``<tool_prefix>_<name>`` when ``tool_prefix`` is set, rooted at
``path`` and read-only.
+ A path that is absolute, carries a scheme or climbs out of the root is
refused. A
+ ``list_files`` result holds at most ``max_files`` entries.
``read_file`` returns a
+ text file a window of lines at a time, at most ``max_output_bytes``
bytes, with the
+ offset to continue from, and a Parquet or Avro file as its row count,
schema and first
+ rows. It refuses images, PDFs and other binary files, and any file
larger than
+ ``max_read_bytes``.
+ * -
:class:`~airflow.providers.common.ai.managed_agents.base.BaseManagedAgentHook`,
+
:class:`~airflow.providers.common.ai.toolsets.managed_agent.ManagedAgentToolset`
and
+
:class:`~airflow.providers.common.ai.toolsets.managed_agent.BaseManagedAgentToolset`
+ - A vendor hook that adopts ``BaseManagedAgentHook`` implements
``resolve_agent``,
+ ``get_agent_capabilities`` and ``invoke_agent``, and
``hook.agent(name)`` binds one
+ agent; the Amazon ``BedrockAgentCoreHook`` and the Google
``AgentEngineHook`` do so.
+ ``ManagedAgentToolset`` exposes a managed agent as one tool named
``tool_name`` that
+ takes a single ``prompt`` and gives the model the answer's text. A
rejection the model
+ can fix by rephrasing is returned to it, up to ``max_retries`` times
(default ``1``);
+ a terminal or transient error fails the call and is left to the task's
retry
+ settings. With ``durable=True`` a completed call is replayed from the
cache only when
+ ``replayable=True``. A subclass of ``BaseManagedAgentToolset``
implements
+ ``agent_ref`` and ``invoke_sync`` (or ``invoke``).
+ * - ``@task.llm_sql`` and
+
:class:`~airflow.providers.common.ai.operators.llm_sql.LLMSQLQueryOperator`
+ - Returns the SQL the model generates from the prompt and never runs it.
Schema
+ context is ``schema_context`` when given, otherwise the tables in
``table_names``
+ introspected through ``db_conn_id``. With ``validate_sql=True`` (the
default) the
+ task fails unless the result is a single statement of a type in
``allowed_sql_types``
+ (by default ``SELECT``, ``UNION``, ``INTERSECT`` and ``EXCEPT``). A
reviewer's edit
+ under ``require_approval`` is validated again against
``allowed_sql_types``, whether
+ or not ``validate_sql`` is set. ``datasource_config`` is not covered:
it reads object
+ storage through DataFusion and is experimental; see below.
+ * - ``@task.llm_file_analysis`` and
+
:class:`~airflow.providers.common.ai.operators.llm_file_analysis.LLMFileAnalysisOperator`
+ - Reads ``file_path``, a file or a prefix on object storage or a local
path, and sends
+ its content to the model in one call. At most ``max_files`` files from
a prefix are
+ included and the rest are noted in the prompt; the task fails when a
file exceeds
+ ``max_file_size_bytes`` or the files together exceed
``max_total_size_bytes``.
+ Text sent to the model is cut to ``max_text_chars``, and CSV, Parquet
and Avro inputs
+ are sampled to ``sample_rows`` rows. PNG, JPG and PDF files are
attached only with
+ ``multi_modal=True``. The output follows the rules of ``@task.llm``.
+ * - ``@task.llm_schema_compare`` and
+
:class:`~airflow.providers.common.ai.operators.llm_schema_compare.LLMSchemaCompareOperator`
+ - Compares at least two schemas, from ``table_names`` on each of
``db_conn_ids`` or from
+ ``data_sources``, and returns a ``SchemaCompareResult`` as a dict. It
takes no
+ ``output_type``. ``context_strategy="basic"`` sends column names and
types;
+ ``"full"`` (the default) adds primary keys, foreign keys and indexes. A
+ ``system_prompt`` replaces the default schema-comparison guidance, not
the fixed
+ preamble and the schemas that surround it. ``data_sources`` is not
covered: it reads a
+ source whose connection is not a ``DbApiHook`` through DataFusion and
is experimental;
+ see below.
+ * - :class:`~airflow.providers.common.ai.hooks.langchain.LangChainHook` and
+
:func:`~airflow.providers.common.ai.toolsets.langchain_bridge.airflow_toolset_to_langchain_tools`
+ - ``get_chat_model()`` and ``get_embedding_model()`` return LangChain
models for
+ identifiers in ``provider:name`` form, from ``llm_model`` or the
connection's
+ ``model`` and ``embed_model`` extras. The connection's password is
passed as
+ ``api_key`` and its host as ``base_url``. ``embed_conn_id`` falls back
to
+ ``llm_conn_id``. The bridge returns one LangChain ``StructuredTool``
per tool of the
+ toolset, with the tool's argument schema, and masks secrets in what a
tool returns.
+ An argument that fails validation, a ``ModelRetry`` (up to the tool's
+ ``max_retries``) or a ``ToolFailed`` reaches the model as an error
result. Once the
+ retries are used up, and for any other exception, the call raises and
the run fails;
+ the exception class, ``ToolCallError``, is in the experimental
``tools`` package.
+ * - Tool-call logging: ``enable_tool_logging`` on ``AgentOperator`` and
+
:class:`~airflow.providers.common.ai.toolsets.logging.ToolLoggingCapability`
+ - With ``enable_tool_logging=True`` (the default), each call to a tool
supplied through
+ ``toolsets=``, ``agent_params["tools"]`` or a capability logs its name
and duration at
+ INFO inside a ``::group::`` block and its arguments at DEBUG. A call
the run pauses
+ for approval, or that asks for a retry, deferral or skip, is logged at
INFO and
+ re-raised; any other exception is logged at ERROR with a traceback and
re-raised.
+ Output tools, provider-native tools and tools another capability adds
through its own
+ wrapper are not logged. When ``capabilities`` already holds a
+ ``ToolLoggingCapability``, the operator adds none and its ``logger`` is
used.
Experimental features
Review Comment:
Rebased onto main and took main's `stability.rst`. The Experimental notes I
had removed for `DecisionPolicy`, `min_confidence`, `ClassifierRetryPolicy`,
`ManagedAgentToolset` and the LangChain hook and bridge are restored, so they
agree with it.
##########
providers/common/ai/docs/stability.rst:
##########
@@ -99,6 +102,110 @@ Pydantic AI toolsets, but the Pydantic AI class they
inherit from can change.
timeout under the default ``on_approval_timeout="fail"``, fails the
task, except on
``@task.llm_branch``, where it skips the downstream tasks unless
``fail_on_reject=True``.
+ * - ``decision_policy`` on ``@task.llm`` and ``@task.llm_branch``, and
+ ``min_confidence`` on
+ :class:`~airflow.providers.common.ai.policies.decision.BranchOption`
+ - Without a ``min_confidence`` the operator behaves as it does without a
policy. With
+ one, it returns (or branches on) the answer itself only when the least
confident
+ field that reported a confidence is at or above the bar. Under the bar,
+ ``on_uncertain="review"`` (the default; needs Airflow 3.1+) opens the
same human
+ review as ``require_approval``, and ``"fail"`` fails the task with
+ ``LowConfidenceError``. A model that reports no confidence counts as
under the bar.
+ ``require_approval=True`` always asks a person. On ``@task.llm_branch``
a
+ ``BranchOption.min_confidence`` replaces the policy's bar for that
branch, and with
+ ``allow_multiple_branches=True`` the strictest bar among the picked
branches applies.
+ The operator pushes a ``decision`` XCom recording the confidence and
the bar that
+ applied. ``@task.llm_sql``, ``@task.llm_file_analysis`` and
+ ``@task.llm_schema_compare`` reject a policy with a bar when they are
created.
+ * - Per-tool approval on ``AgentOperator`` (``tool_approval_timeout``,
+ ``on_tool_approval_timeout`` and ``tool_approval_assigned_users``)
+ - Needs Airflow 3.3 or later; on earlier versions a tool marked for
approval fails the
+ task. A call to a tool marked with Pydantic AI's approval API pauses
the task before
+ the tool runs and asks on the Required Actions page, showing the tool
name and its
+ arguments with secrets masked. **Approve** runs the call; **Reject**
tells the agent
+ the call was denied, with the reviewer's reason, and the run carries
on. A task
+ instance asks at most once per Dag run, retries and clears included; a
second
Review Comment:
That row is gone with the rebase; `stability.rst` is now main's
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
