kaxil commented on code in PR #73384:
URL: https://github.com/apache/airflow/pull/73384#discussion_r4056833972
##########
providers/common/ai/src/airflow/providers/common/ai/toolsets/datafusion.py:
##########
@@ -101,7 +101,8 @@ class DataFusionToolset(AbstractToolset[Any]):
INSERT INTO, etc.). Default ``False`` — only SELECT-family statements
are permitted.
:param max_rows: Maximum number of rows returned from the ``query`` tool.
- Default ``50``.
+ Default ``50``. The query is limited to ``max_rows + 1`` rows, so a
large
+ result is never fully materialized; the extra row only signals
truncation.
:param max_result_bytes: Budget for the serialized ``query`` result, in
bytes.
Review Comment:
The "Rows are fetched, not filtered" paragraph in
`providers/common/ai/docs/toolsets.rst` (around line 254) still says
`DataFusionToolset` "materializes the full result in the engine before the
toolset sees it; in both the payload is bounded but the transfer is not". That
is the behaviour this PR removes, so that sentence wants updating in the same
change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
