aminghadersohi commented on code in PR #44604:
URL: https://github.com/apache/superset/pull/44604#discussion_r4139857491
##########
superset/db_engine_specs/bigquery.py:
##########
@@ -465,13 +465,17 @@ def fetch_data(cls, cursor: Any, limit: int | None =
None) -> list[tuple[Any, ..
# First batch already covers the budget or the result set
if rows_fetched < initial_batch_size or remaining_rows <= 0:
+ # A full sample can also be the entire result. Confirm an
omitted
+ # row; a bounded cursor leaves its own EOF probe to the
executor.
memory_limited = (
- remaining_rows <= 0 and rows_fetched == initial_batch_size
+ remaining_rows <= 0
+ and rows_fetched == initial_batch_size
+ and bool(cursor.fetchmany(1))
Review Comment:
Not applicable: the probe row is deliberately outside the return budget, not
a missing in-budget row. A non-empty probe makes `memory_limited=True`, and
`FetchedRows(first_batch, truncated=True)` explicitly reports the omission. An
empty probe preserves the complete sample with `truncated=False`. Appending the
probe would exceed the cap (including an explicit request limit).
Evidence committed in 071a1868eeab45cf9a2e8d14e96c0db2bb4d9b8f:
`test_fetch_data_eof_probe_preserves_budgeted_rows` passes all 9 cases against
the unchanged production implementation. It uses distinct row values and a
consuming iterator, with an exactly four-row memory budget, 3/4/5 available
rows, and request limits None/4/8. It asserts the entire returned sample,
truncation metadata, and driver calls: 3 and 4 rows are returned completely
with truncated=False; 5 rows return exactly the first four with truncated=True.
The fifth row is only the truncation probe.
Validation: `pytest -q tests/unit_tests/db_engine_specs
tests/unit_tests/sql/execution` — 2,093 passed, 7 skipped. Pre-commit passed on
the new test and all branch-changed files, including mypy (12 applicable hooks
for the branch).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
