kaxil opened a new pull request, #73925: URL: https://github.com/apache/airflow/pull/73925
Follow-up to #73899. `ObjectStorageToolset`'s `read_file` returned a Parquet or Avro file as its schema and first 20 rows, with no row count, and silently ignored `offset` and `limit`. In a run against a real model, asked how many rows a Parquet file held, the agent called `read_file` again with `offset=21`, got the same 20 rows back, and answered that it could not tell. The result now opens with the row count and says that `offset` and `limit` do not apply: ``` Rows: 250. The schema and the first 20 rows follow; offset and limit do not apply to Parquet files. Schema: order_id: int64, region: string, total: double Sample rows: ... ``` The header comes before the sample, so cutting a long result never drops it. Parquet's count comes from the file footer. An Avro file is read in one pass over its blocks: the sample comes from the first blocks and the count from every block header, so only the blocks the sample reaches have their records decoded, and the file is opened once, under the existing `max_read_bytes` check. `LLMFileAnalysisOperator` keeps its current Avro path. A file whose Avro schema is not a record now shows its values in the sample instead of an empty list. Counting reaches every block, which exposed a gap: a corrupt deflate block raises `zlib.error`, and corrupt `.xz` text raises `lzma.LZMAError`, neither of which is an `OSError` or `ValueError`, so either one failed the run. Both are now refused like any other unreadable file, and the model is told why. --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
