dabla opened a new issue, #63344:
URL: https://github.com/apache/airflow/issues/63344
### Apache Airflow version
3.1.7
### If "Other Airflow 3 version" selected, which one?
_No response_
### What happened?
When passing a non-materialized iterable (e.g., a generator) to the rows
parameter of the SQLInsertRowsOperator, the operator crashes during execution.
This happens because the operator attempts to log the length of the rows
parameter using len(rows). Non-materialized iterables do not implement __len__,
which results in a TypeError.
### What you think should happen instead?
The operator should not assume that rows has a defined length and therefore
should avoid calling len(rows).
Additionally, the existing tests for SQLInsertRowsOperator currently pass a
list to the rows parameter. These tests should be updated to use a
non-materialized iterable (such as a generator) to ensure this behavior is
properly supported and to prevent regressions in the future.
### How to reproduce
```
def rows_gen():
yield {"index": 1, "name": "Stallone", "firstname": "Sylvester",
"age": 78}
yield {"index": 2, "name": "Statham", "firstname": "Jason", "age":
57}
yield {"index": 3, "name": "Li", "firstname": "Jet", "age": 61}
yield {"index": 4, "name": "Lundgren", "firstname": "Dolph", "age":
66}
yield {"index": 5, "name": "Norris", "firstname": "Chuck", "age": 84}
operator = SQLInsertRowsOperator(
task_id="test_task",
conn_id="default_conn",
schema="hollywood",
table_name="actors",
rows=rows_gen(), # generator (no __len__)
rows_processor=lambda rows, **context: [tuple(row.values()) for row
in rows],
)
```
### Operating System
Linux
### Versions of Apache Airflow Providers
apache-airflow-providers-common-sql 1.32.0
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
