dstandish commented on code in PR #50788:
URL: https://github.com/apache/airflow/pull/50788#discussion_r2098392968
##########
airflow-core/src/airflow/migrations/versions/0055_3_0_0_remove_pickled_data_from_dagrun_table.py:
##########
@@ -61,12 +74,20 @@ def upgrade():
""")
)
else:
- BATCH_SIZE = 100
+ BATCH_SIZE = 1000
offset = 0
while True:
+ err_count = 0
+ batch_num = offset + 1
+ print(f"converting dag run conf. batch={batch_num}")
rows = conn.execute(
text(
- f"SELECT id,conf FROM dag_run WHERE conf IS not NULL order
by id LIMIT {BATCH_SIZE} OFFSET {offset}"
+ "SELECT id, conf "
+ "FROM dag_run "
+ "WHERE conf IS not NULL "
+ f"AND conf != {empty_val}"
Review Comment:
yes i inferred this because, that's what we do every time we fail to convert
anyway -- when the pickled object is not json serializable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
