This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 6e604f64b6 Rotate fernet key optimisation (#40758)
6e604f64b6 is described below
commit 6e604f64b68bd5ee1b9f8f6f331035665b650b01
Author: Bartosz Jankiewicz <[email protected]>
AuthorDate: Sat Jul 13 10:55:39 2024 +0200
Rotate fernet key optimisation (#40758)
* Batch processing of fernet key rotation to avoid loading entire table.
It's been observed that some users store additional data in `variable` table
which is leading to memory issues during the operation. To be consistent across
the tables added batching for all of them.
* Extracted batch size to a constant
---------
Co-authored-by: bjankiewicz <[email protected]>
---
airflow/cli/commands/rotate_fernet_key_command.py | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/airflow/cli/commands/rotate_fernet_key_command.py
b/airflow/cli/commands/rotate_fernet_key_command.py
index f4c3261573..aa44a81c8e 100644
--- a/airflow/cli/commands/rotate_fernet_key_command.py
+++ b/airflow/cli/commands/rotate_fernet_key_command.py
@@ -30,11 +30,12 @@ from airflow.utils.session import create_session
@providers_configuration_loaded
def rotate_fernet_key(args):
"""Rotates all encrypted connection credentials and variables."""
+ batch_size = 100
with create_session() as session:
conns_query = select(Connection).where(Connection.is_encrypted |
Connection.is_extra_encrypted)
- for conn in session.scalars(conns_query):
+ for conn in session.scalars(conns_query).yield_per(batch_size):
conn.rotate_fernet_key()
- for var in
session.scalars(select(Variable).where(Variable.is_encrypted)):
+ for var in
session.scalars(select(Variable).where(Variable.is_encrypted)).yield_per(batch_size):
var.rotate_fernet_key()
- for trigger in session.scalars(select(Trigger)):
+ for trigger in session.scalars(select(Trigger)).yield_per(batch_size):
trigger.rotate_fernet_key()
