amoghrajesh commented on code in PR #76107:
URL: https://github.com/apache/airflow/pull/76107#discussion_r4439817853
##########
task-sdk/src/airflow/sdk/execution_time/context.py:
##########
@@ -169,6 +169,33 @@ def _unwrap_external_ref(stored: dict) -> str | None:
return stored.get(_EXTERNAL_STATE_REF_KEY)
+def _json_size_exceeds(value: JsonValue, limit: int) -> bool:
+ """
+ Whether ``len(json.dumps(value))`` exceeds ``limit``.
+
+ Encoding stops at the first piece that crosses the limit, so a large list
or dict is never
+ encoded in full. Each string is still escaped whole before it is counted.
+ """
+ size = 0
+ for chunk in json.JSONEncoder().iterencode(value):
+ size += len(chunk)
+ if size > limit:
+ return True
+ return False
+
+
+def _warn_if_oversized(kind: str, key: str, stored: JsonValue) -> None:
+ limit = conf.getint("state_store", "max_value_storage_bytes")
+ if limit > 0 and _json_size_exceeds(stored, limit):
+ log.warning(
+ "%s store value for key %r exceeds configured
max_value_storage_bytes=%d. "
Review Comment:
```suggestion
"%s state store value for key %r exceeds configured
max_value_storage_bytes=%d. "
```
Was certainly missed earlier, but I think this is more accurate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
