kaxil opened a new pull request, #76107: URL: https://github.com/apache/airflow/pull/76107
When `[state_store] max_value_storage_bytes` is set (65535 by default), `TaskStateStoreAccessor.set()` and `AssetStateStoreAccessor.set()` ran `json.dumps` over the whole value only to compare its length with the limit and log a warning. With no `[workers] state_store_backend` configured, that meant a full extra encode of every value on every write. The supervisor's handler for language-SDK task store writes did the same. The check now walks `JSONEncoder.iterencode` and stops at the first piece that takes the length past the limit. It counts exactly what `json.dumps` produces, so the warning fires under the same condition. Tests compare it with `len(json.dumps(...))` for nested, float (including NaN and infinity) and non-ASCII values, and pin the boundary through both setters and the supervisor handler. The warning no longer reports the exact size, because computing that is the cost being removed. Measured on a 24 MB list value with no backend and the default limit (tracemalloc for peak memory): | | Before | After | |---|---|---| | Size check alone | 55.8 ms | **0.44 ms** | | Peak while building the set message | 64.0 MB | **16.0 MB** | | Peak for the whole `set()` | 64.0 MB | **40.6 MB** | The whole-`set()` peak is now set by the msgpack frame sent to the supervisor (40.6 MB, unchanged by this PR). #### Design rationale **Large strings are still encoded whole.** `iterencode` yields each string as a single escaped piece, so a value that is one big string costs the same as before. A dict holding one costs about half: I measured 100 MB down to 50 MB with a 50 MB string. Getting rid of that would mean re-implementing the encoder's size accounting. Large values belong in a `state_store_backend` anyway, and then the value checked is a small ref, which takes microseconds. **Values near the limit cost a little more.** `iterencode` runs the pure-Python encoder. For a value around the limit (72 KB of JSON), the check took 1.99 ms against 0.60 ms with `json.dumps`. Because the walk stops at the limit, that difference has a fixed upper bound at the default setting, however large the value. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
