uranusjr commented on code in PR #68283:
URL: https://github.com/apache/airflow/pull/68283#discussion_r3494047973
##########
providers/common/io/src/airflow/providers/common/io/get_provider_info.py:
##########
@@ -84,6 +84,27 @@ def get_provider_info():
"example": "gz",
"default": "",
},
+ "state_store_objectstorage_path": {
+ "description": "Base path on object storage for the
task/asset state store backend, in URL format.\nWhen set,
StateStoreObjectStorageBackend will persist task and asset state under
this\nprefix, organised as <dag_id>/<run_id>/<task_id>/<map_index>/<key> for
tasks and\nassets/<asset_identifier>/<key> for assets.\n",
+ "version_added": "1.8.0",
+ "type": "string",
+ "example": "s3://conn_id@bucket/task-state/",
+ "default": "",
+ },
+ "state_store_objectstorage_threshold": {
+ "description": "Threshold in bytes for offloading
serialized state store values to object storage. 0 means\nalways offload to
object storage. Any positive number means values will be offloaded\nonly when
their serialized size is at least that many bytes. Must be non-negative.\n",
Review Comment:
I think the system wouldn’t actually break if the value is negative? It
would just act the same (or somehow even stricter) than 0. Maybe we should
either raise a hard error if the value is negative (to match the tone here), or
change this description to a softer tone. (Or maybe we can just change
`_get_threshold` to coerce a negative value to 0.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
