Lee-W commented on code in PR #66848:
URL: https://github.com/apache/airflow/pull/66848#discussion_r3401599982
##########
airflow-core/src/airflow/partition_mappers/base.py:
##########
@@ -129,16 +131,23 @@ class RollupMapper(PartitionMapper):
"""
Partition mapper that rolls up many upstream keys into one downstream key.
- Compose a ``upstream_mapper`` (which normalizes each upstream key to the
+ Compose an ``upstream_mapper`` (which normalizes each upstream key to the
downstream granularity) with a ``window`` that declares the full set of
- upstream keys required for a given downstream key. The scheduler holds
- the Dag run until every upstream key in the window has arrived.
+ upstream keys required for a given downstream key, and a
+ ``wait_policy`` that decides when the downstream Dag run fires given
+ the expected window and the upstream keys that have actually arrived.
+ The default policy waits for every expected upstream key.
"""
is_rollup: ClassVar[bool] = True
def __init__(
- self, *, upstream_mapper: PartitionMapper, window: Window,
max_downstream_keys: int | None = None
+ self,
+ *,
+ upstream_mapper: PartitionMapper,
+ window: Window,
+ wait_policy: WaitPolicy | None = None,
Review Comment:
I thought of it. But decided not to, since I "feel" we didn't use `attrs` as
much in core. probably worth checking whether my assumption is true. but this
is a minor improvement. I'll merge this PR and refactor latter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
