uranusjr commented on code in PR #66848:
URL: https://github.com/apache/airflow/pull/66848#discussion_r3401520670
##########
airflow-core/src/airflow/partition_mappers/base.py:
##########
@@ -129,16 +131,23 @@ class RollupMapper(PartitionMapper):
"""
Partition mapper that rolls up many upstream keys into one downstream key.
- Compose a ``upstream_mapper`` (which normalizes each upstream key to the
+ Compose an ``upstream_mapper`` (which normalizes each upstream key to the
downstream granularity) with a ``window`` that declares the full set of
- upstream keys required for a given downstream key. The scheduler holds
- the Dag run until every upstream key in the window has arrived.
+ upstream keys required for a given downstream key, and a
+ ``wait_policy`` that decides when the downstream Dag run fires given
+ the expected window and the upstream keys that have actually arrived.
+ The default policy waits for every expected upstream key.
"""
is_rollup: ClassVar[bool] = True
def __init__(
- self, *, upstream_mapper: PartitionMapper, window: Window,
max_downstream_keys: int | None = None
+ self,
+ *,
+ upstream_mapper: PartitionMapper,
+ window: Window,
+ wait_policy: WaitPolicy | None = None,
Review Comment:
Probably aren’t essential but we can use attrs here if we want to too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
