echoleein opened a new issue, #77148:
URL: https://github.com/apache/airflow/issues/77148

   ### Under which category would you file this issue?
   
   Airflow Core
   
   ### Apache Airflow version
   
   3.3.1
   
   ### What happened and how to reproduce it?
   
   We are experiencing an intermittent issue with PartitionedAssetTimetable in 
Apache Airflow 3.3.1.
   
   Most scheduled DAG runs work as expected. However, approximately once a 
week, the downstream DAG fails to trigger even though all 7 required asset 
events have been successfully emitted with the same partition key.
   
   File 1 — repro_asset_checker.py
   ```
   from datetime import timedelta
   import time
   
   from airflow import DAG
   from airflow.sdk import Asset, CronPartitionTimetable
   from airflow.operators.python import PythonOperator
   
   # Shared asset definitions
   ASSETS = [
       Asset(f"repro://asset_{i}")
       for i in range(1, 8)
   ]
   
   
   def emit_asset(delay_seconds=0, **context):
       if delay_seconds:
           time.sleep(delay_seconds)
   
       print(
           f"Asset producer completed. "
           f"Partition: {context['partition_key']}"
       )
   
   
   with DAG(
       dag_id="repro_asset_checker",
       schedule=CronPartitionTimetable(
           "0 18 * * *",
           timezone="UTC",
           run_offset=0,
           key_format="%Y%m%d",
       ),
       catchup=False,
       max_active_runs=1,
       tags=["asset-repro"],
   ) as dag:
   
       for i, asset in enumerate(ASSETS):
           PythonOperator(
               task_id=f"emit_asset_{i + 1}",
               python_callable=emit_asset,
               op_kwargs={
                   # First event arrives earlier.
                   # Remaining 6 arrive approximately
                   # 2 seconds later.
                   "delay_seconds": 0 if i == 0 else 2,
               },
               outlets=[asset],
           )
   ```
   
   File 2 — repro_asset_consumer.py
   ```
   from airflow import DAG
   from airflow.sdk import Asset, PartitionedAssetTimetable
   from airflow.operators.python import PythonOperator
   
   ASSETS = [
       Asset(f"repro://asset_{i}")
       for i in range(1, 8)
   ]
   
   
   def process_partition(**context):
       print(
           "Consumer triggered for partition: "
           f"{context['partition_key']}"
       )
   
   
   asset_expression = ASSETS[0]
   
   for asset in ASSETS[1:]:
       asset_expression = asset_expression & asset
   
   
   with DAG(
       dag_id="repro_asset_consumer",
       schedule=PartitionedAssetTimetable(
           assets=asset_expression,
       ),
       catchup=False,
       max_active_runs=1,
       tags=["asset-repro"],
   ) as dag:
   
       PythonOperator(
           task_id="process_partition",
           python_callable=process_partition,
       )
   ```
   
   Since the 7 entries each have a different partitioned_dag_runs ID, it 
appears we received 6 and 1 entries, respectively.
   Observed Result
   ```
   {
     "partitioned_dag_runs": [
       {
         "id": 288,
         "partition_key": "20261115",
         "total_received": 6,
         "total_required": 7,
         "state": "pending",
         "created_dag_run_id": null
       },
       {
         "id": 287,
         "partition_key": "20261115",
         "total_received": 1,
         "total_required": 7,
         "state": "pending",
         "created_dag_run_id": null
       }
     ]
   }
   ```
   
   ### What you think should happen instead?
   
   Expected Result:
   - Partition: 20261115
   - Asset Events: 7 / 7
   - Pending Dag Runs: 0
   - Created Dag Runs: 1
   - Consumer state: success
   
   
   ### Operating System
   
   _No response_
   
   ### Deployment
   
   None
   
   ### Apache Airflow Provider(s)
   
   _No response_
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Official Helm Chart version
   
   Not Applicable
   
   ### Kubernetes Version
   
   _No response_
   
   ### Helm Chart configuration
   
   _No response_
   
   ### Docker Image customizations
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to