potiuk commented on code in PR #72229:
URL: https://github.com/apache/airflow/pull/72229#discussion_r4186189365
##########
providers/amazon/tests/system/amazon/aws/example_athena_spark.py:
##########
@@ -98,6 +108,17 @@ def stop_athena_spark_session(session_id: str) -> None:
)
# [END howto_operator_athena_spark]
+ calculation_execution_id = start_athena_spark_calculation(idle_session_id)
+
+ # [START howto_sensor_athena_spark]
+ await_spark_calculation = AthenaSparkSensor(
+ task_id="await_spark_calculation",
+ calculation_execution_id=run_spark_calculation.output,
Review Comment:
`AthenaSparkOperator.execute()` returns a metadata dict
(`{"calculation_execution_id": ..., "state": ..., ...}`), not the ID string, so
the sensor receives the whole dict.
`get_calculation_execution(CalculationExecutionId=<dict>)` then raises boto3's
`ParamValidationError` on the first poke, and the system test fails every run.
Meanwhile the calculation started by `start_athena_spark_calculation` on line
111 is chained in but never awaited.
```suggestion
calculation_execution_id=calculation_execution_id,
```
This block is also the published `howto_sensor_athena_spark` snippet, so the
docs show the broken wiring too. Relatedly, `athena_spark.rst` suggests getting
the ID "for example by calling `AthenaSparkOperator`". That operator already
waits for the calculation to finish, so pairing it with the sensor isn't
useful; it may be clearer to say the ID comes from a calculation started
without waiting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
