GitHub user nikhilkarve created a discussion: SparkKubernetesSensor rejects
deferrable=True (Invalid arguments)
Hi team,
I’m trying to make a SparkKubernetesSensor non-blocking by using
deferrable=True, but my DAG fails at parse/init time with:
```
TypeError: Invalid arguments were passed to SparkKubernetesSensor (task_id:
ack_spark_job_sensor).
Invalid arguments were: **kwargs: { 'deferrable': True}
```
Snippet:
```
from airflow.providers.cncf.kubernetes.sensors.spark_kubernetes import
SparkKubernetesSensor
spark_job_sensor = SparkKubernetesSensor(
task_id="ack_spark_job_sensor",
poke_interval=60,
timeout=3600,
attach_log=True,
namespace="...",
kubernetes_conn_id="...",
application_name="...",
deferrable=True,
)
```
If we are using it wrong, what would be the correct way to use it?
Environment:
Airflow version: 3.0.6
apache-airflow-providers-cncf-kubernetes version: 10.9.0
GitHub link: https://github.com/apache/airflow/discussions/61209
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
