FrankYang0529 commented on issue #74459:
URL: https://github.com/apache/airflow/issues/74459#issuecomment-6075830130
Tested apache.kafka 2.1.0rc1 and it works as expected:
- #68625: I ran two `AssetWatcher`s with `KafkaSharedStreamTrigger` on the
same topic. The two triggers share one Kafka consumer. Both assets get every
message. Offsets are committed per partition only after the asset events are
stored. After a restart, committed messages are not delivered again. A
connection without `enable.auto.commit=false` is refused. On Airflow 3.1.8 and
2.11.2, importing the shared-stream triggers raises
`AirflowOptionalProviderFeatureException`.
I found some issues for following providers:
- google 22.7.0rc1
- #72577: `operators/kubernetes_engine.py` now imports
`KubernetesPodExecOperator` at module level. `KubernetesPodExecOperator` only
exists in cncf-kubernetes 10.22.0 and later
(https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.22.0/changelog.html).
With an older cncf-kubernetes, importing `operators/kubernetes_engine.py`
raises `AirflowOptionalProviderFeatureException`. So every GKE operator fails
to import, not only `GKEPodExecOperator`. Should we update the GKE module like
the EKS module? In #72542, if `eks.py` cannot import
`KubernetesPodExecOperator`, it defines a fallback class with the same name.
The `__init__` of that fallback class raises the same exception that the GKE
module raises on import. The other EKS operators still import, and only
creating `EksPodExecOperator` fails. I can create a follow-up PR for this.
- dbt.cloud 4.11.0rc1
- #74199: A `DbtCloudRunJobOperator` with `deferrable=True` that works on
4.10.0 fails on 4.11.0rc1 when `hook_params` holds `retry_args` with tenacity
objects. Created a follow-up PR for it
https://github.com/apache/airflow/pull/74489.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
