chxxvi opened a new issue, #57843:
URL: https://github.com/apache/airflow/issues/57843
### Apache Airflow version
3.1.1
### If "Other Airflow 2/3 version" selected, which one?
_No response_
### What happened?
I have several DAGs which use AwaitMessageSensor (from Kafka provider). Each
one has it's own `kafka_config_id` (Connection ID) connection which connect to
same Kafka broker and topic, but have different `group.id` and
`group.instance.id` configured.
When the Airflow Triggerer is configured 1 replica only, it works without
problem.
```
[info ] 8 triggers currently running
[airflow.jobs.triggerer_job_runner] loc=triggerer_job_runner.py:729
```
But when I scale Airflow Triggerer to 3 replicas, I found error logs in the
3 pods reporting that the `group.instance.id` is already used.
```
025-11-05T02:01:49.611344Z [error ]
%0|1762308109.611|FATAL|rdkafka#consumer-289| [thrd:main]: Fatal error: Broker:
Static consumer fenced by other consumer with same group.instance.id: Fatal
consumer error: Broker: Static consumer fenced by other consumer with same
group.instance.id [task.stderr]
2025-11-05T02:01:49.612066Z [info ] trigger completed
[airflow.jobs.triggerer_job_runner] loc=triggerer_job_runner.py:729 name='ID
8000'
2025-11-05T02:01:50.246744Z [error ] Trigger ID 8000 exited with error
Error: KafkaError{code=_FATAL,val=-150,str="Fatal error: Broker: Static
consumer fenced by other consumer with same group.instance.id: Fatal consumer
error: Broker: Static consumer fenced by other consumer with same
group.instance.id"} [airflow.jobs.triggerer_job_runner]
error_detail=[{'exc_type': 'AirflowException', 'exc_value': 'Error:
KafkaError{code=_FATAL,val=-150,str="Fatal error: Broker: Static consumer
fenced by other consumer with same group.instance.id: Fatal consumer error:
Broker: Static consumer fenced by other consumer with same
group.instance.id"}', 'exc_notes': [], 'syntax_error': None, 'is_cause': False,
'frames': [{'filename':
'/home/airflow/.local/lib/python3.12/site-packages/airflow/jobs/triggerer_job_runner.py',
'lineno': 992, 'name': 'cleanup_finished_triggers'}, {'filename':
'/home/airflow/.local/lib/python3.12/site-packages/greenback/_impl.py',
'lineno': 116, 'name': 'greenback_shim'}
, {'filename':
'/home/airflow/.local/lib/python3.12/site-packages/greenback/_impl.py',
'lineno': 201, 'name': '_greenback_shim'}, {'filename':
'/home/airflow/.local/lib/python3.12/site-packages/greenback/_impl.py',
'lineno': 81, 'name': 'trampoline'}, {'filename':
'/home/airflow/.local/lib/python3.12/site-packages/outcome/_impl.py', 'lineno':
185, 'name': 'send'}, {'filename':
'/home/airflow/.local/lib/python3.12/site-packages/airflow/jobs/triggerer_job_runner.py',
'lineno': 1106, 'name': 'run_trigger'}, {'filename':
'/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/apache/kafka/triggers/await_message.py',
'lineno': 111, 'name': 'run'}], 'is_group': False, 'exceptions': []}]
loc=triggerer_job_runner.py:729
2025-11-05T02:01:50.247315Z [error ] Trigger exited without sending an
event. Dependent tasks will be failed. [airflow.jobs.triggerer_job_runner]
loc=triggerer_job_runner.py:729 name='ID 8000'
```
### What you think should happen instead?
_No response_
### How to reproduce
n/a
### Operating System
AlmaLinux 9.6
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
