kulkarni-sp commented on PR #44646:
URL: https://github.com/apache/airflow/pull/44646#issuecomment-2582690885
> > We are currently using Airflow 2.9.3 and encountered the same errors
during our attempt to upgrade to 2.10.3. Therefore, this fix is essential to
unblock our upgrade path.
>
> Could you please be more specific - what exactly error you experienced - i
think that one has no clear issue that it marks as "solving" ? And is it
possible that you apply that patch to verify that this one solves it ?
We are utilizing Airflow with KubernetesExecutor and have several
long-running pods that continuously monitor specified locations for input
files. Initially, we used mapped tasks to specify different input locations,
but this led to a higher number of pods and caused stability issues on our AKS
cluster. Now, we are employing ThreadPoolExecutor for parallel processing,
which triggers the processor DAG upon receiving a valid input file. However,
with Airflow 2.10.3, we encounter the following errors when triggering the
processor DAG:
**[2025-01-10, 12:53:29 UTC] {ThreadPoolExecutor-1_3 logging_mixin.py:190}
INFO - pa: Error while calling processor DAG: '_thread._local' object has no
attribute 'callers'**
```
@task(executor_config=k8s_poller_exec_config_resource_requirements)
def poll_adls_driver(config_list, **kwargs):
args_list = []
for upstream_config in config_list:
args_list.append((upstream_config, kwargs,))
with ThreadPoolExecutor(max_workers=5) as executor:
# Unpack the argument tuples
futures = [executor.submit(poll_adls, *args) for args in
args_list]
for future in futures:
future.result()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
