trlopes1974 commented on issue #39717:
URL: https://github.com/apache/airflow/issues/39717#issuecomment-2265693447
> OK. I think we are getting closer to the root cause thanks to that
stacktrace. No airlfow picking is involved here (pickling is done internally by
deepcopy).
>
> What happens there:
>
> 1. Mini-scheduler is enabled
> 2. During that mini-scheduling - partial_subset() of the dag is prepared
(including deepcopy of the task + all the downstream tasks it has.
> 3. deepcopying one of the tasks fails - because the tasks apparently uses
CFFI to interact with a C code and the implementation of whatever you have
there does not allow for deepcopying the task
>
> I think a good solution would be @ashb @ephraimbuddy - following what
you've done in #27506 - to just skip mini-scheduler when something like that
happens. In this case that would mean exiting mini-scheduler without raising
any error if anything wrong happens during partial_subset.
>
> WDYT @ashb @ephraimbuddy ?
Maybe this is related to SSH /SFTP operators ? I did found a similar issue
refering paramiko ( used in SSH / SFTP)
for instance, one of the failing tasks:
ssh_command_remove_operator
```
ssh_command_remove_operator = SSHOperator(
task_id='ssh_command_remove_operator',
command='/usr/bin/sudo -s -- eval \'/usr/bin/su - orauser -c
"/home/oradb/correcao_dados/correcao_dados.sh {{
ti.xcom_pull(task_ids="prepare_task",key="WORKORDERID") }} REMOVE"\'' ,
ssh_hook=ssh_hook,
cmd_timeout = 60,
conn_timeout= 90,
get_pty=True,
do_xcom_push = True
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
