hafid-d commented on issue #8177:
URL: https://github.com/apache/airflow/issues/8177#issuecomment-909377861
Hi @potiuk ! thanks for your reply. So my method is the following:
```
def labelling(*args, **kwargs):
from app.labelling.pipeline import labelling_transform
dataset_1 = kwargs['ti'].xcom_pull(key='dataset_1',
task_ids='extracting')
category_path = kwargs['ti'].xcom_pull(key='category_path',
task_ids='extracting')
dataset_2 = labelling_transform(kwargs['dataset_1'], MODEL_PATH_TRANS,
kwargs['category_path'])
kwargs['ti'].xcom_push(key='dataset_2', value=dataset_2)
```
I imported `labelling_transform` function at the beginning, but should I
also import all the packages that `labelling_transform` uses inside the
`labelling` function ?
I've also seen that it was not possible to use the context variable `ti`
when using PythonvirtualenvOperator so I tried the following :
```
def labelling(*args, **kwargs):
from app.labelling.pipeline import labelling_transform
dataset_2 = labelling_transform(kwargs['dataset_1'], MODEL_PATH_TRANS,
kwargs['category_path'])
kwargs['ti'].xcom_push(key='dataset_2', value=dataset_2)
opr_labeling = PythonVirtualenvOperator(
task_id='labeling',
python_callable=labeling,
op_kwargs={
'dataset_1': "{{ ti.xcom_pull(key='dataset_1',
task_ids='extracting') }}",
'category_path': "{{ ti.xcom_pull(key='category_path',
task_ids='extracting') }}"
},
dag=dag_subdag)
```
Above, I tried to handle the pull but not sure how to handle the push of
dataset_2 from the virtualenv. With the above code I am getting the following
error: `subprocess.CalledProcessError: Command
'['/tmp/venv1gu9mhz9/bin/python','/tmp/venv1gu9mhz9/script.py','/tmp/venv1gu9mhz9/script.in','/tmp/venv1gu9mhz9/script.out',
'/tmp/venv1gu9mhz9/string_args.txt']' returned non-zero exit status 11`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
