gerardo commented on a change in pull request #3393: [AIRFLOW-2499] Dockerised
CI pipeline
URL: https://github.com/apache/incubator-airflow/pull/3393#discussion_r189708965
##########
File path: airflow/operators/python_operator.py
##########
@@ -300,9 +300,9 @@ def _write_args(self, input_filename):
with open(input_filename, 'wb') as f:
arg_dict = ({'args': self.op_args, 'kwargs': self.op_kwargs})
if self.use_dill:
- dill.dump(arg_dict, f)
+ dill.dump(arg_dict, f, protocol=2)
Review comment:
These tests were failing because the default pickling protocol for python 2
and 3 are different, so, if the python version being used to run airflow is
different than the one passed to `PythonVirtualenvOperator` to create a
virtualenv, then `pickle.load` fails.
These tests are not failing on master, but they kept failing under this
docker setup. Not sure if it's actually a bug in my setup or is it an actual
issue in this operator, that's why I didn't create another PR, but I can
submit these changes as a separate PR if you think it's worth it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
