potiuk commented on code in PR #25780:
URL: https://github.com/apache/airflow/pull/25780#discussion_r963015658
##########
airflow/operators/python.py:
##########
@@ -501,27 +555,152 @@ def _iter_serializable_context_keys(self):
elif 'pendulum' in self.requirements:
yield from self.PENDULUM_SERIALIZABLE_CONTEXT_KEYS
- def _write_string_args(self, filename):
- with open(filename, 'w') as file:
- file.write('\n'.join(map(str, self.string_args)))
- def _read_result(self, filename):
- if os.stat(filename).st_size == 0:
- return None
- with open(filename, 'rb') as file:
- try:
- return self.pickling_library.load(file)
- except ValueError:
- self.log.error(
- "Error deserializing result. Note that result
deserialization "
- "is not supported across major Python versions."
+class ExternalPythonOperator(_BasePythonVirtualenvOperator):
+ """
+ Allows one to run a function in a virtualenv that is not re-created but
used as is
+ without the overhead of creating the virtualenv (with certain caveats).
+
+ The function must be defined using def, and not be
+ part of a class. All imports must happen inside the function
+ and no variables outside the scope may be referenced. A global scope
+ variable named virtualenv_string_args will be available (populated by
+ string_args). In addition, one can pass stuff through op_args and
op_kwargs, and one
+ can use a return value.
+ Note that if your virtualenv runs in a different Python major version than
Airflow,
+ you cannot use return values, op_args, op_kwargs, or use any macros that
are being provided to
+ Airflow through plugins. You can use string_args though.
+
+ .. seealso::
+ For more information on how to use this operator, take a look at the
guide:
+ :ref:`howto/operator:ExternalPythonOperator`
+
+ :param python: Full path string (file-system specific) that points to a
Python binary inside
+ a virtualenv that should be used (in ``VENV/bin`` folder). Should be
absolute path
+ (so usually start with "/" or "X:/" depending on the filesystem/os
used).
+ :param python_callable: A python function with no references to outside
variables,
+ defined with def, which will be run in a virtualenv
+ :param use_dill: Whether to use dill to serialize
+ the args and result (pickle is default). This allow more complex types
+ but if dill is not preinstalled in your venv, the task will fail with
use_dill enabled.
+ :param op_args: A list of positional arguments to pass to python_callable.
+ :param op_kwargs: A dict of keyword arguments to pass to python_callable.
+ :param string_args: Strings that are present in the global var
virtualenv_string_args,
+ available to python_callable at runtime as a list[str]. Note that args
are split
+ by newline.
Review Comment:
They are still useful (and necessary) when you want to use different Python
version than the one Airflow is run on. I belive even piclkled plain string
will not deserilize nicely if you go down with Python versions (this is what I
believe the original reasoning for having string args was and it stlll holds
for ExtrenalPython operator. I'd leave it - even if it might be a bit
confusing, it migh save some headaches. Also I think one of the benefits of
having PythonExternalOperator is "productionizing" of PythonVirtualenvOperator.
The latter can help the users to iterate and develop , where the former can
allow to use the same tasks without the venv creation overhed in production. So
having 1-1 feature parity between those two is important to be able to
seamlessly switch between them.
##########
airflow/operators/python.py:
##########
@@ -501,27 +555,152 @@ def _iter_serializable_context_keys(self):
elif 'pendulum' in self.requirements:
yield from self.PENDULUM_SERIALIZABLE_CONTEXT_KEYS
- def _write_string_args(self, filename):
- with open(filename, 'w') as file:
- file.write('\n'.join(map(str, self.string_args)))
- def _read_result(self, filename):
- if os.stat(filename).st_size == 0:
- return None
- with open(filename, 'rb') as file:
- try:
- return self.pickling_library.load(file)
- except ValueError:
- self.log.error(
- "Error deserializing result. Note that result
deserialization "
- "is not supported across major Python versions."
+class ExternalPythonOperator(_BasePythonVirtualenvOperator):
+ """
+ Allows one to run a function in a virtualenv that is not re-created but
used as is
+ without the overhead of creating the virtualenv (with certain caveats).
+
+ The function must be defined using def, and not be
+ part of a class. All imports must happen inside the function
+ and no variables outside the scope may be referenced. A global scope
+ variable named virtualenv_string_args will be available (populated by
+ string_args). In addition, one can pass stuff through op_args and
op_kwargs, and one
+ can use a return value.
+ Note that if your virtualenv runs in a different Python major version than
Airflow,
+ you cannot use return values, op_args, op_kwargs, or use any macros that
are being provided to
+ Airflow through plugins. You can use string_args though.
+
+ .. seealso::
+ For more information on how to use this operator, take a look at the
guide:
+ :ref:`howto/operator:ExternalPythonOperator`
+
+ :param python: Full path string (file-system specific) that points to a
Python binary inside
+ a virtualenv that should be used (in ``VENV/bin`` folder). Should be
absolute path
+ (so usually start with "/" or "X:/" depending on the filesystem/os
used).
+ :param python_callable: A python function with no references to outside
variables,
+ defined with def, which will be run in a virtualenv
+ :param use_dill: Whether to use dill to serialize
+ the args and result (pickle is default). This allow more complex types
+ but if dill is not preinstalled in your venv, the task will fail with
use_dill enabled.
+ :param op_args: A list of positional arguments to pass to python_callable.
+ :param op_kwargs: A dict of keyword arguments to pass to python_callable.
+ :param string_args: Strings that are present in the global var
virtualenv_string_args,
+ available to python_callable at runtime as a list[str]. Note that args
are split
+ by newline.
Review Comment:
They are still useful (and necessary as I understand) when you want to use
different Python version than the one Airflow is run on. I belive even piclkled
plain string will not deserilize nicely if you go down with Python versions
(this is what I believe the original reasoning for having string args was and
it stlll holds for ExtrenalPython operator. I'd leave it - even if it might be
a bit confusing, it migh save some headaches. Also I think one of the benefits
of having PythonExternalOperator is "productionizing" of
PythonVirtualenvOperator. The latter can help the users to iterate and develop
, where the former can allow to use the same tasks without the venv creation
overhed in production. So having 1-1 feature parity between those two is
important to be able to seamlessly switch between them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
