potiuk commented on issue #34958: URL: https://github.com/apache/airflow/issues/34958#issuecomment-1763936076
Likely you have a problem in your parameters that are causing the script to fail before the output script is created. The script that is getting executed is https://github.com/apache/airflow/blob/main/airflow/utils/python_virtualenv_script.jinja2 and when you look at it, it should create the file as empty file automatically in the "Write output step" below. Likely your Python code (args, op_kwargs or other) fails and raises exception before the file is created. So the root cause is somewhere in the parameters that you pass to the virtualenv operator - they likely cause the generated Python code to fail. While we should probably handle the case better (hence marked that as good-first-issue for someone to pick it up) you should look for a cause in the parameters you passed. Adding here what parameters you use might be helpful for you to realise how to fix your code and for anyone (maybe even you ) who would like to improve it might help in reproducing it and adding better diagnostics for the situation. So - please. add more information on what is the code that triggers this situation @gky249 ----- ```python import {{ pickling_library }} import sys {% if expect_airflow %} {# Check whether Airflow is available in the environment. # If it is, we'll want to ensure that we integrate any macros that are being provided # by plugins prior to unpickling the task context. #} if sys.version_info >= (3,6): try: from airflow.plugins_manager import integrate_macros_plugins integrate_macros_plugins() except ImportError: {# Airflow is not available in this environment, therefore we won't # be able to integrate any plugin macros. #} pass {% endif %} {% if op_args or op_kwargs %} with open(sys.argv[1], "rb") as file: arg_dict = {{ pickling_library }}.load(file) {% else %} arg_dict = {"args": [], "kwargs": {}} {% endif %} {% if string_args_global | default(true) -%} # Read string args with open(sys.argv[3], "r") as file: virtualenv_string_args = list(map(lambda x: x.strip(), list(file))) {% endif %} # Script {{ python_callable_source }} try: res = {{ python_callable }}(*arg_dict["args"], **arg_dict["kwargs"]) except Exception as e: with open(sys.argv[4], "w") as file: file.write(str(e)) raise # Write output with open(sys.argv[2], "wb") as file: if res is not None: {{ pickling_library }}.dump(res, file) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
