mik-laj edited a comment on issue #6389: [AIRFLOW-5663] Switch to real-time logging in PythonVirtualenvOperator URL: https://github.com/apache/airflow/pull/6389#issuecomment-546360165 Hello, I used the example below. ```diff From 860d5addc8ab6ffec9327a4a861bf7547ea9d5cc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kamil=20Bregu=C5=82a?= <[email protected]> Date: Fri, 25 Oct 2019 15:43:42 +0200 Subject: Add example for PythonVirtualenvOperator --- .../example_dags/example_python_operator.py | 26 ++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/airflow/example_dags/example_python_operator.py b/airflow/example_dags/example_python_operator.py index 86403ceb2..68e617731 100644 --- a/airflow/example_dags/example_python_operator.py +++ b/airflow/example_dags/example_python_operator.py @@ -24,7 +24,7 @@ from pprint import pprint import airflow from airflow.models import DAG -from airflow.operators.python_operator import PythonOperator +from airflow.operators.python_operator import PythonOperator, PythonVirtualenvOperator args = { 'owner': 'Airflow', @@ -71,3 +71,27 @@ for i in range(5): run_this >> task # [END howto_operator_python_kwargs] + + +def callable_virtualenv(): + from colorama import Fore, Back, Style + from time import sleep + print(Fore.RED + 'some red text') + print(Back.GREEN + 'and with a green background') + print(Style.DIM + 'and in dim text') + print(Style.RESET_ALL) + for _ in range(10): + print(Style.DIM + 'Please wait...') + sleep(10) + print('Finished') + + +virtualenv_task = PythonVirtualenvOperator( + task_id="virtualenv_python", + python_callable=callable_virtualenv, + requirements=[ + "colorama==0.4.0" + ], + system_site_packages=False, + dag=dag, +) -- 2.20.1 ``` Unfortunately, the messages do not show in real time. Did I miss something? ``` root@1ddb884f5c23:/opt/airflow# airflow tasks test example_python_operator virtualenv_python 2019-01-01 /opt/airflow/airflow/models/dagbag.py:21: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp [2019-10-25 13:38:59,355] {settings.py:175} INFO - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=57962 /usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>. """) [2019-10-25 13:38:59,893] {__init__.py:53} INFO - Using executor LocalExecutor [2019-10-25 13:38:59,893] {dagbag.py:88} INFO - Filling up the DagBag from /opt/airflow/airflow/example_dags [2019-10-25 13:39:00,399] {taskinstance.py:632} INFO - Dependencies all met for <TaskInstance: example_python_operator.virtualenv_python 2019-01-01T00:00:00+00:00 [None]> [2019-10-25 13:39:00,410] {taskinstance.py:632} INFO - Dependencies all met for <TaskInstance: example_python_operator.virtualenv_python 2019-01-01T00:00:00+00:00 [None]> [2019-10-25 13:39:00,410] {taskinstance.py:843} INFO - -------------------------------------------------------------------------------- [2019-10-25 13:39:00,410] {taskinstance.py:844} INFO - Starting attempt 1 of 1 [2019-10-25 13:39:00,411] {taskinstance.py:845} INFO - -------------------------------------------------------------------------------- [2019-10-25 13:39:00,413] {taskinstance.py:864} INFO - Executing <Task(PythonVirtualenvOperator): virtualenv_python> on 2019-01-01T00:00:00+00:00 [2019-10-25 13:39:00,428] {python_operator.py:135} INFO - Exporting the following env vars: AIRFLOW_CTX_DAG_ID=example_python_operator AIRFLOW_CTX_TASK_ID=virtualenv_python AIRFLOW_CTX_EXECUTION_DATE=2019-01-01T00:00:00+00:00 [2019-10-25 13:39:00,429] {python_operator.py:334} INFO - Executing cmd ['virtualenv', '/tmp/venvnw9fmnxz'] [2019-10-25 13:39:00,439] {python_operator.py:340} INFO - Got output [2019-10-25 13:39:01,674] {python_operator.py:343} INFO - b"Using base prefix '/usr/local'\n" [2019-10-25 13:39:01,676] {python_operator.py:343} INFO - b'New python executable in /tmp/venvnw9fmnxz/bin/python\n' [2019-10-25 13:39:01,676] {python_operator.py:343} INFO - b'Installing setuptools, pip, wheel...\n' [2019-10-25 13:39:05,634] {python_operator.py:343} INFO - b'done.\n' [2019-10-25 13:39:05,648] {python_operator.py:334} INFO - Executing cmd ['/tmp/venvnw9fmnxz/bin/pip', 'install', 'colorama==0.4.0'] [2019-10-25 13:39:05,656] {python_operator.py:340} INFO - Got output [2019-10-25 13:39:06,315] {python_operator.py:343} INFO - b'Collecting colorama==0.4.0\n' [2019-10-25 13:39:06,474] {python_operator.py:343} INFO - b' Downloading https://files.pythonhosted.org/packages/0a/93/6e8289231675d561d476d656c2ee3a868c1cca207e16c118d4503b25e2bf/colorama-0.4.0-py2.py3-none-any.whl\n' [2019-10-25 13:39:06,523] {python_operator.py:343} INFO - b'Installing collected packages: colorama\n' [2019-10-25 13:39:06,542] {python_operator.py:343} INFO - b'Successfully installed colorama-0.4.0\n' [2019-10-25 13:39:06,824] {python_operator.py:334} INFO - Executing cmd ['/tmp/venvnw9fmnxz/bin/python', '/tmp/venvnw9fmnxz/script.py', '/tmp/venvnw9fmnxz/script.in', '/tmp/venvnw9fmnxz/script.out', '/tmp/venvnw9fmnxz/string_args.txt'] [2019-10-25 13:39:06,832] {python_operator.py:340} INFO - Got output [2019-10-25 13:40:46,901] {python_operator.py:343} INFO - b'\x1b[31msome red text\n' [2019-10-25 13:40:46,902] {python_operator.py:343} INFO - b'\x1b[42mand with a green background\n' [2019-10-25 13:40:46,904] {python_operator.py:343} INFO - b'\x1b[2mand in dim text\n' [2019-10-25 13:40:46,904] {python_operator.py:343} INFO - b'\x1b[0m\n' [2019-10-25 13:40:46,905] {python_operator.py:343} INFO - b'\x1b[2mPlease wait...\n' [2019-10-25 13:40:46,905] {python_operator.py:343} INFO - b'\x1b[2mPlease wait...\n' [2019-10-25 13:40:46,906] {python_operator.py:343} INFO - b'\x1b[2mPlease wait...\n' [2019-10-25 13:40:46,907] {python_operator.py:343} INFO - b'\x1b[2mPlease wait...\n' [2019-10-25 13:40:46,907] {python_operator.py:343} INFO - b'\x1b[2mPlease wait...\n' [2019-10-25 13:40:46,908] {python_operator.py:343} INFO - b'\x1b[2mPlease wait...\n' [2019-10-25 13:40:46,908] {python_operator.py:343} INFO - b'\x1b[2mPlease wait...\n' [2019-10-25 13:40:46,908] {python_operator.py:343} INFO - b'\x1b[2mPlease wait...\n' [2019-10-25 13:40:46,909] {python_operator.py:343} INFO - b'\x1b[2mPlease wait...\n' [2019-10-25 13:40:46,909] {python_operator.py:343} INFO - b'\x1b[2mPlease wait...\n' [2019-10-25 13:40:46,910] {python_operator.py:343} INFO - b'Finished\n' [2019-10-25 13:40:46,959] {python_operator.py:144} INFO - Done. Returned value was: None ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
