[GitHub] [airflow] potiuk commented on a change in pull request #6285: [AIRFLOW-XXX] Updates to Breeze documentation from GSOD

[GitBox]

potiuk commented on a change in pull request #6285: [AIRFLOW-XXX] Updates to 
Breeze documentation from GSOD
URL: https://github.com/apache/airflow/pull/6285#discussion_r334264045
 
 

 ##
 File path: BREEZE.rst
 ##
 @@ -603,50 +588,38 @@ your local sources into the ``/opt/airflow`` location of 
the sources within the
 :align: center
 :alt: Source code mapping
 
+Breeze Command-Line Interface Reference
+===
 
-Airflow Breeze flags
-
+Airflow Breeze Syntax
+-
 
-These are the current flags of the `./breeze <./breeze>`_ script
+This is the current syntax for  `./breeze <./breeze>`_:
 
 Review comment:
   FYI: this output is copy directly from ./breeze --help, so we will 
need to synchronize it with 
https://github.com/PolideaInternal/airflow/blob/gsod-breeze-documentation-update/breeze
 where it is generated from. I will do it later before we merge so no worries 
about it for now - let's update it in the docs and I will put it back to the 
help script.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #6285: [AIRFLOW-XXX] Updates to Breeze documentation from GSOD

[GitBox]

potiuk commented on a change in pull request #6285: [AIRFLOW-XXX] Updates to 
Breeze documentation from GSOD
URL: https://github.com/apache/airflow/pull/6285#discussion_r334263955
 
 

 ##
 File path: BREEZE.rst
 ##
 @@ -347,74 +329,105 @@ can check whether your problem is fixed.
 4. `Clean up Docker images <#cleaning-up-the-images>`_.
 5. Restart your Docker Engine and try again.
 6. Restart your machine and try again.
-7. Remove and re-install Docker CE and try again.
+7. Re-install Docker CE and try again.
 
 In case the problems are not solved, you can set the VERBOSE variable to 
"true" (``export VERBOSE="true"``),
 rerun the failed command, copy-and-paste the output from your terminal to the 
_`Airflow Slack `_  
#troubleshooting channel and add the problem description.
 
 
-Using Breeze for Other Tasks
-
+Testing in Breeze
+=
+
+Running Unit Tests in Airflow Breeze
+---
 
 Review comment:
   One comment here - in .rst the -- lines length below header must much 
the header length :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #6285: [AIRFLOW-XXX] Updates to Breeze documentation from GSOD

[GitBox]

potiuk commented on a change in pull request #6285: [AIRFLOW-XXX] Updates to 
Breeze documentation from GSOD
URL: https://github.com/apache/airflow/pull/6285#discussion_r334263891
 
 

 ##
 File path: BREEZE.rst
 ##
 @@ -474,61 +453,47 @@ Run pylint checks for all files:
  ./breeze --static-check-all-files pylint
 
 
-The ``license`` check is also run via separate script and separate docker 
image containing
+The ``license`` check is run via a separate script and a separate docker image 
containing the 
 Apache RAT verification tool that checks for Apache-compatibility of licences 
within the codebase.
-It does not take pre-commit parameters as extra args.
+It does not take pre-commit parameters as extra arguments.
 
 .. code-block:: bash
 
  ./breeze --static-check-all-files licenses
 
-Building the documentation
---
-
-The documentation is build using ``-O``, ``--build-docs`` command:
-
-.. code-block:: bash
-
- ./breeze --build-docs
-
-Results of the build can be found in ``docs/_build`` folder. Often errors 
during documentation generation
-come from the docstrings of auto-api generated classes. During the docs 
building auto-api generated
-files are stored in ``docs/_api`` folder - so that in case of problems with 
documentation you can
-find where the problems with documentation originated from.
-
-Running tests directly from host
-
+Running Tests Directly from the Host
+
 
-If you wish to run tests only and not drop into shell, you can run them by 
providing
--t, --test-target flag. You can add extra nosetest flags after -- in the 
commandline.
+If you wish to only run tests and not to drop into shell, you can do this by 
providing the
+-t, --test-target flag. You can add extra nosetest flags after -- in the 
command line.
 
 .. code-block:: bash
 
  ./breeze --test-target tests/hooks/test_druid_hook.py -- 
--logging-level=DEBUG
 
-You can run the whole test suite with special '.' test target:
+You can run the whole test suite with a special '.' test target:
 
 .. code-block:: bash
 
 ./breeze --test-target .
 
-You can also specify individual tests or group of tests:
+You can also specify individual tests or a group of tests:
 
 .. code-block:: bash
 
 ./breeze --test-target tests.core:TestCore
 
-Pulling the latest images
+Pulling the Latest Images
 -
 
-Sometimes the image on DockerHub is rebuilt from the scratch. This happens for 
example when there is a
-security update of the python version that all the images are based on.
-In this case it is usually faster to pull latest images rather than rebuild 
them
-from the scratch.
+Sometimes the image on the Docker Hub needs to be rebuilt from scratch. This 
is required, for example, when there is a
 
 Review comment:
   The image in DockerHub is rebuilt automatically when there is new security 
release of python image. In order to use the new "security patched" images,  it 
is better to pull the latest version (--force-pull-images) rather than continue 
with already built images locally. I will update this behaviour slightly to 
work as follows (maybe it will be easier to explain):
   - when new security patched python images are released, the images in 
DockerHub are rebuilt from the scratch
   - when you continue running breeze locally, you will use the already 
downloaded images (based old python version).
   - if you want to update to latest security-patched images you need to run 
breeze with "--force-pull-images" flag.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #6285: [AIRFLOW-XXX] Updates to Breeze documentation from GSOD

[GitBox]

potiuk commented on a change in pull request #6285: [AIRFLOW-XXX] Updates to 
Breeze documentation from GSOD
URL: https://github.com/apache/airflow/pull/6285#discussion_r334263780
 
 

 ##
 File path: BREEZE.rst
 ##
 @@ -290,165 +264,170 @@ It is as easy as copy this line into your code:
 
import ipdb; ipdb.set_trace()
 
-Once you hit the line you will be dropped into interactive ipdb  debugger 
where you have colors
-and auto-completion to guide your debugging. This works from the console where 
you started your program.
-Note that in case of ``nosetest`` you need to provide ``--nocapture`` flag to 
avoid nosetests
+Once you hit the line, you will be dropped into an interactive ``ipdb`` 
debugger where you have colors
+and autocompletion to guide your debugging. This works from the console where 
you started your program.
+Note that in case of ``nosetest`` you need to provide the ``--nocapture`` flag 
to avoid nosetests
 capturing the stdout of your process.
 
-Airflow directory structure inside Docker
--
-
-When you are in the container note that following directories are used:
-
-.. code-block:: text
-
-  /opt/airflow - here sources of Airflow are mounted from the host 
(AIRFLOW_SOURCES)
-  /root/airflow - all the "dynamic" Airflow files are created here: 
(AIRFLOW_HOME)
-  airflow.db - sqlite database in case sqlite is used
-  dags - folder where non-test dags are stored (test dags are in 
/opt/airflow/tests/dags)
-  logs - logs from airflow executions are created there
-  unittest.cfg - unit test configuration generated when entering the 
environment
-  webserver_config.py - webserver configuration generated when running 
airflow in the container
 
-Note that when run in your local environment ``/root/airflow/logs`` folder is 
actually mounted from your
-``logs`` directory in airflow sources, so all logs created in the container 
are automatically visible in the host
-as well. Every time you enter the container the logs directory is cleaned so 
that logs do not accumulate.
-
-Port forwarding
+Port Forwarding
 ---
 
 When you run Airflow Breeze, the following ports are automatically forwarded:
 
-* 28080 -> forwarded to airflow webserver -> airflow-testing:8080
-* 25433 -> forwarded to postgres database -> postgres:5432
-* 23306 -> forwarded to mysql database  -> mysql:3306
+* 28080 -> forwarded to Airflow webserver -> airflow-testing:8080
+* 25433 -> forwarded to Postgres database -> postgres:5432
+* 23306 -> forwarded to Mysql database  -> mysql:3306
 
-You can connect to those ports/databases using:
+You can connect to these ports/databases using:
 
 * Webserver: ``http://127.0.0.1:28080``
 * Postgres: 
``jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres=airflow``
 * Mysql: ``jdbc:mysql://localhost:23306/airflow?user=root``
 
-Note that you need to start the webserver manually with ``airflow webserver`` 
command if you want to connect
-to the webserver (you can use ``tmux`` to multiply terminals).
+Start the webserver manually with the ``airflow webserver`` command if you 
want to connect
+to the webserver. You can use ``tmux`` to multiply terminals.
 
-For databases you need to run ``airflow db reset`` at least once (or run some 
tests) after you started
-Airflow Breeze to get the database/tables created. You can connect to databases
-with IDE or any other Database client:
+For databases, you need to run ``airflow db reset`` at least once (or run some 
tests) after you started
+Airflow Breeze to get the database/tables created. You can connect to 
databases with IDE or any other database client:
 
 .. image:: images/database_view.png
 :align: center
 :alt: Database view
 
-You can change host port numbers used by setting appropriate environment 
variables:
+You can change the used host port numbers by setting appropriate environment 
variables:
 
 * ``WEBSERVER_HOST_PORT``
 * ``POSTGRES_HOST_PORT``
 * ``MYSQL_HOST_PORT``
 
-When you set those variables, next time when you enter the environment the new 
ports should be in effect.
+If you set these variables, next time when you enter the environment the new 
ports should be in effect.
 
-Cleaning up the images
---
+Building the Documentation
+--
 
-You might need to cleanup your Docker environment occasionally. The images are 
quite big
-(1.5GB for both images needed for static code analysis and CI tests). And if 
you often rebuild/update
-images you might end up with some unused image data.
+To build documentation in Breeze, use the ``-O``, ``--build-docs`` command:
 
-Cleanup can be performed with ``docker system prune`` command.
-Make sure to `Stop Breeze <#stopping-breeze>`_ first with ``./breeze 
--stop-environment``.
+.. code-block:: bash
 
-If you run into disk space errors, we recommend you prune your docker images 
using the
-``docker system prune --all`` command. You might need to restart the 

[GitHub] [airflow] potiuk commented on a change in pull request #6285: [AIRFLOW-XXX] Updates to Breeze documentation from GSOD

[GitBox]

potiuk commented on a change in pull request #6285: [AIRFLOW-XXX] Updates to 
Breeze documentation from GSOD
URL: https://github.com/apache/airflow/pull/6285#discussion_r334263747
 
 

 ##
 File path: BREEZE.rst
 ##
 @@ -290,165 +264,170 @@ It is as easy as copy this line into your code:
 
import ipdb; ipdb.set_trace()
 
-Once you hit the line you will be dropped into interactive ipdb  debugger 
where you have colors
-and auto-completion to guide your debugging. This works from the console where 
you started your program.
-Note that in case of ``nosetest`` you need to provide ``--nocapture`` flag to 
avoid nosetests
+Once you hit the line, you will be dropped into an interactive ``ipdb`` 
debugger where you have colors
+and autocompletion to guide your debugging. This works from the console where 
you started your program.
+Note that in case of ``nosetest`` you need to provide the ``--nocapture`` flag 
to avoid nosetests
 capturing the stdout of your process.
 
-Airflow directory structure inside Docker
--
-
-When you are in the container note that following directories are used:
-
-.. code-block:: text
-
-  /opt/airflow - here sources of Airflow are mounted from the host 
(AIRFLOW_SOURCES)
-  /root/airflow - all the "dynamic" Airflow files are created here: 
(AIRFLOW_HOME)
-  airflow.db - sqlite database in case sqlite is used
-  dags - folder where non-test dags are stored (test dags are in 
/opt/airflow/tests/dags)
-  logs - logs from airflow executions are created there
-  unittest.cfg - unit test configuration generated when entering the 
environment
-  webserver_config.py - webserver configuration generated when running 
airflow in the container
 
-Note that when run in your local environment ``/root/airflow/logs`` folder is 
actually mounted from your
-``logs`` directory in airflow sources, so all logs created in the container 
are automatically visible in the host
-as well. Every time you enter the container the logs directory is cleaned so 
that logs do not accumulate.
-
-Port forwarding
+Port Forwarding
 ---
 
 When you run Airflow Breeze, the following ports are automatically forwarded:
 
-* 28080 -> forwarded to airflow webserver -> airflow-testing:8080
-* 25433 -> forwarded to postgres database -> postgres:5432
-* 23306 -> forwarded to mysql database  -> mysql:3306
+* 28080 -> forwarded to Airflow webserver -> airflow-testing:8080
+* 25433 -> forwarded to Postgres database -> postgres:5432
+* 23306 -> forwarded to Mysql database  -> mysql:3306
 
-You can connect to those ports/databases using:
+You can connect to these ports/databases using:
 
 * Webserver: ``http://127.0.0.1:28080``
 * Postgres: 
``jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres=airflow``
 * Mysql: ``jdbc:mysql://localhost:23306/airflow?user=root``
 
-Note that you need to start the webserver manually with ``airflow webserver`` 
command if you want to connect
-to the webserver (you can use ``tmux`` to multiply terminals).
+Start the webserver manually with the ``airflow webserver`` command if you 
want to connect
+to the webserver. You can use ``tmux`` to multiply terminals.
 
-For databases you need to run ``airflow db reset`` at least once (or run some 
tests) after you started
-Airflow Breeze to get the database/tables created. You can connect to databases
-with IDE or any other Database client:
+For databases, you need to run ``airflow db reset`` at least once (or run some 
tests) after you started
+Airflow Breeze to get the database/tables created. You can connect to 
databases with IDE or any other database client:
 
 .. image:: images/database_view.png
 :align: center
 :alt: Database view
 
-You can change host port numbers used by setting appropriate environment 
variables:
+You can change the used host port numbers by setting appropriate environment 
variables:
 
 * ``WEBSERVER_HOST_PORT``
 * ``POSTGRES_HOST_PORT``
 * ``MYSQL_HOST_PORT``
 
-When you set those variables, next time when you enter the environment the new 
ports should be in effect.
+If you set these variables, next time when you enter the environment the new 
ports should be in effect.
 
-Cleaning up the images
---
+Building the Documentation
+--
 
-You might need to cleanup your Docker environment occasionally. The images are 
quite big
-(1.5GB for both images needed for static code analysis and CI tests). And if 
you often rebuild/update
-images you might end up with some unused image data.
+To build documentation in Breeze, use the ``-O``, ``--build-docs`` command:
 
-Cleanup can be performed with ``docker system prune`` command.
-Make sure to `Stop Breeze <#stopping-breeze>`_ first with ``./breeze 
--stop-environment``.
+.. code-block:: bash
 
-If you run into disk space errors, we recommend you prune your docker images 
using the
-``docker system prune --all`` command. You might need to restart the 

[GitHub] [airflow] potiuk commented on a change in pull request #6285: [AIRFLOW-XXX] Updates to Breeze documentation from GSOD

[GitBox]

potiuk commented on a change in pull request #6285: [AIRFLOW-XXX] Updates to 
Breeze documentation from GSOD
URL: https://github.com/apache/airflow/pull/6285#discussion_r334263723
 
 

 ##
 File path: BREEZE.rst
 ##
 @@ -290,165 +264,170 @@ It is as easy as copy this line into your code:
 
import ipdb; ipdb.set_trace()
 
-Once you hit the line you will be dropped into interactive ipdb  debugger 
where you have colors
-and auto-completion to guide your debugging. This works from the console where 
you started your program.
-Note that in case of ``nosetest`` you need to provide ``--nocapture`` flag to 
avoid nosetests
+Once you hit the line, you will be dropped into an interactive ``ipdb`` 
debugger where you have colors
+and autocompletion to guide your debugging. This works from the console where 
you started your program.
+Note that in case of ``nosetest`` you need to provide the ``--nocapture`` flag 
to avoid nosetests
 capturing the stdout of your process.
 
-Airflow directory structure inside Docker
--
-
-When you are in the container note that following directories are used:
-
-.. code-block:: text
-
-  /opt/airflow - here sources of Airflow are mounted from the host 
(AIRFLOW_SOURCES)
-  /root/airflow - all the "dynamic" Airflow files are created here: 
(AIRFLOW_HOME)
-  airflow.db - sqlite database in case sqlite is used
-  dags - folder where non-test dags are stored (test dags are in 
/opt/airflow/tests/dags)
-  logs - logs from airflow executions are created there
-  unittest.cfg - unit test configuration generated when entering the 
environment
-  webserver_config.py - webserver configuration generated when running 
airflow in the container
 
-Note that when run in your local environment ``/root/airflow/logs`` folder is 
actually mounted from your
-``logs`` directory in airflow sources, so all logs created in the container 
are automatically visible in the host
-as well. Every time you enter the container the logs directory is cleaned so 
that logs do not accumulate.
-
-Port forwarding
+Port Forwarding
 ---
 
 When you run Airflow Breeze, the following ports are automatically forwarded:
 
-* 28080 -> forwarded to airflow webserver -> airflow-testing:8080
-* 25433 -> forwarded to postgres database -> postgres:5432
-* 23306 -> forwarded to mysql database  -> mysql:3306
+* 28080 -> forwarded to Airflow webserver -> airflow-testing:8080
+* 25433 -> forwarded to Postgres database -> postgres:5432
+* 23306 -> forwarded to Mysql database  -> mysql:3306
 
-You can connect to those ports/databases using:
+You can connect to these ports/databases using:
 
 * Webserver: ``http://127.0.0.1:28080``
 * Postgres: 
``jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres=airflow``
 * Mysql: ``jdbc:mysql://localhost:23306/airflow?user=root``
 
-Note that you need to start the webserver manually with ``airflow webserver`` 
command if you want to connect
-to the webserver (you can use ``tmux`` to multiply terminals).
+Start the webserver manually with the ``airflow webserver`` command if you 
want to connect
+to the webserver. You can use ``tmux`` to multiply terminals.
 
-For databases you need to run ``airflow db reset`` at least once (or run some 
tests) after you started
-Airflow Breeze to get the database/tables created. You can connect to databases
-with IDE or any other Database client:
+For databases, you need to run ``airflow db reset`` at least once (or run some 
tests) after you started
+Airflow Breeze to get the database/tables created. You can connect to 
databases with IDE or any other database client:
 
 .. image:: images/database_view.png
 :align: center
 :alt: Database view
 
-You can change host port numbers used by setting appropriate environment 
variables:
+You can change the used host port numbers by setting appropriate environment 
variables:
 
 * ``WEBSERVER_HOST_PORT``
 * ``POSTGRES_HOST_PORT``
 * ``MYSQL_HOST_PORT``
 
-When you set those variables, next time when you enter the environment the new 
ports should be in effect.
+If you set these variables, next time when you enter the environment the new 
ports should be in effect.
 
-Cleaning up the images
---
+Building the Documentation
+--
 
-You might need to cleanup your Docker environment occasionally. The images are 
quite big
-(1.5GB for both images needed for static code analysis and CI tests). And if 
you often rebuild/update
-images you might end up with some unused image data.
+To build documentation in Breeze, use the ``-O``, ``--build-docs`` command:
 
-Cleanup can be performed with ``docker system prune`` command.
-Make sure to `Stop Breeze <#stopping-breeze>`_ first with ``./breeze 
--stop-environment``.
+.. code-block:: bash
 
-If you run into disk space errors, we recommend you prune your docker images 
using the
-``docker system prune --all`` command. You might need to restart the 

[GitHub] [airflow] potiuk commented on a change in pull request #6285: [AIRFLOW-XXX] Updates to Breeze documentation from GSOD

[GitBox]

potiuk commented on a change in pull request #6285: [AIRFLOW-XXX] Updates to 
Breeze documentation from GSOD
URL: https://github.com/apache/airflow/pull/6285#discussion_r334263709
 
 

 ##
 File path: BREEZE.rst
 ##
 @@ -474,61 +453,47 @@ Run pylint checks for all files:
  ./breeze --static-check-all-files pylint
 
 
-The ``license`` check is also run via separate script and separate docker 
image containing
+The ``license`` check is run via a separate script and a separate docker image 
containing the 
 Apache RAT verification tool that checks for Apache-compatibility of licences 
within the codebase.
-It does not take pre-commit parameters as extra args.
+It does not take pre-commit parameters as extra arguments.
 
 .. code-block:: bash
 
  ./breeze --static-check-all-files licenses
 
-Building the documentation
---
-
-The documentation is build using ``-O``, ``--build-docs`` command:
-
-.. code-block:: bash
-
- ./breeze --build-docs
-
-Results of the build can be found in ``docs/_build`` folder. Often errors 
during documentation generation
-come from the docstrings of auto-api generated classes. During the docs 
building auto-api generated
-files are stored in ``docs/_api`` folder - so that in case of problems with 
documentation you can
-find where the problems with documentation originated from.
-
-Running tests directly from host
-
+Running Tests Directly from the Host
 
 Review comment:
   It's bad naming indeed. The tests are still running in the container, but 
they are triggered from the host. You do not need to enter container separately 
for that - the script will start and enter container automatically.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-5646) ADD GcpSpeechToTextRecognizeLongRunningSpeechOperator

[taira yoshiki (Jira)]

taira yoshiki created AIRFLOW-5646:
--

 Summary: ADD GcpSpeechToTextRecognizeLongRunningSpeechOperator
 Key: AIRFLOW-5646
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5646
 Project: Apache Airflow
  Issue Type: New Feature
  Components: gcp, hooks, operators
Affects Versions: 1.10.5
Reporter: taira yoshiki


Hello, 

I want to add an operator corresponding to the following API.

https://speech.googleapis.com/v1/speech:longrunningrecognize



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] codecov-io edited a comment on issue #6313: [AIRFLOW-5643] Reduce duplicated logic in S3Hook

[GitBox]

codecov-io edited a comment on issue #6313: [AIRFLOW-5643] Reduce duplicated 
logic in S3Hook
URL: https://github.com/apache/airflow/pull/6313#issuecomment-541310309
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6313?src=pr=h1) 
Report
   > Merging 
[#6313](https://codecov.io/gh/apache/airflow/pull/6313?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/c0d98a72394b68df27db4d1cab4a94021980e0fb?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6313/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6313?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6313  +/-   ##
   ==
   + Coverage   80.34%   80.35%   +<.01% 
   ==
 Files 616  616  
 Lines   3573335724   -9 
   ==
   - Hits2871128706   -5 
   + Misses   7022 7018   -4
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6313?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/hooks/S3\_hook.py](https://codecov.io/gh/apache/airflow/pull/6313/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9TM19ob29rLnB5)
 | `95.62% <100%> (+1.35%)` | :arrow_up: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6313/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `56.55% <0%> (-0.35%)` | :arrow_down: |
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6313/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `93.77% <0%> (+0.5%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6313?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6313?src=pr=footer). 
Last update 
[c0d98a7...1fc26b2](https://codecov.io/gh/apache/airflow/pull/6313?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6223: [AIRFLOW-5578] Adding the ability to define custom IndexView

[GitBox]

codecov-io edited a comment on issue #6223: [AIRFLOW-5578] Adding the ability 
to define custom IndexView
URL: https://github.com/apache/airflow/pull/6223#issuecomment-537093847
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6223?src=pr=h1) 
Report
   > Merging 
[#6223](https://codecov.io/gh/apache/airflow/pull/6223?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/76fe45e1d127b657b1aad5c0fd657e011f5a09bc?src=pr=desc)
 will **decrease** coverage by `0.01%`.
   > The diff coverage is `88.88%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6223/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6223?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6223  +/-   ##
   ==
   - Coverage   80.05%   80.04%   -0.02% 
   ==
 Files 610  610  
 Lines   3526435293  +29 
   ==
   + Hits2823228249  +17 
   - Misses   7032 7044  +12
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6223?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[...rflow/config\_templates/default\_webserver\_config.py](https://codecov.io/gh/apache/airflow/pull/6223/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWdfdGVtcGxhdGVzL2RlZmF1bHRfd2Vic2VydmVyX2NvbmZpZy5weQ==)
 | `0% <ø> (ø)` | :arrow_up: |
   | 
[airflow/www/blueprints.py](https://codecov.io/gh/apache/airflow/pull/6223/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvYmx1ZXByaW50cy5weQ==)
 | `100% <100%> (ø)` | :arrow_up: |
   | 
[airflow/www/app.py](https://codecov.io/gh/apache/airflow/pull/6223/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvYXBwLnB5)
 | `95.9% <80%> (-0.66%)` | :arrow_down: |
   | 
[airflow/executors/dask\_executor.py](https://codecov.io/gh/apache/airflow/pull/6223/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvZGFza19leGVjdXRvci5weQ==)
 | `2% <0%> (-4%)` | :arrow_down: |
   | 
[airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6223/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==)
 | `89.9% <0%> (-1.53%)` | :arrow_down: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6223/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `56.37% <0%> (-0.52%)` | :arrow_down: |
   | 
[airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/6223/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5)
 | `95.27% <0%> (ø)` | :arrow_up: |
   | 
[...rflow/gcp/example\_dags/example\_gcp\_dlp\_operator.py](https://codecov.io/gh/apache/airflow/pull/6223/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfZ2NwX2RscF9vcGVyYXRvci5weQ==)
 | `0% <0%> (ø)` | :arrow_up: |
   | 
[...ample\_dags/example\_emr\_job\_flow\_automatic\_steps.py](https://codecov.io/gh/apache/airflow/pull/6223/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2Vtcl9qb2JfZmxvd19hdXRvbWF0aWNfc3RlcHMucHk=)
 | `0% <0%> (ø)` | :arrow_up: |
   | 
[airflow/utils/log/wasb\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/6223/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvd2FzYl90YXNrX2hhbmRsZXIucHk=)
 | `42.46% <0%> (ø)` | :arrow_up: |
   | ... and [218 
more](https://codecov.io/gh/apache/airflow/pull/6223/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6223?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6223?src=pr=footer). 
Last update 
[76fe45e...ddac251](https://codecov.io/gh/apache/airflow/pull/6223?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5249) BigQueryCheckOperator fails for datasets outside of 'US' region

[Thomas Pilewicz (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950139#comment-16950139
 ] 

Thomas Pilewicz commented on AIRFLOW-5249:
--

Actually, just tested with 1.10.5 and the issue doesn't show up. I can still 
try a fix, but would it make sense as it only affects 1.10.2?

> BigQueryCheckOperator fails for datasets outside of 'US' region
> ---
>
> Key: AIRFLOW-5249
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5249
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: operators
>Affects Versions: 1.10.2
>Reporter: Michael
>Assignee: Thomas Pilewicz
>Priority: Blocker
>
> When I try to use the BigQueryCheckOperator or BigQueryValueCheckOperator on 
> a dataset that is not in the 'US' location my task fails with the following 
> error
> {code:java}
> [2019-08-15 07:26:19,378] {__init__.py:1580} ERROR - BigQuery job status 
> check failed. Final error was: 404
> Traceback (most recent call last):
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
>  line 1241, in run_with_configuration
> jobId=self.running_job_id).execute()
>   File "/usr/local/lib/python3.6/site-packages/googleapiclient/_helpers.py", 
> line 130, in positional_wrapper
> return wrapped(*args, **kwargs)
>   File "/usr/local/lib/python3.6/site-packages/googleapiclient/http.py", line 
> 855, in execute
> raise HttpError(resp, content, uri=self.uri)
> googleapiclient.errors.HttpError:  https://www.googleapis.com/bigquery/v2/projects/anz-data-cde-airflow/jobs/job_ISDpiVtd7U1p-6N9wT378LfwoFHc?alt=json
>  returned "Not found: Job 
> anz-data-cde-airflow:job_ISDpiVtd7U1p-6N9wT378LfwoFHc">
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.6/site-packages/airflow/models/__init__.py", 
> line 1441, in _run_raw_task
> result = task_copy.execute(context=context)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/operators/check_operator.py", 
> line 81, in execute
> records = self.get_db_hook().get_first(self.sql)
>   File "/usr/local/lib/python3.6/site-packages/airflow/hooks/dbapi_hook.py", 
> line 138, in get_first
> cur.execute(sql)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
>  line 1821, in execute
> self.job_id = self.run_query(sql)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
>  line 849, in run_query
> return self.run_with_configuration(configuration)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
>  line 1263, in run_with_configuration
> format(err.resp.status))
> Exception: BigQuery job status check failed. Final error was: 404
> [2019-08-15 07:26:19,388] {__init__.py:1611} INFO - Marking task as FAILED.
> {code}
> This is the same error I get when I try to run the BigQuery operator without 
> specifying a location. When I run the same operator on a dataset that is in 
> the US region It succeeds.
> The BigQueryCheckOperator does not accept a location as one of its arguments 
> and does not pass a location to the BigQueryHook, I believe this is the 
> source of the problem. 
>  
> I realise a task (AIRFLOW-3601) was already created to fix a similar issue to 
> this one, but the referenced task calls out the two operators I'm having an 
> issue with as out of scope and after commenting on that task I have not 
> received a response.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] codecov-io edited a comment on issue #6285: [AIRFLOW-XXX] Updates to Breeze documentation from GSOD

[GitBox]

codecov-io edited a comment on issue #6285: [AIRFLOW-XXX] Updates to Breeze 
documentation from GSOD
URL: https://github.com/apache/airflow/pull/6285#issuecomment-539536232
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=h1) 
Report
   > Merging 
[#6285](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/0d71f335561800f117be41c575b20770f2345d50?src=pr=desc)
 will **increase** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6285/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6285  +/-   ##
   ==
   + Coverage   80.32%   80.34%   +0.01% 
   ==
 Files 612  616   +4 
 Lines   3539535733 +338 
   ==
   + Hits2843228710 +278 
   - Misses   6963 7023  +60
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==)
 | `58.89% <0%> (-6.34%)` | :arrow_down: |
   | 
[airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5)
 | `85% <0%> (-5%)` | :arrow_down: |
   | 
[airflow/models/variable.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdmFyaWFibGUucHk=)
 | `93.42% <0%> (-1.25%)` | :arrow_down: |
   | 
[airflow/gcp/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2JpZ3F1ZXJ5LnB5)
 | `86.6% <0%> (ø)` | :arrow_up: |
   | 
[...gle/marketing\_platform/sensors/campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlci9nb29nbGUvbWFya2V0aW5nX3BsYXRmb3JtL3NlbnNvcnMvY2FtcGFpZ25fbWFuYWdlci5weQ==)
 | `100% <0%> (ø)` | |
   | 
[...oogle/marketing\_platform/hooks/campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlci9nb29nbGUvbWFya2V0aW5nX3BsYXRmb3JtL2hvb2tzL2NhbXBhaWduX21hbmFnZXIucHk=)
 | `100% <0%> (ø)` | |
   | 
[...\_platform/example\_dags/example\_campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlci9nb29nbGUvbWFya2V0aW5nX3BsYXRmb3JtL2V4YW1wbGVfZGFncy9leGFtcGxlX2NhbXBhaWduX21hbmFnZXIucHk=)
 | `0% <0%> (ø)` | |
   | 
[...e/marketing\_platform/operators/campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlci9nb29nbGUvbWFya2V0aW5nX3BsYXRmb3JtL29wZXJhdG9ycy9jYW1wYWlnbl9tYW5hZ2VyLnB5)
 | `91.73% <0%> (ø)` | |
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `93.77% <0%> (+0.02%)` | :arrow_up: |
   | 
[airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5)
 | `66.21% <0%> (+0.03%)` | :arrow_up: |
   | ... and [10 
more](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=footer). 
Last update 
[0d71f33...b3a8a93](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6285: [AIRFLOW-XXX] Updates to Breeze documentation from GSOD

[GitBox]

codecov-io edited a comment on issue #6285: [AIRFLOW-XXX] Updates to Breeze 
documentation from GSOD
URL: https://github.com/apache/airflow/pull/6285#issuecomment-539536232
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=h1) 
Report
   > Merging 
[#6285](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/0d71f335561800f117be41c575b20770f2345d50?src=pr=desc)
 will **increase** coverage by `0.02%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6285/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6285  +/-   ##
   ==
   + Coverage   80.32%   80.35%   +0.02% 
   ==
 Files 612  616   +4 
 Lines   3539535733 +338 
   ==
   + Hits2843228713 +281 
   - Misses   6963 7020  +57
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==)
 | `58.89% <0%> (-6.34%)` | :arrow_down: |
   | 
[airflow/models/variable.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdmFyaWFibGUucHk=)
 | `93.42% <0%> (-1.25%)` | :arrow_down: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `56.72% <0%> (-0.18%)` | :arrow_down: |
   | 
[airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5)
 | `90% <0%> (ø)` | :arrow_up: |
   | 
[airflow/gcp/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2JpZ3F1ZXJ5LnB5)
 | `86.6% <0%> (ø)` | :arrow_up: |
   | 
[...gle/marketing\_platform/sensors/campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlci9nb29nbGUvbWFya2V0aW5nX3BsYXRmb3JtL3NlbnNvcnMvY2FtcGFpZ25fbWFuYWdlci5weQ==)
 | `100% <0%> (ø)` | |
   | 
[...oogle/marketing\_platform/hooks/campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlci9nb29nbGUvbWFya2V0aW5nX3BsYXRmb3JtL2hvb2tzL2NhbXBhaWduX21hbmFnZXIucHk=)
 | `100% <0%> (ø)` | |
   | 
[...\_platform/example\_dags/example\_campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlci9nb29nbGUvbWFya2V0aW5nX3BsYXRmb3JtL2V4YW1wbGVfZGFncy9leGFtcGxlX2NhbXBhaWduX21hbmFnZXIucHk=)
 | `0% <0%> (ø)` | |
   | 
[...e/marketing\_platform/operators/campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlci9nb29nbGUvbWFya2V0aW5nX3BsYXRmb3JtL29wZXJhdG9ycy9jYW1wYWlnbl9tYW5hZ2VyLnB5)
 | `91.73% <0%> (ø)` | |
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `93.77% <0%> (+0.02%)` | :arrow_up: |
   | ... and [11 
more](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=footer). 
Last update 
[0d71f33...67d503b](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6285: [AIRFLOW-XXX] Updates to Breeze documentation from GSOD

[GitBox]

codecov-io edited a comment on issue #6285: [AIRFLOW-XXX] Updates to Breeze 
documentation from GSOD
URL: https://github.com/apache/airflow/pull/6285#issuecomment-539536232
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=h1) 
Report
   > Merging 
[#6285](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/0d71f335561800f117be41c575b20770f2345d50?src=pr=desc)
 will **increase** coverage by `0.02%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6285/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6285  +/-   ##
   ==
   + Coverage   80.32%   80.35%   +0.02% 
   ==
 Files 612  616   +4 
 Lines   3539535733 +338 
   ==
   + Hits2843228713 +281 
   - Misses   6963 7020  +57
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==)
 | `58.89% <0%> (-6.34%)` | :arrow_down: |
   | 
[airflow/models/variable.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdmFyaWFibGUucHk=)
 | `93.42% <0%> (-1.25%)` | :arrow_down: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `56.72% <0%> (-0.18%)` | :arrow_down: |
   | 
[airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5)
 | `90% <0%> (ø)` | :arrow_up: |
   | 
[airflow/gcp/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2JpZ3F1ZXJ5LnB5)
 | `86.6% <0%> (ø)` | :arrow_up: |
   | 
[...gle/marketing\_platform/sensors/campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlci9nb29nbGUvbWFya2V0aW5nX3BsYXRmb3JtL3NlbnNvcnMvY2FtcGFpZ25fbWFuYWdlci5weQ==)
 | `100% <0%> (ø)` | |
   | 
[...oogle/marketing\_platform/hooks/campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlci9nb29nbGUvbWFya2V0aW5nX3BsYXRmb3JtL2hvb2tzL2NhbXBhaWduX21hbmFnZXIucHk=)
 | `100% <0%> (ø)` | |
   | 
[...\_platform/example\_dags/example\_campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlci9nb29nbGUvbWFya2V0aW5nX3BsYXRmb3JtL2V4YW1wbGVfZGFncy9leGFtcGxlX2NhbXBhaWduX21hbmFnZXIucHk=)
 | `0% <0%> (ø)` | |
   | 
[...e/marketing\_platform/operators/campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlci9nb29nbGUvbWFya2V0aW5nX3BsYXRmb3JtL29wZXJhdG9ycy9jYW1wYWlnbl9tYW5hZ2VyLnB5)
 | `91.73% <0%> (ø)` | |
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `93.77% <0%> (+0.02%)` | :arrow_up: |
   | ... and [11 
more](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=footer). 
Last update 
[0d71f33...67d503b](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6285: [AIRFLOW-XXX] Updates to Breeze documentation from GSOD

[GitBox]

codecov-io edited a comment on issue #6285: [AIRFLOW-XXX] Updates to Breeze 
documentation from GSOD
URL: https://github.com/apache/airflow/pull/6285#issuecomment-539536232
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=h1) 
Report
   > Merging 
[#6285](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/0d71f335561800f117be41c575b20770f2345d50?src=pr=desc)
 will **increase** coverage by `0.02%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6285/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6285  +/-   ##
   ==
   + Coverage   80.32%   80.35%   +0.02% 
   ==
 Files 612  616   +4 
 Lines   3539535733 +338 
   ==
   + Hits2843228713 +281 
   - Misses   6963 7020  +57
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==)
 | `58.89% <0%> (-6.34%)` | :arrow_down: |
   | 
[airflow/models/variable.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdmFyaWFibGUucHk=)
 | `93.42% <0%> (-1.25%)` | :arrow_down: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `56.72% <0%> (-0.18%)` | :arrow_down: |
   | 
[airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5)
 | `90% <0%> (ø)` | :arrow_up: |
   | 
[airflow/gcp/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2JpZ3F1ZXJ5LnB5)
 | `86.6% <0%> (ø)` | :arrow_up: |
   | 
[...gle/marketing\_platform/sensors/campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlci9nb29nbGUvbWFya2V0aW5nX3BsYXRmb3JtL3NlbnNvcnMvY2FtcGFpZ25fbWFuYWdlci5weQ==)
 | `100% <0%> (ø)` | |
   | 
[...oogle/marketing\_platform/hooks/campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlci9nb29nbGUvbWFya2V0aW5nX3BsYXRmb3JtL2hvb2tzL2NhbXBhaWduX21hbmFnZXIucHk=)
 | `100% <0%> (ø)` | |
   | 
[...\_platform/example\_dags/example\_campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlci9nb29nbGUvbWFya2V0aW5nX3BsYXRmb3JtL2V4YW1wbGVfZGFncy9leGFtcGxlX2NhbXBhaWduX21hbmFnZXIucHk=)
 | `0% <0%> (ø)` | |
   | 
[...e/marketing\_platform/operators/campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlci9nb29nbGUvbWFya2V0aW5nX3BsYXRmb3JtL29wZXJhdG9ycy9jYW1wYWlnbl9tYW5hZ2VyLnB5)
 | `91.73% <0%> (ø)` | |
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `93.77% <0%> (+0.02%)` | :arrow_up: |
   | ... and [11 
more](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=footer). 
Last update 
[0d71f33...67d503b](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6285: [AIRFLOW-XXX] Updates to Breeze documentation from GSOD

[GitBox]

codecov-io edited a comment on issue #6285: [AIRFLOW-XXX] Updates to Breeze 
documentation from GSOD
URL: https://github.com/apache/airflow/pull/6285#issuecomment-539536232
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=h1) 
Report
   > Merging 
[#6285](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/0d71f335561800f117be41c575b20770f2345d50?src=pr=desc)
 will **increase** coverage by `0.02%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6285/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6285  +/-   ##
   ==
   + Coverage   80.32%   80.35%   +0.02% 
   ==
 Files 612  616   +4 
 Lines   3539535733 +338 
   ==
   + Hits2843228713 +281 
   - Misses   6963 7020  +57
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==)
 | `58.89% <0%> (-6.34%)` | :arrow_down: |
   | 
[airflow/models/variable.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdmFyaWFibGUucHk=)
 | `93.42% <0%> (-1.25%)` | :arrow_down: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `56.72% <0%> (-0.18%)` | :arrow_down: |
   | 
[airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5)
 | `90% <0%> (ø)` | :arrow_up: |
   | 
[airflow/gcp/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2JpZ3F1ZXJ5LnB5)
 | `86.6% <0%> (ø)` | :arrow_up: |
   | 
[...gle/marketing\_platform/sensors/campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlci9nb29nbGUvbWFya2V0aW5nX3BsYXRmb3JtL3NlbnNvcnMvY2FtcGFpZ25fbWFuYWdlci5weQ==)
 | `100% <0%> (ø)` | |
   | 
[...oogle/marketing\_platform/hooks/campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlci9nb29nbGUvbWFya2V0aW5nX3BsYXRmb3JtL2hvb2tzL2NhbXBhaWduX21hbmFnZXIucHk=)
 | `100% <0%> (ø)` | |
   | 
[...\_platform/example\_dags/example\_campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlci9nb29nbGUvbWFya2V0aW5nX3BsYXRmb3JtL2V4YW1wbGVfZGFncy9leGFtcGxlX2NhbXBhaWduX21hbmFnZXIucHk=)
 | `0% <0%> (ø)` | |
   | 
[...e/marketing\_platform/operators/campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlci9nb29nbGUvbWFya2V0aW5nX3BsYXRmb3JtL29wZXJhdG9ycy9jYW1wYWlnbl9tYW5hZ2VyLnB5)
 | `91.73% <0%> (ø)` | |
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `93.77% <0%> (+0.02%)` | :arrow_up: |
   | ... and [11 
more](https://codecov.io/gh/apache/airflow/pull/6285/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=footer). 
Last update 
[0d71f33...67d503b](https://codecov.io/gh/apache/airflow/pull/6285?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] efedotova commented on a change in pull request #6285: [AIRFLOW-XXX] Updates to Breeze documentation from GSOD

[GitBox]

efedotova commented on a change in pull request #6285: [AIRFLOW-XXX] Updates to 
Breeze documentation from GSOD
URL: https://github.com/apache/airflow/pull/6285#discussion_r334251058
 
 

 ##
 File path: BREEZE.rst
 ##
 @@ -290,165 +264,170 @@ It is as easy as copy this line into your code:
 
import ipdb; ipdb.set_trace()
 
-Once you hit the line you will be dropped into interactive ipdb  debugger 
where you have colors
-and auto-completion to guide your debugging. This works from the console where 
you started your program.
-Note that in case of ``nosetest`` you need to provide ``--nocapture`` flag to 
avoid nosetests
+Once you hit the line, you will be dropped into an interactive ``ipdb`` 
debugger where you have colors
+and autocompletion to guide your debugging. This works from the console where 
you started your program.
+Note that in case of ``nosetest`` you need to provide the ``--nocapture`` flag 
to avoid nosetests
 capturing the stdout of your process.
 
-Airflow directory structure inside Docker
--
-
-When you are in the container note that following directories are used:
-
-.. code-block:: text
-
-  /opt/airflow - here sources of Airflow are mounted from the host 
(AIRFLOW_SOURCES)
-  /root/airflow - all the "dynamic" Airflow files are created here: 
(AIRFLOW_HOME)
-  airflow.db - sqlite database in case sqlite is used
-  dags - folder where non-test dags are stored (test dags are in 
/opt/airflow/tests/dags)
-  logs - logs from airflow executions are created there
-  unittest.cfg - unit test configuration generated when entering the 
environment
-  webserver_config.py - webserver configuration generated when running 
airflow in the container
 
-Note that when run in your local environment ``/root/airflow/logs`` folder is 
actually mounted from your
-``logs`` directory in airflow sources, so all logs created in the container 
are automatically visible in the host
-as well. Every time you enter the container the logs directory is cleaned so 
that logs do not accumulate.
-
-Port forwarding
+Port Forwarding
 ---
 
 When you run Airflow Breeze, the following ports are automatically forwarded:
 
-* 28080 -> forwarded to airflow webserver -> airflow-testing:8080
-* 25433 -> forwarded to postgres database -> postgres:5432
-* 23306 -> forwarded to mysql database  -> mysql:3306
+* 28080 -> forwarded to Airflow webserver -> airflow-testing:8080
+* 25433 -> forwarded to Postgres database -> postgres:5432
+* 23306 -> forwarded to Mysql database  -> mysql:3306
 
-You can connect to those ports/databases using:
+You can connect to these ports/databases using:
 
 * Webserver: ``http://127.0.0.1:28080``
 * Postgres: 
``jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres=airflow``
 * Mysql: ``jdbc:mysql://localhost:23306/airflow?user=root``
 
-Note that you need to start the webserver manually with ``airflow webserver`` 
command if you want to connect
-to the webserver (you can use ``tmux`` to multiply terminals).
+Start the webserver manually with the ``airflow webserver`` command if you 
want to connect
+to the webserver. You can use ``tmux`` to multiply terminals.
 
-For databases you need to run ``airflow db reset`` at least once (or run some 
tests) after you started
-Airflow Breeze to get the database/tables created. You can connect to databases
-with IDE or any other Database client:
+For databases, you need to run ``airflow db reset`` at least once (or run some 
tests) after you started
+Airflow Breeze to get the database/tables created. You can connect to 
databases with IDE or any other database client:
 
 .. image:: images/database_view.png
 :align: center
 :alt: Database view
 
-You can change host port numbers used by setting appropriate environment 
variables:
+You can change the used host port numbers by setting appropriate environment 
variables:
 
 * ``WEBSERVER_HOST_PORT``
 * ``POSTGRES_HOST_PORT``
 * ``MYSQL_HOST_PORT``
 
-When you set those variables, next time when you enter the environment the new 
ports should be in effect.
+If you set these variables, next time when you enter the environment the new 
ports should be in effect.
 
-Cleaning up the images
---
+Building the Documentation
+--
 
-You might need to cleanup your Docker environment occasionally. The images are 
quite big
-(1.5GB for both images needed for static code analysis and CI tests). And if 
you often rebuild/update
-images you might end up with some unused image data.
+To build documentation in Breeze, use the ``-O``, ``--build-docs`` command:
 
-Cleanup can be performed with ``docker system prune`` command.
-Make sure to `Stop Breeze <#stopping-breeze>`_ first with ``./breeze 
--stop-environment``.
+.. code-block:: bash
 
-If you run into disk space errors, we recommend you prune your docker images 
using the
-``docker system prune --all`` command. You might need to restart 

[GitHub] [airflow] efedotova commented on a change in pull request #6285: [AIRFLOW-XXX] Updates to Breeze documentation from GSOD

[GitBox]

efedotova commented on a change in pull request #6285: [AIRFLOW-XXX] Updates to 
Breeze documentation from GSOD
URL: https://github.com/apache/airflow/pull/6285#discussion_r334251004
 
 

 ##
 File path: BREEZE.rst
 ##
 @@ -290,165 +264,170 @@ It is as easy as copy this line into your code:
 
import ipdb; ipdb.set_trace()
 
-Once you hit the line you will be dropped into interactive ipdb  debugger 
where you have colors
-and auto-completion to guide your debugging. This works from the console where 
you started your program.
-Note that in case of ``nosetest`` you need to provide ``--nocapture`` flag to 
avoid nosetests
+Once you hit the line, you will be dropped into an interactive ``ipdb`` 
debugger where you have colors
+and autocompletion to guide your debugging. This works from the console where 
you started your program.
+Note that in case of ``nosetest`` you need to provide the ``--nocapture`` flag 
to avoid nosetests
 capturing the stdout of your process.
 
-Airflow directory structure inside Docker
--
-
-When you are in the container note that following directories are used:
-
-.. code-block:: text
-
-  /opt/airflow - here sources of Airflow are mounted from the host 
(AIRFLOW_SOURCES)
-  /root/airflow - all the "dynamic" Airflow files are created here: 
(AIRFLOW_HOME)
-  airflow.db - sqlite database in case sqlite is used
-  dags - folder where non-test dags are stored (test dags are in 
/opt/airflow/tests/dags)
-  logs - logs from airflow executions are created there
-  unittest.cfg - unit test configuration generated when entering the 
environment
-  webserver_config.py - webserver configuration generated when running 
airflow in the container
 
-Note that when run in your local environment ``/root/airflow/logs`` folder is 
actually mounted from your
-``logs`` directory in airflow sources, so all logs created in the container 
are automatically visible in the host
-as well. Every time you enter the container the logs directory is cleaned so 
that logs do not accumulate.
-
-Port forwarding
+Port Forwarding
 ---
 
 When you run Airflow Breeze, the following ports are automatically forwarded:
 
-* 28080 -> forwarded to airflow webserver -> airflow-testing:8080
-* 25433 -> forwarded to postgres database -> postgres:5432
-* 23306 -> forwarded to mysql database  -> mysql:3306
+* 28080 -> forwarded to Airflow webserver -> airflow-testing:8080
+* 25433 -> forwarded to Postgres database -> postgres:5432
+* 23306 -> forwarded to Mysql database  -> mysql:3306
 
-You can connect to those ports/databases using:
+You can connect to these ports/databases using:
 
 * Webserver: ``http://127.0.0.1:28080``
 * Postgres: 
``jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres=airflow``
 * Mysql: ``jdbc:mysql://localhost:23306/airflow?user=root``
 
-Note that you need to start the webserver manually with ``airflow webserver`` 
command if you want to connect
-to the webserver (you can use ``tmux`` to multiply terminals).
+Start the webserver manually with the ``airflow webserver`` command if you 
want to connect
+to the webserver. You can use ``tmux`` to multiply terminals.
 
-For databases you need to run ``airflow db reset`` at least once (or run some 
tests) after you started
-Airflow Breeze to get the database/tables created. You can connect to databases
-with IDE or any other Database client:
+For databases, you need to run ``airflow db reset`` at least once (or run some 
tests) after you started
+Airflow Breeze to get the database/tables created. You can connect to 
databases with IDE or any other database client:
 
 .. image:: images/database_view.png
 :align: center
 :alt: Database view
 
-You can change host port numbers used by setting appropriate environment 
variables:
+You can change the used host port numbers by setting appropriate environment 
variables:
 
 * ``WEBSERVER_HOST_PORT``
 * ``POSTGRES_HOST_PORT``
 * ``MYSQL_HOST_PORT``
 
-When you set those variables, next time when you enter the environment the new 
ports should be in effect.
+If you set these variables, next time when you enter the environment the new 
ports should be in effect.
 
-Cleaning up the images
---
+Building the Documentation
+--
 
-You might need to cleanup your Docker environment occasionally. The images are 
quite big
-(1.5GB for both images needed for static code analysis and CI tests). And if 
you often rebuild/update
-images you might end up with some unused image data.
+To build documentation in Breeze, use the ``-O``, ``--build-docs`` command:
 
-Cleanup can be performed with ``docker system prune`` command.
-Make sure to `Stop Breeze <#stopping-breeze>`_ first with ``./breeze 
--stop-environment``.
+.. code-block:: bash
 
-If you run into disk space errors, we recommend you prune your docker images 
using the
-``docker system prune --all`` command. You might need to restart 

[GitHub] [airflow] efedotova commented on a change in pull request #6285: [AIRFLOW-XXX] Updates to Breeze documentation from GSOD

[GitBox]

efedotova commented on a change in pull request #6285: [AIRFLOW-XXX] Updates to 
Breeze documentation from GSOD
URL: https://github.com/apache/airflow/pull/6285#discussion_r334250868
 
 

 ##
 File path: BREEZE.rst
 ##
 @@ -474,61 +453,47 @@ Run pylint checks for all files:
  ./breeze --static-check-all-files pylint
 
 
-The ``license`` check is also run via separate script and separate docker 
image containing
+The ``license`` check is run via a separate script and a separate docker image 
containing the 
 Apache RAT verification tool that checks for Apache-compatibility of licences 
within the codebase.
-It does not take pre-commit parameters as extra args.
+It does not take pre-commit parameters as extra arguments.
 
 .. code-block:: bash
 
  ./breeze --static-check-all-files licenses
 
-Building the documentation
---
-
-The documentation is build using ``-O``, ``--build-docs`` command:
-
-.. code-block:: bash
-
- ./breeze --build-docs
-
-Results of the build can be found in ``docs/_build`` folder. Often errors 
during documentation generation
-come from the docstrings of auto-api generated classes. During the docs 
building auto-api generated
-files are stored in ``docs/_api`` folder - so that in case of problems with 
documentation you can
-find where the problems with documentation originated from.
-
-Running tests directly from host
-
+Running Tests Directly from the Host
 
 Review comment:
   I don't see any 'from the host' specifics below. does it mean running tests 
outside the container?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] efedotova commented on a change in pull request #6285: [AIRFLOW-XXX] Updates to Breeze documentation from GSOD

[GitBox]

efedotova commented on a change in pull request #6285: [AIRFLOW-XXX] Updates to 
Breeze documentation from GSOD
URL: https://github.com/apache/airflow/pull/6285#discussion_r334250709
 
 

 ##
 File path: BREEZE.rst
 ##
 @@ -474,61 +453,47 @@ Run pylint checks for all files:
  ./breeze --static-check-all-files pylint
 
 
-The ``license`` check is also run via separate script and separate docker 
image containing
+The ``license`` check is run via a separate script and a separate docker image 
containing the 
 Apache RAT verification tool that checks for Apache-compatibility of licences 
within the codebase.
-It does not take pre-commit parameters as extra args.
+It does not take pre-commit parameters as extra arguments.
 
 .. code-block:: bash
 
  ./breeze --static-check-all-files licenses
 
-Building the documentation
---
-
-The documentation is build using ``-O``, ``--build-docs`` command:
-
-.. code-block:: bash
-
- ./breeze --build-docs
-
-Results of the build can be found in ``docs/_build`` folder. Often errors 
during documentation generation
-come from the docstrings of auto-api generated classes. During the docs 
building auto-api generated
-files are stored in ``docs/_api`` folder - so that in case of problems with 
documentation you can
-find where the problems with documentation originated from.
-
-Running tests directly from host
-
+Running Tests Directly from the Host
+
 
-If you wish to run tests only and not drop into shell, you can run them by 
providing
--t, --test-target flag. You can add extra nosetest flags after -- in the 
commandline.
+If you wish to only run tests and not to drop into shell, you can do this by 
providing the
+-t, --test-target flag. You can add extra nosetest flags after -- in the 
command line.
 
 .. code-block:: bash
 
  ./breeze --test-target tests/hooks/test_druid_hook.py -- 
--logging-level=DEBUG
 
-You can run the whole test suite with special '.' test target:
+You can run the whole test suite with a special '.' test target:
 
 .. code-block:: bash
 
 ./breeze --test-target .
 
-You can also specify individual tests or group of tests:
+You can also specify individual tests or a group of tests:
 
 .. code-block:: bash
 
 ./breeze --test-target tests.core:TestCore
 
-Pulling the latest images
+Pulling the Latest Images
 -
 
-Sometimes the image on DockerHub is rebuilt from the scratch. This happens for 
example when there is a
-security update of the python version that all the images are based on.
-In this case it is usually faster to pull latest images rather than rebuild 
them
-from the scratch.
+Sometimes the image on the Docker Hub needs to be rebuilt from scratch. This 
is required, for example, when there is a
 
 Review comment:
   not sure my edits are correct here. the original version sounded like the 
image rebuilt is done automatically.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] jojo19893 commented on a change in pull request #6303: AIRFLOW-5126 Read aws_session_token in extra_config of the aws hook

[GitBox]

jojo19893 commented on a change in pull request #6303: AIRFLOW-5126 Read 
aws_session_token in extra_config of the aws hook
URL: https://github.com/apache/airflow/pull/6303#discussion_r334250199
 
 

 ##
 File path: airflow/contrib/hooks/aws_hook.py
 ##
 @@ -132,7 +134,7 @@ def _get_credentials(self, region_name):
 role_arn = "arn:aws:iam::{}:role/{}" \
 .format(aws_account_id, aws_iam_role)
 
-if role_arn is not None:
+if role_arn is not None and aws_session_token is None:
 
 Review comment:
   Not sure if you want to obtain a token if you already have one ? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #6317: [AIRFLOW-5644] Simplify TriggerDagRunOperator usage

[GitBox]

codecov-io commented on issue #6317: [AIRFLOW-5644] Simplify 
TriggerDagRunOperator usage
URL: https://github.com/apache/airflow/pull/6317#issuecomment-541349899
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6317?src=pr=h1) 
Report
   > Merging 
[#6317](https://codecov.io/gh/apache/airflow/pull/6317?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/c0d98a72394b68df27db4d1cab4a94021980e0fb?src=pr=desc)
 will **decrease** coverage by `0.07%`.
   > The diff coverage is `92.85%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6317/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6317?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6317  +/-   ##
   ==
   - Coverage   80.34%   80.27%   -0.08% 
   ==
 Files 616  616  
 Lines   3573335710  -23 
   ==
   - Hits2871128667  -44 
   - Misses   7022 7043  +21
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6317?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[...low/example\_dags/example\_trigger\_controller\_dag.py](https://codecov.io/gh/apache/airflow/pull/6317/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV90cmlnZ2VyX2NvbnRyb2xsZXJfZGFnLnB5)
 | `100% <100%> (+43.75%)` | :arrow_up: |
   | 
[airflow/operators/dagrun\_operator.py](https://codecov.io/gh/apache/airflow/pull/6317/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZGFncnVuX29wZXJhdG9yLnB5)
 | `96.42% <100%> (+1.69%)` | :arrow_up: |
   | 
[airflow/utils/dates.py](https://codecov.io/gh/apache/airflow/pull/6317/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYXRlcy5weQ==)
 | `82.6% <100%> (ø)` | :arrow_up: |
   | 
[airflow/example\_dags/example\_trigger\_target\_dag.py](https://codecov.io/gh/apache/airflow/pull/6317/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV90cmlnZ2VyX3RhcmdldF9kYWcucHk=)
 | `90% <75%> (-2.31%)` | :arrow_down: |
   | 
[airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6317/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6317/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5)
 | `85% <0%> (-5%)` | :arrow_down: |
   | 
[airflow/hooks/postgres\_hook.py](https://codecov.io/gh/apache/airflow/pull/6317/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9wb3N0Z3Jlc19ob29rLnB5)
 | `94.73% <0%> (-1.76%)` | :arrow_down: |
   | 
[airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6317/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5)
 | `91.52% <0%> (-1.7%)` | :arrow_down: |
   | 
[airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/6317/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5)
 | `86.44% <0%> (-1.7%)` | :arrow_down: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6317/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `56.55% <0%> (-0.35%)` | :arrow_down: |
   | ... and [1 
more](https://codecov.io/gh/apache/airflow/pull/6317/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6317?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6317?src=pr=footer). 
Last update 
[c0d98a7...34fb393](https://codecov.io/gh/apache/airflow/pull/6317?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feluelle commented on a change in pull request #6314: [AIRFLOW-5497] Update docstring with param description

[GitBox]

feluelle commented on a change in pull request #6314: [AIRFLOW-5497] Update 
docstring with param description
URL: https://github.com/apache/airflow/pull/6314#discussion_r334248305
 
 

 ##
 File path: airflow/utils/dag_processing.py
 ##
 @@ -298,6 +298,8 @@ def list_py_file_paths(directory, 
safe_mode=conf.getboolean('core', 'DAG_DISCOVE
 contains Airflow DAG definitions. If not provided, use the
 core.DAG_DISCOVERY_SAFE_MODE configuration setting. If not set, default
 to safe.
 
 Review comment:
   ```suggestion
to safe.
   :type safe_mode: bool
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #6314: [AIRFLOW-5497] Update docstring with param description

[GitBox]

codecov-io commented on issue #6314: [AIRFLOW-5497] Update docstring with param 
description
URL: https://github.com/apache/airflow/pull/6314#issuecomment-541345431
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6314?src=pr=h1) 
Report
   > Merging 
[#6314](https://codecov.io/gh/apache/airflow/pull/6314?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/c0d98a72394b68df27db4d1cab4a94021980e0fb?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6314/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6314?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6314  +/-   ##
   ==
   + Coverage   80.34%   80.35%   +<.01% 
   ==
 Files 616  616  
 Lines   3573335733  
   ==
   + Hits2871128712   +1 
   + Misses   7022 7021   -1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6314?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6314/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `56.55% <ø> (-0.35%)` | :arrow_down: |
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6314/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `93.77% <0%> (+0.5%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6314?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6314?src=pr=footer). 
Last update 
[c0d98a7...14dab2e](https://codecov.io/gh/apache/airflow/pull/6314?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #6316: [AIRFLOW-5645] Precise webserver status monitoring possibilities in quickstart

[GitBox]

codecov-io commented on issue #6316: [AIRFLOW-5645] Precise webserver status 
monitoring possibilities in quickstart
URL: https://github.com/apache/airflow/pull/6316#issuecomment-541344369
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6316?src=pr=h1) 
Report
   > Merging 
[#6316](https://codecov.io/gh/apache/airflow/pull/6316?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/c0d98a72394b68df27db4d1cab4a94021980e0fb?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6316/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6316?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6316  +/-   ##
   ==
   + Coverage   80.34%   80.35%   +<.01% 
   ==
 Files 616  616  
 Lines   3573335733  
   ==
   + Hits2871128713   +2 
   + Misses   7022 7020   -2
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6316?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6316/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `56.72% <0%> (-0.18%)` | :arrow_down: |
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6316/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `93.77% <0%> (+0.5%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6316?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6316?src=pr=footer). 
Last update 
[c0d98a7...a378ac9](https://codecov.io/gh/apache/airflow/pull/6316?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5644) Simplify TriggerDagRunOperator usage

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950089#comment-16950089
 ] 

ASF GitHub Bot commented on AIRFLOW-5644:
-

BasPH commented on pull request #6317: [AIRFLOW-5644] Simplify 
TriggerDagRunOperator usage
URL: https://github.com/apache/airflow/pull/6317
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-5644
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   This PR refactors the TriggerDagRunOperator to provide a much more intuitive 
behaviour, i.e. it now has a `conf` argument to which a dict can be provided, 
for configuration for the triggered Dag(Run).
   
   Before:
   
   ```python
   def _trigger_task(context, object):
   object.payload = {"message": "Hello world"}
   return object
   
   trigger_task = TriggerDagRunOperator(
   task_id='test_trigger_dagrun',
   trigger_dag_id="example_trigger_target_dag",
   python_callable=_trigger_task,
   dag=dag,
   )
   ```
   
   After:
   
   ```python
   trigger_task = TriggerDagRunOperator(
   task_id='test_trigger_dagrun',
   trigger_dag_id="example_trigger_target_dag",
   conf={"message": "Hello world"},
   dag=dag,
   )
   ```
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   TriggerDagRunOperator tests were extracted from core.py and placed in a 
dedicated test_dagrun_operator.py file. I added additional tests for validating 
correct behaviour.
   
   These tests were a bit tricky because they rely on passing state via the 
database. To make these idempotent and not rely on external files (i.e. example 
DAGs), the `setUp()` writes a small DAG to a temporary file, which is used 
throughout the tests, and in the `tearDown()` all state is removed from the DB.
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Simplify TriggerDagRunOperator usage
> 
>
> Key: AIRFLOW-5644
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5644
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: operators
>Affects Versions: 2.0.0
>Reporter: Bas Harenslak
>Priority: Major
>
> The TriggerDagRunOperator usage is rather odd at the moment, especially the 
> way to pass a conf.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] codecov-io commented on issue #6315: [AIRFLOW-5640] fix get_email_address_list types

[GitBox]

codecov-io commented on issue #6315: [AIRFLOW-5640] fix get_email_address_list 
types
URL: https://github.com/apache/airflow/pull/6315#issuecomment-541341253
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6315?src=pr=h1) 
Report
   > Merging 
[#6315](https://codecov.io/gh/apache/airflow/pull/6315?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/c0d98a72394b68df27db4d1cab4a94021980e0fb?src=pr=desc)
 will **decrease** coverage by `0.3%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6315/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6315?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6315  +/-   ##
   ==
   - Coverage   80.34%   80.04%   -0.31% 
   ==
 Files 616  616  
 Lines   3573335742   +9 
   ==
   - Hits2871128611 -100 
   - Misses   7022 7131 +109
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6315?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/6315/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5)
 | `95.53% <ø> (ø)` | :arrow_up: |
   | 
[airflow/utils/email.py](https://codecov.io/gh/apache/airflow/pull/6315/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9lbWFpbC5weQ==)
 | `100% <100%> (ø)` | :arrow_up: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6315/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6315/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6315/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `45.25% <0%> (-46.72%)` | :arrow_down: |
   | 
[airflow/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/6315/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2t1YmVfY2xpZW50LnB5)
 | `33.33% <0%> (-41.67%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6315/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `70.14% <0%> (-28.36%)` | :arrow_down: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6315/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `56.72% <0%> (-0.18%)` | :arrow_down: |
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6315/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `93.77% <0%> (+0.5%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6315?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6315?src=pr=footer). 
Last update 
[c0d98a7...23bc2c5](https://codecov.io/gh/apache/airflow/pull/6315?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] BasPH opened a new pull request #6317: [AIRFLOW-5644] Simplify TriggerDagRunOperator usage

[GitBox]

BasPH opened a new pull request #6317: [AIRFLOW-5644] Simplify 
TriggerDagRunOperator usage
URL: https://github.com/apache/airflow/pull/6317
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-5644
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   This PR refactors the TriggerDagRunOperator to provide a much more intuitive 
behaviour, i.e. it now has a `conf` argument to which a dict can be provided, 
for configuration for the triggered Dag(Run).
   
   Before:
   
   ```python
   def _trigger_task(context, object):
   object.payload = {"message": "Hello world"}
   return object
   
   trigger_task = TriggerDagRunOperator(
   task_id='test_trigger_dagrun',
   trigger_dag_id="example_trigger_target_dag",
   python_callable=_trigger_task,
   dag=dag,
   )
   ```
   
   After:
   
   ```python
   trigger_task = TriggerDagRunOperator(
   task_id='test_trigger_dagrun',
   trigger_dag_id="example_trigger_target_dag",
   conf={"message": "Hello world"},
   dag=dag,
   )
   ```
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   TriggerDagRunOperator tests were extracted from core.py and placed in a 
dedicated test_dagrun_operator.py file. I added additional tests for validating 
correct behaviour.
   
   These tests were a bit tricky because they rely on passing state via the 
database. To make these idempotent and not rely on external files (i.e. example 
DAGs), the `setUp()` writes a small DAG to a temporary file, which is used 
throughout the tests, and in the `tearDown()` all state is removed from the DB.
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] milton0825 commented on a change in pull request #6309: [AIRFLOW-3783][AIRFLOW-5395] Switch to HEADER arg to unload data from Redshift to S3

[GitBox]

milton0825 commented on a change in pull request #6309: 
[AIRFLOW-3783][AIRFLOW-5395] Switch to HEADER arg to unload data from Redshift 
to S3
URL: https://github.com/apache/airflow/pull/6309#discussion_r334244845
 
 

 ##
 File path: airflow/operators/redshift_to_s3_operator.py
 ##
 @@ -85,52 +93,16 @@ def __init__(
 self.autocommit = autocommit
 self.include_header = include_header
 
-if self.include_header and 'PARALLEL OFF' not in [uo.upper().strip() 
for uo in self.unload_options]:
-self.unload_options = list(self.unload_options) + ['PARALLEL OFF', 
]
+if self.include_header and 'HEADER' not in [uo.upper().strip() for uo 
in self.unload_options]:
+self.unload_options = list(self.unload_options) + ['HEADER', ]
 
 def execute(self, context):
-self.hook = PostgresHook(postgres_conn_id=self.redshift_conn_id)
-self.s3 = S3Hook(aws_conn_id=self.aws_conn_id, verify=self.verify)
-credentials = self.s3.get_credentials()
-unload_options = '\n\t\t\t'.join(self.unload_options)
-
-if self.include_header:
-self.log.info("Retrieving headers from %s.%s...",
-  self.schema, self.table)
-
-columns_query = """SELECT column_name
-FROM information_schema.columns
-WHERE table_schema = '{schema}'
-AND   table_name = '{table}'
-ORDER BY ordinal_position
-""".format(schema=self.schema,
-   table=self.table)
-
-cursor = self.hook.get_conn().cursor()
-cursor.execute(columns_query)
-rows = cursor.fetchall()
-columns = [row[0] for row in rows]
-column_names = ', '.join("{0}".format(c) for c in columns)
-column_headers = ', '.join("\\'{0}\\'".format(c) for c in columns)
-column_castings = ', '.join("CAST({0} AS text) AS {0}".format(c)
-for c in columns)
-
-select_query = """SELECT {column_names} FROM
-(SELECT 2 sort_order, {column_castings}
- FROM {schema}.{table}
-UNION ALL
-SELECT 1 sort_order, {column_headers})
- ORDER BY sort_order"""\
-.format(column_names=column_names,
-column_castings=column_castings,
-column_headers=column_headers,
-schema=self.schema,
-table=self.table)
-else:
-select_query = "SELECT * FROM {schema}.{table}"\
-.format(schema=self.schema,
-table=self.table)
+postgres_hook = PostgresHook(postgres_conn_id=self.redshift_conn_id)
+s3_hook = S3Hook(aws_conn_id=self.aws_conn_id, verify=self.verify)
 
+credentials = s3_hook.get_credentials()
+unload_options = '\n\t\t\t'.join(self.unload_options)
+select_query = "SELECT * FROM 
{schema}.{table}".format(schema=self.schema, table=self.table)
 
 Review comment:
   Yup I think allowing to specify `select_query` should work. We either have 
to deprecate the `schema` and `table` field, or we need to have some validation 
to check that these fields cannot be used at the same time.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5645) In quickstart, precise how airflow commands can be monitored from web UI

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950074#comment-16950074
 ] 

ASF GitHub Bot commented on AIRFLOW-5645:
-

tpilewicz commented on pull request #6316: [AIRFLOW-5645] Precise webserver 
status monitoring possibilities in quickstart
URL: https://github.com/apache/airflow/pull/6316
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW-5645) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   My PR adds details in the quickstart of Airflow, precising that the web UI 
can be used to monitor the evolution of state of Task Instances and DAG runs. I 
suggest adding this because it is a central feature of Airflow.
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   Only adding precisions in the documentation.
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> In quickstart, precise how airflow commands can be monitored from web UI
> 
>
> Key: AIRFLOW-5645
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5645
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: documentation
>Affects Versions: 1.10.5
>Reporter: Thomas Pilewicz
>Assignee: Thomas Pilewicz
>Priority: Minor
>
> The quickstart suggests to run two airflow commands from the CLI to trigger 
> some DAG runs and task instances : airflow tasks run, and airflow dags 
> backfill. It states that the status of the jobs will be displayed to the user 
> but doesn't mention that the webserver can be used to monitor them.
> I suggest developing this statement to tell the user that he will have 
> feedback from the CLI, but can also see the status of his tasks in the web 
> UI, which is a significant feature of Airflow.
> I'll send a PR with this formulation if that's OK



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] tpilewicz opened a new pull request #6316: [AIRFLOW-5645] Precise webserver status monitoring possibilities in quickstart

[GitBox]

tpilewicz opened a new pull request #6316: [AIRFLOW-5645] Precise webserver 
status monitoring possibilities in quickstart
URL: https://github.com/apache/airflow/pull/6316
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW-5645) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   My PR adds details in the quickstart of Airflow, precising that the web UI 
can be used to monitor the evolution of state of Task Instances and DAG runs. I 
suggest adding this because it is a central feature of Airflow.
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   Only adding precisions in the documentation.
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-5645) In quickstart, precise how airflow commands can be monitored from web UI

[Thomas Pilewicz (Jira)]

Thomas Pilewicz created AIRFLOW-5645:


 Summary: In quickstart, precise how airflow commands can be 
monitored from web UI
 Key: AIRFLOW-5645
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5645
 Project: Apache Airflow
  Issue Type: Improvement
  Components: documentation
Affects Versions: 1.10.5
Reporter: Thomas Pilewicz
Assignee: Thomas Pilewicz


The quickstart suggests to run two airflow commands from the CLI to trigger 
some DAG runs and task instances : airflow tasks run, and airflow dags 
backfill. It states that the status of the jobs will be displayed to the user 
but doesn't mention that the webserver can be used to monitor them.

I suggest developing this statement to tell the user that he will have feedback 
from the CLI, but can also see the status of his tasks in the web UI, which is 
a significant feature of Airflow.

I'll send a PR with this formulation if that's OK



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (AIRFLOW-5261) RBAC can't use LDAP -- 'python-ldap' package is missing from setup.py

[Jostein Leira (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jostein Leira reassigned AIRFLOW-5261:
--

Assignee: Jostein Leira

> RBAC can't use LDAP -- 'python-ldap' package is missing from setup.py
> -
>
> Key: AIRFLOW-5261
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5261
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webserver
>Affects Versions: 1.10.4
>Reporter: Mathew Wicks
>Assignee: Jostein Leira
>Priority: Major
>
> Currently with airflow 1.10.4, if you setup a clean python 3 environment, and 
> install airflow with:
> {code:python}
> pip install airflow[ldap]
> {code}
> You will be missing a required package for LDAP to work with the new RBAC, 
> giving:
> {code:java}
> Traceback (most recent call last):
>   File 
> "PYTHON_HOME/lib/python3.6/site-packages/flask_appbuilder/security/manager.py",
>  line 725, in auth_user_ldap
> ldap.set_option(ldap.OPT_X_TLS_CACERTFILE, self.auth_ldap_tls_cacertfile)
> ModuleNotFoundError: No module named 'ldap'
> {code}
> This is because the 'ldap' tag in setup.py doesn't include the 'python-ldap' 
> package, which is needed for 'import ldap' to succeed.
> The change would need to be made here: 
> [https://github.com/apache/airflow/blob/1.10.4/setup.py#L229]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] ddelange commented on issue #5036: [AIRFLOW-XXX] Pin psycopg2 due to breaking change

[GitBox]

ddelange commented on issue #5036: [AIRFLOW-XXX] Pin psycopg2 due to breaking 
change
URL: https://github.com/apache/airflow/pull/5036#issuecomment-541329181
 
 
   is this pin still necessary? Having psycopg2<2.8 prints warnings in logs 
repeatedly as mentioned in #4508 and requires `libpq-dev` installed before 
airflow as mentioned by @ashb


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Assigned] (AIRFLOW-4682) Make airflow/task Pylint compatible

[Thiago Medeiros (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thiago Medeiros reassigned AIRFLOW-4682:


Assignee: Thiago Medeiros

> Make airflow/task Pylint compatible
> ---
>
> Key: AIRFLOW-4682
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4682
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: ci
>Affects Versions: 2.0.0
>Reporter: Bas Harenslak
>Assignee: Thiago Medeiros
>Priority: Major
>
> Fix all Pylint messages in airflow/task. To start; running 
> scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the 
> files mentioned in your issue from the blacklist. (2) Run 
> scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted 
> files. (3) Fix all messages and create PR.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (AIRFLOW-5261) RBAC can't use LDAP -- 'python-ldap' package is missing from setup.py

[Sergio Kef (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergio Kef reassigned AIRFLOW-5261:
---

Assignee: (was: Sergio Kef)

> RBAC can't use LDAP -- 'python-ldap' package is missing from setup.py
> -
>
> Key: AIRFLOW-5261
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5261
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webserver
>Affects Versions: 1.10.4
>Reporter: Mathew Wicks
>Priority: Major
>
> Currently with airflow 1.10.4, if you setup a clean python 3 environment, and 
> install airflow with:
> {code:python}
> pip install airflow[ldap]
> {code}
> You will be missing a required package for LDAP to work with the new RBAC, 
> giving:
> {code:java}
> Traceback (most recent call last):
>   File 
> "PYTHON_HOME/lib/python3.6/site-packages/flask_appbuilder/security/manager.py",
>  line 725, in auth_user_ldap
> ldap.set_option(ldap.OPT_X_TLS_CACERTFILE, self.auth_ldap_tls_cacertfile)
> ModuleNotFoundError: No module named 'ldap'
> {code}
> This is because the 'ldap' tag in setup.py doesn't include the 'python-ldap' 
> package, which is needed for 'import ldap' to succeed.
> The change would need to be made here: 
> [https://github.com/apache/airflow/blob/1.10.4/setup.py#L229]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (AIRFLOW-5623) latest_only_operator fails for schedule_interval='@once'

[Martin Winkel (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martin Winkel reassigned AIRFLOW-5623:
--

Assignee: Martin Winkel

> latest_only_operator fails for schedule_interval='@once'
> 
>
> Key: AIRFLOW-5623
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5623
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: operators
>Affects Versions: 1.10.5
>Reporter: Gerben Oostra
>Assignee: Martin Winkel
>Priority: Minor
>
> Observation: In a dag with schedule_interval set to @once, the 
> `latest_only_operator` fails with the following error:
>  
> {{[2019-10-09 09:51:37,346] \{latest_only_operator.py:48} INFO - Checking 
> latest only with left_window: None right_window: None now: 2019-10-09 
> 07:51:37.346697+00:00
> [2019-10-09 09:51:37,347] \{models.py:1736} ERROR - '<' not supported between 
> instances of 'NoneType' and 'datetime.datetime'
> Traceback (most recent call last):
>   File 
> "//anaconda/envs/airflow/lib/python3.6/site-packages/airflow/models.py", 
> line 1633, in _run_raw_task
> result = task_copy.execute(context=context)
>   File 
> "//anaconda/envs/airflow/lib/python3.6/site-packages/airflow/operators/latest_only_operator.py",
>  line 51, in execute
> if not left_window < now <= right_window:
> TypeError: '<' not supported between instances of 'NoneType' and 
> 'datetime.datetime'
> [2019-10-09 09:51:37,363] \{models.py:1756} INFO - Marking task as 
> UP_FOR_RETRY}}
> I expected it to succeed, and allow the remainder of the dag to be ran. (if 
> an @once dag is running, it is always the latest)
> Rootcause analysis:
> If the `schedule_interval` of a dag is `@once`, the dag's field 
> `self._schedule_interval` is set to `None`.
> The `latest_only_operator` determines the window by passing the execution 
> date to the dags `following_schedule()`. There the dag's 
> `self._schedule_interval` type is compared to `six.string_types` and 
> `timedelta`. Both type checks fail, so nothing (`None`) is returned.
> Causing the time window comparison to fail.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (AIRFLOW-5261) RBAC can't use LDAP -- 'python-ldap' package is missing from setup.py

[Sergio Kef (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergio Kef reassigned AIRFLOW-5261:
---

Assignee: Sergio Kef

> RBAC can't use LDAP -- 'python-ldap' package is missing from setup.py
> -
>
> Key: AIRFLOW-5261
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5261
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webserver
>Affects Versions: 1.10.4
>Reporter: Mathew Wicks
>Assignee: Sergio Kef
>Priority: Major
>
> Currently with airflow 1.10.4, if you setup a clean python 3 environment, and 
> install airflow with:
> {code:python}
> pip install airflow[ldap]
> {code}
> You will be missing a required package for LDAP to work with the new RBAC, 
> giving:
> {code:java}
> Traceback (most recent call last):
>   File 
> "PYTHON_HOME/lib/python3.6/site-packages/flask_appbuilder/security/manager.py",
>  line 725, in auth_user_ldap
> ldap.set_option(ldap.OPT_X_TLS_CACERTFILE, self.auth_ldap_tls_cacertfile)
> ModuleNotFoundError: No module named 'ldap'
> {code}
> This is because the 'ldap' tag in setup.py doesn't include the 'python-ldap' 
> package, which is needed for 'import ldap' to succeed.
> The change would need to be made here: 
> [https://github.com/apache/airflow/blob/1.10.4/setup.py#L229]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] lucacavazzana removed a comment on issue #6090: [AIRFLOW-5470] Add Apache Livy REST operator

[GitBox]

lucacavazzana removed a comment on issue #6090: [AIRFLOW-5470] Add Apache Livy 
REST operator
URL: https://github.com/apache/airflow/pull/6090#issuecomment-541308037
 
 
   asking for an opinion: is it better to put the reference to the new 
hook/operator in the Spark paragraph or should I create a new one dedicated to 
Livy?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5640) BaseOperator email parameter is wrongly typed and not documented

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950028#comment-16950028
 ] 

ASF GitHub Bot commented on AIRFLOW-5640:
-

SaturnFromTitan commented on pull request #6315: [AIRFLOW-5640] fix 
get_email_address_list types
URL: https://github.com/apache/airflow/pull/6315
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Tests
   
   - [x] My PR adds the following unit tests:
   - 
[airflow/tests/utils/test_email.py](https://github.com/SaturnFromTitan/airflow/blob/5640_fix_to_email_list_types/tests/utils/test_email.py#L47-L63)
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> BaseOperator email parameter is wrongly typed and not documented
> 
>
> Key: AIRFLOW-5640
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5640
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: operators
>Affects Versions: 1.10.5
>Reporter: Cedrik Neumann
>Assignee: Martin Winkel
>Priority: Major
>
> The {{email}} field is not documented in BaseOperator and furthermore the 
> type annotation {{str}} is wrong 
> [here|https://github.com/apache/airflow/blob/master/airflow/models/baseoperator.py#L273].
> The method {{get_email_address_list}} clearly accepts lists of strings as 
> well as comma and semicolon delimited lists: 
> [here|https://github.com/apache/airflow/blob/88989200a66291580088188f06a6db503ac823e2/airflow/utils/email.py#L123]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] SaturnFromTitan opened a new pull request #6315: [AIRFLOW-5640] fix get_email_address_list types

[GitBox]

SaturnFromTitan opened a new pull request #6315: [AIRFLOW-5640] fix 
get_email_address_list types
URL: https://github.com/apache/airflow/pull/6315
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Tests
   
   - [x] My PR adds the following unit tests:
   - 
[airflow/tests/utils/test_email.py](https://github.com/SaturnFromTitan/airflow/blob/5640_fix_to_email_list_types/tests/utils/test_email.py#L47-L63)
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feluelle commented on issue #6308: [AIRFLOW-5634] Don't allow editing of DagModelView

[GitBox]

feluelle commented on issue #6308: [AIRFLOW-5634] Don't allow editing of 
DagModelView
URL: https://github.com/apache/airflow/pull/6308#issuecomment-541321166
 
 
   I think editing dag runs would then be a different ticket/PR.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feluelle commented on issue #6309: [AIRFLOW-3783][AIRFLOW-5395] Switch to HEADER arg to unload data from Redshift to S3

[GitBox]

feluelle commented on issue #6309: [AIRFLOW-3783][AIRFLOW-5395] Switch to 
HEADER arg to unload data from Redshift to S3
URL: https://github.com/apache/airflow/pull/6309#issuecomment-541321206
 
 
   > This fixes AIRFLOW-5395 by simply not having to mention the columns 
anymore right?
   
   Yes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feluelle commented on a change in pull request #6308: [AIRFLOW-5634] Don't allow editing of DagModelView

[GitBox]

feluelle commented on a change in pull request #6308: [AIRFLOW-5634] Don't 
allow editing of DagModelView
URL: https://github.com/apache/airflow/pull/6308#discussion_r334235657
 
 

 ##
 File path: tests/www/test_views.py
 ##
 @@ -446,10 +446,6 @@ def test_xcom(self):
 resp = self.client.get(url, follow_redirects=True)
 self.check_content_in_response('XCom', resp)
 
-def test_edit_dagrun_page(self):
 
 Review comment:
   This was also on my todo list.
   Actually, if you look closer :D you can see what my intention was. I wanted 
to edit [dagruns 
](https://github.com/apache/airflow/blob/master/airflow/www/views.py#L2306) not 
the 
[dag](https://github.com/apache/airflow/blob/master/airflow/www/views.py#L2553).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feluelle commented on a change in pull request #6308: [AIRFLOW-5634] Don't allow editing of DagModelView

[GitBox]

feluelle commented on a change in pull request #6308: [AIRFLOW-5634] Don't 
allow editing of DagModelView
URL: https://github.com/apache/airflow/pull/6308#discussion_r334235455
 
 

 ##
 File path: tests/www/test_views.py
 ##
 @@ -446,10 +446,6 @@ def test_xcom(self):
 resp = self.client.get(url, follow_redirects=True)
 self.check_content_in_response('XCom', resp)
 
-def test_edit_dagrun_page(self):
-resp = self.client.get('dagmodel/edit/example_bash_operator', 
follow_redirects=False)
-self.assertEqual(resp.status_code, 200)
-
 def test_edit_dagrun_url(self):
 
 Review comment:
   This can be removed as well.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feluelle commented on a change in pull request #6309: [AIRFLOW-3783][AIRFLOW-5395] Switch to HEADER arg to unload data from Redshift to S3

[GitBox]

feluelle commented on a change in pull request #6309: 
[AIRFLOW-3783][AIRFLOW-5395] Switch to HEADER arg to unload data from Redshift 
to S3
URL: https://github.com/apache/airflow/pull/6309#discussion_r334235150
 
 

 ##
 File path: airflow/operators/redshift_to_s3_operator.py
 ##
 @@ -85,52 +93,16 @@ def __init__(
 self.autocommit = autocommit
 self.include_header = include_header
 
-if self.include_header and 'PARALLEL OFF' not in [uo.upper().strip() 
for uo in self.unload_options]:
-self.unload_options = list(self.unload_options) + ['PARALLEL OFF', 
]
+if self.include_header and 'HEADER' not in [uo.upper().strip() for uo 
in self.unload_options]:
+self.unload_options = list(self.unload_options) + ['HEADER', ]
 
 def execute(self, context):
-self.hook = PostgresHook(postgres_conn_id=self.redshift_conn_id)
-self.s3 = S3Hook(aws_conn_id=self.aws_conn_id, verify=self.verify)
-credentials = self.s3.get_credentials()
-unload_options = '\n\t\t\t'.join(self.unload_options)
-
-if self.include_header:
-self.log.info("Retrieving headers from %s.%s...",
-  self.schema, self.table)
-
-columns_query = """SELECT column_name
-FROM information_schema.columns
-WHERE table_schema = '{schema}'
-AND   table_name = '{table}'
-ORDER BY ordinal_position
-""".format(schema=self.schema,
-   table=self.table)
-
-cursor = self.hook.get_conn().cursor()
-cursor.execute(columns_query)
-rows = cursor.fetchall()
-columns = [row[0] for row in rows]
-column_names = ', '.join("{0}".format(c) for c in columns)
-column_headers = ', '.join("\\'{0}\\'".format(c) for c in columns)
-column_castings = ', '.join("CAST({0} AS text) AS {0}".format(c)
-for c in columns)
-
-select_query = """SELECT {column_names} FROM
-(SELECT 2 sort_order, {column_castings}
- FROM {schema}.{table}
-UNION ALL
-SELECT 1 sort_order, {column_headers})
- ORDER BY sort_order"""\
-.format(column_names=column_names,
-column_castings=column_castings,
-column_headers=column_headers,
-schema=self.schema,
-table=self.table)
-else:
-select_query = "SELECT * FROM {schema}.{table}"\
-.format(schema=self.schema,
-table=self.table)
+postgres_hook = PostgresHook(postgres_conn_id=self.redshift_conn_id)
+s3_hook = S3Hook(aws_conn_id=self.aws_conn_id, verify=self.verify)
 
+credentials = s3_hook.get_credentials()
+unload_options = '\n\t\t\t'.join(self.unload_options)
+select_query = "SELECT * FROM 
{schema}.{table}".format(schema=self.schema, table=self.table)
 
 Review comment:
   ..thinking about it, I think it's easier to specify `columns` and `order_by` 
in the `__init__`, because if we want to add `select_query` to it we need to 
differentiate between using `select_query` or `schema` & `table` - that's more 
complex.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feluelle commented on a change in pull request #6309: [AIRFLOW-3783][AIRFLOW-5395] Switch to HEADER arg to unload data from Redshift to S3

[GitBox]

feluelle commented on a change in pull request #6309: 
[AIRFLOW-3783][AIRFLOW-5395] Switch to HEADER arg to unload data from Redshift 
to S3
URL: https://github.com/apache/airflow/pull/6309#discussion_r334234652
 
 

 ##
 File path: airflow/operators/redshift_to_s3_operator.py
 ##
 @@ -85,52 +93,16 @@ def __init__(
 self.autocommit = autocommit
 self.include_header = include_header
 
-if self.include_header and 'PARALLEL OFF' not in [uo.upper().strip() 
for uo in self.unload_options]:
-self.unload_options = list(self.unload_options) + ['PARALLEL OFF', 
]
+if self.include_header and 'HEADER' not in [uo.upper().strip() for uo 
in self.unload_options]:
+self.unload_options = list(self.unload_options) + ['HEADER', ]
 
 def execute(self, context):
-self.hook = PostgresHook(postgres_conn_id=self.redshift_conn_id)
-self.s3 = S3Hook(aws_conn_id=self.aws_conn_id, verify=self.verify)
-credentials = self.s3.get_credentials()
-unload_options = '\n\t\t\t'.join(self.unload_options)
-
-if self.include_header:
-self.log.info("Retrieving headers from %s.%s...",
-  self.schema, self.table)
-
-columns_query = """SELECT column_name
-FROM information_schema.columns
-WHERE table_schema = '{schema}'
-AND   table_name = '{table}'
-ORDER BY ordinal_position
-""".format(schema=self.schema,
-   table=self.table)
-
-cursor = self.hook.get_conn().cursor()
-cursor.execute(columns_query)
-rows = cursor.fetchall()
-columns = [row[0] for row in rows]
-column_names = ', '.join("{0}".format(c) for c in columns)
-column_headers = ', '.join("\\'{0}\\'".format(c) for c in columns)
-column_castings = ', '.join("CAST({0} AS text) AS {0}".format(c)
-for c in columns)
-
-select_query = """SELECT {column_names} FROM
-(SELECT 2 sort_order, {column_castings}
- FROM {schema}.{table}
-UNION ALL
-SELECT 1 sort_order, {column_headers})
- ORDER BY sort_order"""\
-.format(column_names=column_names,
-column_castings=column_castings,
-column_headers=column_headers,
-schema=self.schema,
-table=self.table)
-else:
-select_query = "SELECT * FROM {schema}.{table}"\
-.format(schema=self.schema,
-table=self.table)
+postgres_hook = PostgresHook(postgres_conn_id=self.redshift_conn_id)
+s3_hook = S3Hook(aws_conn_id=self.aws_conn_id, verify=self.verify)
 
+credentials = s3_hook.get_credentials()
+unload_options = '\n\t\t\t'.join(self.unload_options)
+select_query = "SELECT * FROM 
{schema}.{table}".format(schema=self.schema, table=self.table)
 
 Review comment:
   Yes, we may even consider allowing to specify the complete `select_query`? 
WDYT?
   ```suggestion
   select_query = self.select_query or "SELECT * FROM 
{schema}.{table}".format(schema=self.schema, table=self.table)
   ```
   
   So that you can also define the order of the rows returned for example.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Assigned] (AIRFLOW-5249) BigQueryCheckOperator fails for datasets outside of 'US' region

[Thomas Pilewicz (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Pilewicz reassigned AIRFLOW-5249:


Assignee: Thomas Pilewicz

> BigQueryCheckOperator fails for datasets outside of 'US' region
> ---
>
> Key: AIRFLOW-5249
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5249
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: operators
>Affects Versions: 1.10.2
>Reporter: Michael
>Assignee: Thomas Pilewicz
>Priority: Blocker
>
> When I try to use the BigQueryCheckOperator or BigQueryValueCheckOperator on 
> a dataset that is not in the 'US' location my task fails with the following 
> error
> {code:java}
> [2019-08-15 07:26:19,378] {__init__.py:1580} ERROR - BigQuery job status 
> check failed. Final error was: 404
> Traceback (most recent call last):
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
>  line 1241, in run_with_configuration
> jobId=self.running_job_id).execute()
>   File "/usr/local/lib/python3.6/site-packages/googleapiclient/_helpers.py", 
> line 130, in positional_wrapper
> return wrapped(*args, **kwargs)
>   File "/usr/local/lib/python3.6/site-packages/googleapiclient/http.py", line 
> 855, in execute
> raise HttpError(resp, content, uri=self.uri)
> googleapiclient.errors.HttpError:  https://www.googleapis.com/bigquery/v2/projects/anz-data-cde-airflow/jobs/job_ISDpiVtd7U1p-6N9wT378LfwoFHc?alt=json
>  returned "Not found: Job 
> anz-data-cde-airflow:job_ISDpiVtd7U1p-6N9wT378LfwoFHc">
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.6/site-packages/airflow/models/__init__.py", 
> line 1441, in _run_raw_task
> result = task_copy.execute(context=context)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/operators/check_operator.py", 
> line 81, in execute
> records = self.get_db_hook().get_first(self.sql)
>   File "/usr/local/lib/python3.6/site-packages/airflow/hooks/dbapi_hook.py", 
> line 138, in get_first
> cur.execute(sql)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
>  line 1821, in execute
> self.job_id = self.run_query(sql)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
>  line 849, in run_query
> return self.run_with_configuration(configuration)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
>  line 1263, in run_with_configuration
> format(err.resp.status))
> Exception: BigQuery job status check failed. Final error was: 404
> [2019-08-15 07:26:19,388] {__init__.py:1611} INFO - Marking task as FAILED.
> {code}
> This is the same error I get when I try to run the BigQuery operator without 
> specifying a location. When I run the same operator on a dataset that is in 
> the US region It succeeds.
> The BigQueryCheckOperator does not accept a location as one of its arguments 
> and does not pass a location to the BigQueryHook, I believe this is the 
> source of the problem. 
>  
> I realise a task (AIRFLOW-3601) was already created to fix a similar issue to 
> this one, but the referenced task calls out the two operators I'm having an 
> issue with as out of scope and after commenting on that task I have not 
> received a response.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5249) BigQueryCheckOperator fails for datasets outside of 'US' region

[Thomas Pilewicz (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950009#comment-16950009
 ] 

Thomas Pilewicz commented on AIRFLOW-5249:
--

Hey, I will try to fix this

> BigQueryCheckOperator fails for datasets outside of 'US' region
> ---
>
> Key: AIRFLOW-5249
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5249
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: operators
>Affects Versions: 1.10.2
>Reporter: Michael
>Assignee: Thomas Pilewicz
>Priority: Blocker
>
> When I try to use the BigQueryCheckOperator or BigQueryValueCheckOperator on 
> a dataset that is not in the 'US' location my task fails with the following 
> error
> {code:java}
> [2019-08-15 07:26:19,378] {__init__.py:1580} ERROR - BigQuery job status 
> check failed. Final error was: 404
> Traceback (most recent call last):
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
>  line 1241, in run_with_configuration
> jobId=self.running_job_id).execute()
>   File "/usr/local/lib/python3.6/site-packages/googleapiclient/_helpers.py", 
> line 130, in positional_wrapper
> return wrapped(*args, **kwargs)
>   File "/usr/local/lib/python3.6/site-packages/googleapiclient/http.py", line 
> 855, in execute
> raise HttpError(resp, content, uri=self.uri)
> googleapiclient.errors.HttpError:  https://www.googleapis.com/bigquery/v2/projects/anz-data-cde-airflow/jobs/job_ISDpiVtd7U1p-6N9wT378LfwoFHc?alt=json
>  returned "Not found: Job 
> anz-data-cde-airflow:job_ISDpiVtd7U1p-6N9wT378LfwoFHc">
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.6/site-packages/airflow/models/__init__.py", 
> line 1441, in _run_raw_task
> result = task_copy.execute(context=context)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/operators/check_operator.py", 
> line 81, in execute
> records = self.get_db_hook().get_first(self.sql)
>   File "/usr/local/lib/python3.6/site-packages/airflow/hooks/dbapi_hook.py", 
> line 138, in get_first
> cur.execute(sql)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
>  line 1821, in execute
> self.job_id = self.run_query(sql)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
>  line 849, in run_query
> return self.run_with_configuration(configuration)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/bigquery_hook.py",
>  line 1263, in run_with_configuration
> format(err.resp.status))
> Exception: BigQuery job status check failed. Final error was: 404
> [2019-08-15 07:26:19,388] {__init__.py:1611} INFO - Marking task as FAILED.
> {code}
> This is the same error I get when I try to run the BigQuery operator without 
> specifying a location. When I run the same operator on a dataset that is in 
> the US region It succeeds.
> The BigQueryCheckOperator does not accept a location as one of its arguments 
> and does not pass a location to the BigQueryHook, I believe this is the 
> source of the problem. 
>  
> I realise a task (AIRFLOW-3601) was already created to fix a similar issue to 
> this one, but the referenced task calls out the two operators I'm having an 
> issue with as out of scope and after commenting on that task I have not 
> received a response.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (AIRFLOW-4675) Make airflow/lineage Pylint compatible

[enrica (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

enrica reassigned AIRFLOW-4675:
---

Assignee: enrica

> Make airflow/lineage Pylint compatible
> --
>
> Key: AIRFLOW-4675
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4675
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: ci
>Affects Versions: 2.0.0
>Reporter: Bas Harenslak
>Assignee: enrica
>Priority: Major
>
> Fix all Pylint messages in airflow/lineage. To start; running 
> scripts/ci/ci_pylint.sh on master should produce no messages. (1) Remove the 
> files mentioned in your issue from the blacklist. (2) Run 
> scripts/ci/ci_pylint.sh to see all messages on the no longer blacklisted 
> files. (3) Fix all messages and create PR.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5497) dag_processing::list_py_file_paths method pydoc needs param added

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949998#comment-16949998
 ] 

ASF GitHub Bot commented on AIRFLOW-5497:
-

chicochico commented on pull request #6314: [AIRFLOW-5497] Update docstring 
with param description
URL: https://github.com/apache/airflow/pull/6314
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason: Documentation change
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> dag_processing::list_py_file_paths method pydoc needs param added
> -
>
> Key: AIRFLOW-5497
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5497
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: utils
>Affects Versions: 1.10.6
>Reporter: Jakob Homan
>Priority: Minor
>  Labels: ccoss2019, newbie
>
> Note: This ticket's being created to facilitate a new contributor's workshop 
> for Airflow. After the workshop has completed, I'll mark these all available 
> for anyone that might like to take them on.
> The {{list_py_file_paths}} method pydoc is missing an entry for the 
> {{include_examples}} parameter.  We should add it.
> airflow/utils/dag_processing.py:291
> {code:java}
> def list_py_file_paths(directory, safe_mode=conf.getboolean('core', 
> 'DAG_DISCOVERY_SAFE_MODE', fallback=True),
>include_examples=None):
> """
> Traverse a directory and look for Python files.
> :param directory: the directory to traverse
> :type directory: unicode
> :param safe_mode: whether to use a heuristic to determine whether a file
> contains Airflow DAG definitions. If not provided, use the
> core.DAG_DISCOVERY_SAFE_MODE configuration setting. If not set, 
> default
> to safe.
> :return: a list of paths to Python files in the specified directory
> :rtype: list[unicode] {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] chicochico opened a new pull request #6314: [AIRFLOW-5497] Update docstring with param description

[GitBox]

chicochico opened a new pull request #6314: [AIRFLOW-5497] Update docstring 
with param description
URL: https://github.com/apache/airflow/pull/6314
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason: Documentation change
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-5644) Simplify TriggerDagRunOperator usage

[Bas Harenslak (Jira)]

Bas Harenslak created AIRFLOW-5644:
--

 Summary: Simplify TriggerDagRunOperator usage
 Key: AIRFLOW-5644
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5644
 Project: Apache Airflow
  Issue Type: Improvement
  Components: operators
Affects Versions: 2.0.0
Reporter: Bas Harenslak


The TriggerDagRunOperator usage is rather odd at the moment, especially the way 
to pass a conf.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] codecov-io commented on issue #6313: [AIRFLOW-5643] Reduce duplicated logic in S3Hook

[GitBox]

codecov-io commented on issue #6313: [AIRFLOW-5643] Reduce duplicated logic in 
S3Hook
URL: https://github.com/apache/airflow/pull/6313#issuecomment-541310309
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6313?src=pr=h1) 
Report
   > Merging 
[#6313](https://codecov.io/gh/apache/airflow/pull/6313?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/c0d98a72394b68df27db4d1cab4a94021980e0fb?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6313/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6313?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6313  +/-   ##
   ==
   + Coverage   80.34%   80.35%   +<.01% 
   ==
 Files 616  616  
 Lines   3573335724   -9 
   ==
   - Hits2871128705   -6 
   + Misses   7022 7019   -3
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6313?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/hooks/S3\_hook.py](https://codecov.io/gh/apache/airflow/pull/6313/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9TM19ob29rLnB5)
 | `95.62% <100%> (+1.35%)` | :arrow_up: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6313/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `56.37% <0%> (-0.52%)` | :arrow_down: |
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6313/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `93.77% <0%> (+0.5%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6313?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6313?src=pr=footer). 
Last update 
[c0d98a7...1fc26b2](https://codecov.io/gh/apache/airflow/pull/6313?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #6313: [AIRFLOW-5643] Reduce duplicated logic in S3Hook

[GitBox]

codecov-io edited a comment on issue #6313: [AIRFLOW-5643] Reduce duplicated 
logic in S3Hook
URL: https://github.com/apache/airflow/pull/6313#issuecomment-541310309
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/6313?src=pr=h1) 
Report
   > Merging 
[#6313](https://codecov.io/gh/apache/airflow/pull/6313?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/c0d98a72394b68df27db4d1cab4a94021980e0fb?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/6313/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/6313?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#6313  +/-   ##
   ==
   + Coverage   80.34%   80.35%   +<.01% 
   ==
 Files 616  616  
 Lines   3573335724   -9 
   ==
   - Hits2871128705   -6 
   + Misses   7022 7019   -3
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/6313?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/hooks/S3\_hook.py](https://codecov.io/gh/apache/airflow/pull/6313/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9TM19ob29rLnB5)
 | `95.62% <100%> (+1.35%)` | :arrow_up: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6313/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `56.37% <0%> (-0.52%)` | :arrow_down: |
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6313/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `93.77% <0%> (+0.5%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/6313?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/6313?src=pr=footer). 
Last update 
[c0d98a7...1fc26b2](https://codecov.io/gh/apache/airflow/pull/6313?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Work started] (AIRFLOW-5643) S3Hook logic is duplicated and could be clearer

[Louis Guitton (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on AIRFLOW-5643 started by Louis Guitton.
--
> S3Hook logic is duplicated and could be clearer
> ---
>
> Key: AIRFLOW-5643
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5643
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: aws, hooks
>Affects Versions: 1.10.5
>Reporter: Louis Guitton
>Assignee: Louis Guitton
>Priority: Trivial
>
> S3Hook.load_bytes is duplicating the logic of S3Hook.load_file_obj
> [https://github.com/apache/airflow/blob/master/airflow/hooks/S3_hook.py#L464-L539|https://github.com/apache/airflow/blob/master/airflow/hooks/S3_hook.py#L465-L539]
> Instead, we should stay consistent : S3Hook.load_string is already delegating 
> the logic to S3Hook.load_bytes, so we can use the same approach to delegate 
> to S3Hook.load_file_obj



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (AIRFLOW-5643) S3Hook logic is duplicated and could be clearer

[Louis Guitton (Jira)]

Louis Guitton created AIRFLOW-5643:
--

 Summary: S3Hook logic is duplicated and could be clearer
 Key: AIRFLOW-5643
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5643
 Project: Apache Airflow
  Issue Type: Improvement
  Components: aws, hooks
Affects Versions: 1.10.5
Reporter: Louis Guitton
Assignee: Louis Guitton


S3Hook.load_bytes is duplicating the logic of S3Hook.load_file_obj

[https://github.com/apache/airflow/blob/master/airflow/hooks/S3_hook.py#L464-L539|https://github.com/apache/airflow/blob/master/airflow/hooks/S3_hook.py#L465-L539]

Instead, we should stay consistent : S3Hook.load_string is already delegating 
the logic to S3Hook.load_bytes, so we can use the same approach to delegate to 
S3Hook.load_file_obj



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] lucacavazzana commented on issue #6090: [AIRFLOW-5470] Add Apache Livy REST operator

[GitBox]

lucacavazzana commented on issue #6090: [AIRFLOW-5470] Add Apache Livy REST 
operator
URL: https://github.com/apache/airflow/pull/6090#issuecomment-541308037
 
 
   asking for an opinion: is it better to put the reference to the new 
hook/operator in the Spark paragraph or should I create a new one dedicated to 
Livy?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Closed] (AIRFLOW-4543) Update slack operator to support slackclient v2

[Sergio Kef (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergio Kef closed AIRFLOW-4543.
---
Resolution: Won't Fix

Closing because 3.5 cannot be supported

> Update slack operator to support slackclient v2
> ---
>
> Key: AIRFLOW-4543
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4543
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: hooks, operators
>Reporter: Sergio Kef
>Assignee: Sergio Kef
>Priority: Major
>
> Official [Slack API for python|https://pypi.org/project/slackclient/] has 
> recently released 
> [v.2|https://github.com/slackapi/python-slackclient/wiki/Migrating-to-2.x0]
> Among others some important points:
>  * Async IO
>  * SSL and Proxy
>  * Dropping 2.7 support
> Opening this ticket to work on the upgrade. Current functionalities will be 
> migrated and will try to extend functionalities, if possible.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] louisguitton opened a new pull request #6313: [NO-TICKET] Clarify logic S3Hook

[GitBox]

louisguitton opened a new pull request #6313: [NO-TICKET] Clarify logic S3Hook
URL: https://github.com/apache/airflow/pull/6313
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Assigned] (AIRFLOW-5640) BaseOperator email parameter is wrongly typed and not documented

[Martin Winkel (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martin Winkel reassigned AIRFLOW-5640:
--

Assignee: Martin Winkel

> BaseOperator email parameter is wrongly typed and not documented
> 
>
> Key: AIRFLOW-5640
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5640
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: operators
>Affects Versions: 1.10.5
>Reporter: Cedrik Neumann
>Assignee: Martin Winkel
>Priority: Major
>
> The {{email}} field is not documented in BaseOperator and furthermore the 
> type annotation {{str}} is wrong 
> [here|https://github.com/apache/airflow/blob/master/airflow/models/baseoperator.py#L273].
> The method {{get_email_address_list}} clearly accepts lists of strings as 
> well as comma and semicolon delimited lists: 
> [here|https://github.com/apache/airflow/blob/88989200a66291580088188f06a6db503ac823e2/airflow/utils/email.py#L123]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-5639) DagFileProcessor: parse dag files every time, which consume lots of resources and is unnecessary

[chen xianxin (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

chen xianxin updated AIRFLOW-5639:
--
Description: 
 
h2. Code

[https://github.com/apache/airflow/blob/v1-10-stable/airflow/models/dagbag.py#L166-L170]
h2. Problem description 

*self.file_last_changed* doesn't work, beacause it's set to empty when Dagbag 
is initialized. so even no changes were made to the file, it will also be 
imported. I have sure this problem by print log.

Generate bagdags from files take about 50% time of dag files processing, if 
only generate bagdags when files are changed,  lots of resources will be saved.

  was:
 
h2. Code

[https://github.com/apache/airflow/blob/v1-10-stable/airflow/models/dagbag.py#L166-L170]
h2. Problem description 

*self.file_last_changed* doesn't work, beacause it's set to empty when 
initializing. so even no changes were made to the file, it will also parse


> DagFileProcessor: parse dag files every time, which consume lots of resources 
> and is unnecessary
> 
>
> Key: AIRFLOW-5639
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5639
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: scheduler
>Affects Versions: 1.10.5
>Reporter: chen xianxin
>Priority: Major
>
>  
> h2. Code
> [https://github.com/apache/airflow/blob/v1-10-stable/airflow/models/dagbag.py#L166-L170]
> h2. Problem description 
> *self.file_last_changed* doesn't work, beacause it's set to empty when Dagbag 
> is initialized. so even no changes were made to the file, it will also be 
> imported. I have sure this problem by print log.
> Generate bagdags from files take about 50% time of dag files processing, if 
> only generate bagdags when files are changed,  lots of resources will be 
> saved.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-5642) sqlalchemy.exc.OperationalError happens when sql is too long, which cause scheduler processor will be exited

[chen xianxin (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

chen xianxin updated AIRFLOW-5642:
--
Description: 
h2. code

https://github.com/apache/airflow/blob/v1-10-stable/airflow/jobs/scheduler_job.py#L1060-L1069
h2. Error description

{{if there is too many queued_tasks, sqlalchemy.exc.OperationalError will 
occur. details are as follow:}}

{{<}}

{{sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2006, "MySQL 
server has gone away (BrokenPipeError(32, 'Broken pipe'))")}}
 [SQL: SELECT ...]

(Background on this error at: [http://sqlalche.me/e/e3q8])

{{> }}

  was:
h2. code

airflow\jobs\scheduler_job.py:_change_state_for_tasks_failed_to_execute(self, 
session)

<

...

filter_for_ti_state_change = (
    [and_(
    TI.dag_id == dag_id,
    TI.task_id == task_id,
    TI.execution_date == execution_date,
    TI._try_number == try_number - 1,
    TI.state == State.QUEUED)
    for dag_id, task_id, execution_date, try_number
    in self.executor.queued_tasks.keys()])
ti_query = (session.query(TI)
    .filter(or_(*filter_for_ti_state_change)))
tis_to_set_to_scheduled = (ti_query
    .with_for_update()
    .all())

...

>
h2. Error description

{{if there is too many queued_tasks, sqlalchemy.exc.OperationalError will 
occur. details are as follow:}}

{{<}}

{{sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2006, "MySQL 
server has gone away (BrokenPipeError(32, 'Broken pipe'))")}}
[SQL: SELECT ...]

(Background on this error at: http://sqlalche.me/e/e3q8)

{{> }}


> sqlalchemy.exc.OperationalError happens when sql is too long, which cause 
> scheduler processor will be exited
> 
>
> Key: AIRFLOW-5642
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5642
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: 1.10.5
>Reporter: chen xianxin
>Priority: Minor
>
> h2. code
> https://github.com/apache/airflow/blob/v1-10-stable/airflow/jobs/scheduler_job.py#L1060-L1069
> h2. Error description
> {{if there is too many queued_tasks, sqlalchemy.exc.OperationalError will 
> occur. details are as follow:}}
> {{<}}
> {{sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2006, 
> "MySQL server has gone away (BrokenPipeError(32, 'Broken pipe'))")}}
>  [SQL: SELECT ...]
> (Background on this error at: [http://sqlalche.me/e/e3q8])
> {{> }}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-5639) DagFileProcessor: parse dag files every time, which consume lots of resources and is unnecessary

[chen xianxin (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

chen xianxin updated AIRFLOW-5639:
--
Description: 
 
h2. Code

[https://github.com/apache/airflow/blob/v1-10-stable/airflow/models/dagbag.py#L166-L170]
h2. Problem description 

*self.file_last_changed* doesn't work, beacause it's set to empty when 
initializing. so even no changes were made to the file, it will also parse

  was:
h2. Code

airflow\models\dagbag.py: process_file(self, filepath, only_if_updated=True, 
safe_mode=True)

<

file_last_changed_on_disk = datetime.fromtimestamp(os.path.getmtime(filepath))
 if only_if_updated \
     and filepath in self.file_last_changed \
     and file_last_changed_on_disk == *self.file_last_changed*[filepath]:
     return found_dags

>

https://github.com/apache/airflow/blob/v1-10-stable/airflow/models/dagbag.py#L166-L170
h2. Problem description 

*self.file_last_changed* doesn't work, beacause it's set to empty when 
initializing. so even no changes were made to the file, it will also parse


> DagFileProcessor: parse dag files every time, which consume lots of resources 
> and is unnecessary
> 
>
> Key: AIRFLOW-5639
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5639
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: scheduler
>Affects Versions: 1.10.5
>Reporter: chen xianxin
>Priority: Major
>
>  
> h2. Code
> [https://github.com/apache/airflow/blob/v1-10-stable/airflow/models/dagbag.py#L166-L170]
> h2. Problem description 
> *self.file_last_changed* doesn't work, beacause it's set to empty when 
> initializing. so even no changes were made to the file, it will also parse



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-5639) DagFileProcessor: parse dag files every time, which consume lots of resources and is unnecessary

[chen xianxin (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

chen xianxin updated AIRFLOW-5639:
--
Description: 
h2. Code

airflow\models\dagbag.py: process_file(self, filepath, only_if_updated=True, 
safe_mode=True)

<

file_last_changed_on_disk = datetime.fromtimestamp(os.path.getmtime(filepath))
 if only_if_updated \
     and filepath in self.file_last_changed \
     and file_last_changed_on_disk == *self.file_last_changed*[filepath]:
     return found_dags

>

https://github.com/apache/airflow/blob/v1-10-stable/airflow/models/dagbag.py#L166-L170
h2. Problem description 

*self.file_last_changed* doesn't work, beacause it's set to empty when 
initializing. so even no changes were made to the file, it will also parse

  was:
h2. Code

airflow\models\dagbag.py: process_file(self, filepath, only_if_updated=True, 
safe_mode=True)

<

file_last_changed_on_disk = datetime.fromtimestamp(os.path.getmtime(filepath))
 if only_if_updated \
     and filepath in self.file_last_changed \
     and file_last_changed_on_disk == *self.file_last_changed*[filepath]:
     return found_dags

>
h2. Problem description 

*self.file_last_changed* doesn't work, beacause it's set to empty when 
initializing. so even no changes were made to the file, it will also parse


> DagFileProcessor: parse dag files every time, which consume lots of resources 
> and is unnecessary
> 
>
> Key: AIRFLOW-5639
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5639
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: scheduler
>Affects Versions: 1.10.5
>Reporter: chen xianxin
>Priority: Major
>
> h2. Code
> airflow\models\dagbag.py: process_file(self, filepath, only_if_updated=True, 
> safe_mode=True)
> <
> file_last_changed_on_disk = datetime.fromtimestamp(os.path.getmtime(filepath))
>  if only_if_updated \
>      and filepath in self.file_last_changed \
>      and file_last_changed_on_disk == *self.file_last_changed*[filepath]:
>      return found_dags
> >
> https://github.com/apache/airflow/blob/v1-10-stable/airflow/models/dagbag.py#L166-L170
> h2. Problem description 
> *self.file_last_changed* doesn't work, beacause it's set to empty when 
> initializing. so even no changes were made to the file, it will also parse



--
This message was sent by Atlassian Jira
(v8.3.4#803005)