date:20190405

milton0825 commented on a change in pull request #5010: [AIRFLOW-4192] 
Reorganize task context variables
URL: https://github.com/apache/airflow/pull/5010#discussion_r272785021
 
 

 ##
 File path: UPDATING.md
 ##
 @@ -24,6 +24,24 @@ assists users migrating to a new version.
 
 ## Airflow Master
 
+### Some task context variables are removed
+The following task context variables are removed. Some are obsolete, some are 
duplicates whose 
+value is also given via other variables, and some are "derived" values from 
other variables which 
+are also given in the task context.
+
+* `yesterday_ds`: can be derived from `execution_date`
 
 Review comment:
   Can you share how you derive `yesterday_ds` with `execution_date` with 
macros in the docs?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] milton0825 commented on a change in pull request #5010: [AIRFLOW-4192] Reorganize task context variables

milton0825 commented on a change in pull request #5010: [AIRFLOW-4192] 
Reorganize task context variables
URL: https://github.com/apache/airflow/pull/5010#discussion_r272785021
 
 

 ##
 File path: UPDATING.md
 ##
 @@ -24,6 +24,24 @@ assists users migrating to a new version.
 
 ## Airflow Master
 
+### Some task context variables are removed
+The following task context variables are removed. Some are obsolete, some are 
duplicates whose 
+value is also given via other variables, and some are "derived" values from 
other variables which 
+are also given in the task context.
+
+* `yesterday_ds`: can be derived from `execution_date`
 
 Review comment:
   Can you also show the derivation here?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] KevinYang21 commented on issue #5037: [AIRFLOW-4237] Including Try Number of Task in Gantt Chart

KevinYang21 commented on issue #5037: [AIRFLOW-4237] Including Try Number of 
Task in Gantt Chart
URL: https://github.com/apache/airflow/pull/5037#issuecomment-480476313
 
 
   This sounds to be a pretty useful change. Do you mind post a screenshot on 
how it would look like after the change please?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] KevinYang21 commented on issue #5048: [AIRFLOW-3370] Add stdout output options to Elasticsearch task log handler

KevinYang21 commented on issue #5048: [AIRFLOW-3370] Add stdout output options 
to Elasticsearch task log handler
URL: https://github.com/apache/airflow/pull/5048#issuecomment-480476272
 
 
   subscribing to this PR :D


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] KevinYang21 commented on issue #5050: [AIRFLOW-4251] Instrument DagRun schedule delay

KevinYang21 commented on issue #5050: [AIRFLOW-4251] Instrument DagRun schedule 
delay
URL: https://github.com/apache/airflow/pull/5050#issuecomment-480476229
 
 
   IMO this can be useful when we're analyzing delays, so we know where the 
delay comes from, scheduler or executor. On the other hand, I think the story 
would be more comprehensive if we have task instance level metrics--for a DAG 
with 1k tasks we now get only 1 data point per dag run just like a DAG with 1 
task, which make it less representitive. In th end people may be more 
interested about task instance delays instead of DAG run delay.
   
   About the performance, I think it is not too bad to have it here since we do 
it in the dag parsing subprocess so it is effectively O(# DAG/# subprocesses). 
If data points coming from TI level stat are too many maybe we can try some 
random sampling? Also if it is TI level stat intuitively it would be in the 
main scheduler loop which performance matters more.
   
   All this reminds me about some old discussion we had earlier. If we do want 
to start having a story around scheduling performance, we might need to 
consider the parsing time of DAG files and even kick that out from our 
metrics--otherwise if I as a user introduce u a large # of large DAG files then 
ur metrics will spike.
   
   Just some random thoughts around this topic :D


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feng-tao edited a comment on issue #5010: [AIRFLOW-4192] Reorganize task context variables

feng-tao edited a comment on issue #5010: [AIRFLOW-4192] Reorganize task 
context variables
URL: https://github.com/apache/airflow/pull/5010#issuecomment-480475220
 
 
   sorry, I thought you remove the prev_ds and next_ds. For the yesterday_ds 
and tomrrow_ds, I am fine on removing it.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feng-tao removed a comment on issue #5010: [AIRFLOW-4192] Reorganize task context variables

feng-tao removed a comment on issue #5010: [AIRFLOW-4192] Reorganize task 
context variables
URL: https://github.com/apache/airflow/pull/5010#issuecomment-480475225
 
 
   cc @milton0825 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feng-tao commented on issue #5010: [AIRFLOW-4192] Reorganize task context variables

feng-tao commented on issue #5010: [AIRFLOW-4192] Reorganize task context 
variables
URL: https://github.com/apache/airflow/pull/5010#issuecomment-480475220
 
 
   hold on, there is a reason for these macro heres(yesterday_ds, tomorrow_ds) 
. Please take a look at this pr for more 
detail(https://github.com/apache/airflow/pull/3418).
   
   tldr, when user write hql, they could use this in the hql and then change 
the value in backfill.
   
   I am against removing these macros.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feng-tao commented on issue #5010: [AIRFLOW-4192] Reorganize task context variables

feng-tao commented on issue #5010: [AIRFLOW-4192] Reorganize task context 
variables
URL: https://github.com/apache/airflow/pull/5010#issuecomment-480475225
 
 
   cc @milton0825 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4252) Remove unused sessions



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811437#comment-16811437
 ] 

ASF GitHub Bot commented on AIRFLOW-4252:
-

Fokko commented on pull request #5051: [AIRFLOW-4252] Remove the unused sessions
URL: https://github.com/apache/airflow/pull/5051
 
 
   Some housekeeping
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-4252\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-4252
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [ ] Passes `flake8`
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Remove unused sessions
> --
>
> Key: AIRFLOW-4252
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4252
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] Fokko opened a new pull request #5051: [AIRFLOW-4252] Remove the unused sessions

Fokko opened a new pull request #5051: [AIRFLOW-4252] Remove the unused sessions
URL: https://github.com/apache/airflow/pull/5051
 
 
   Some housekeeping
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-4252\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-4252
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [ ] Passes `flake8`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-4252) Remove unused sessions

2019-04-05 Thread Fokko Driesprong (JIRA)

Fokko Driesprong created AIRFLOW-4252:
-

 Summary: Remove unused sessions
 Key: AIRFLOW-4252
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4252
 Project: Apache Airflow
  Issue Type: Improvement
Reporter: Fokko Driesprong
Assignee: Fokko Driesprong






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] milton0825 edited a comment on issue #5050: [AIRFLOW-4251] Instrument DagRun schedule delay

milton0825 edited a comment on issue #5050: [AIRFLOW-4251] Instrument DagRun 
schedule delay
URL: https://github.com/apache/airflow/pull/5050#issuecomment-480473472
 
 
   I think this metric is useful to track the schedule delay per `DAG`. We can 
use the metric to tell how behind a `DAG` is to its schedule and can quickly 
check the historic trend on a dashboard like Grafana.
   
   Canary monitoring serves for the purpose of tracking the delay from DAG 
schedule to first task execution, which may or may not represents the schedule 
delay for each DAG.
   
   What do you think?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] milton0825 commented on a change in pull request #4898: [AIRFLOW-XXXX] how to setup simple test env with mysql

milton0825 commented on a change in pull request #4898: [AIRFLOW-] how to 
setup simple test env with mysql
URL: https://github.com/apache/airflow/pull/4898#discussion_r272783637
 
 

 ##
 File path: docs/howto/setup-test-environment-mysql.rst
 ##
 @@ -0,0 +1,43 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+..http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+
+Setup Test Environment using MySQL
+==
+
+By default, Airflow uses SQLite as database backend
+and ``SequentialExecutor`` to execute tasks as SQLite
+does not support multiple connections. Since
+``SequentialExecutor`` runs one instance at a time,
+some parallel execution logic will not be exercised
+in this default setup.
+
+To test out the parallel execution setup, we can use
+MySQL as database backend and ``LocalExecutor`` as
+the executor. Checkout the following setups to launch
+a MySQL database container:
+
+.. code-block:: bash
+
+  # Launch MySQL docker container
+  docker-compose -f scripts/ci/docker-compose.yml run -p3306:3306 mysql
+
+  # Open airflow.cfg and add the following:
+  # sql_alchemy_conn = mysql://root@127.0.0.1:3306/airflow
+  # executor = LocalExecutor
+  vim $AIRFLOW_HOME/airflow.cfg
+
+airflow initdb
 
 Review comment:
   fixed
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] milton0825 commented on issue #5050: [AIRFLOW-4251] Instrument DagRun schedule delay

milton0825 commented on issue #5050: [AIRFLOW-4251] Instrument DagRun schedule 
delay
URL: https://github.com/apache/airflow/pull/5050#issuecomment-480473472
 
 
   I think this metric is useful to track the schedule delay per `DAG`. We can 
use the metric to tell how behind a `DAG` is to its schedule and can quickly 
check the historic trend on a dashboard like Grafana.
   
   Canary monitoring serves for the purpose of tracking the delay from DAG 
schedule to first task execution, which may or may not represents the schedule 
delay for each DAG.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] Fokko commented on issue #4922: [AIRFLOW-3844] Add foreign keys for task_instances and dag_runs to dag_model

Fokko commented on issue #4922: [AIRFLOW-3844] Add foreign keys for 
task_instances and dag_runs to dag_model
URL: https://github.com/apache/airflow/pull/4922#issuecomment-480473202
 
 
   I'm happy to help you on this one Julian. The `.sync_to_db()` made things 
worse. In the `airflow resetdb` the example dags are hit, because we now flush 
the DAGs to the DB directly, it can be that the DB is not ready yet (since it 
is still between migrations, and therefore the migrations fail).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feng-tao commented on issue #5034: AIRFLOW-4218 Support to Provide http args to K8executor while calling…

feng-tao commented on issue #5034: AIRFLOW-4218 Support to Provide http args to 
K8executor while calling…
URL: https://github.com/apache/airflow/pull/5034#issuecomment-480468797
 
 
   cc @dimberman 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feng-tao commented on issue #5050: [AIRFLOW-4251] Instrument DagRun schedule delay

feng-tao commented on issue #5050: [AIRFLOW-4251] Instrument DagRun schedule 
delay
URL: https://github.com/apache/airflow/pull/5050#issuecomment-480468747
 
 
   yeah, but what is the difference between running this vs running canary 
monitoring dag? Now you will generate O(#DAG) for this dagrun stats.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #5050: [AIRFLOW-4251] Instrument DagRun schedule delay

codecov-io commented on issue #5050: [AIRFLOW-4251] Instrument DagRun schedule 
delay
URL: https://github.com/apache/airflow/pull/5050#issuecomment-480467332
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5050?src=pr=h1) 
Report
   > Merging 
[#5050](https://codecov.io/gh/apache/airflow/pull/5050?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/b93f2649aed7fe566b987ec4cb135a2ec643acd6?src=pr=desc)
 will **decrease** coverage by `0.01%`.
   > The diff coverage is `0%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5050/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5050?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#5050  +/-   ##
   ==
   - Coverage   76.36%   76.34%   -0.02% 
   ==
 Files 471  471  
 Lines   3029030294   +4 
   ==
   - Hits2313023129   -1 
   - Misses   7160 7165   +5
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/5050?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/jobs.py](https://codecov.io/gh/apache/airflow/pull/5050/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5)
 | `78.57% <0%> (-0.28%)` | :arrow_down: |
   | 
[airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/5050/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=)
 | `92.95% <0%> (-0.05%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5050?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5050?src=pr=footer). 
Last update 
[b93f264...22dcc16](https://codecov.io/gh/apache/airflow/pull/5050?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj edited a comment on issue #5043: [AIRFLOW-4236] Add num_retries to MySqlToGoogleCloudStorageOperator

mik-laj edited a comment on issue #5043: [AIRFLOW-4236] Add num_retries to 
MySqlToGoogleCloudStorageOperator
URL: https://github.com/apache/airflow/pull/5043#issuecomment-480453668
 
 
   Other operators have a fixed number of retry attempts. 
   
https://github.com/apache/airflow/blob/e27950a75ce287c094e550fba07d1c8de5dc4143/airflow/contrib/hooks/gcp_transfer_hook.py#L92
   
https://github.com/apache/airflow/blob/383ec374ea148d04af94496dc9e3283263d67d5f/airflow/contrib/hooks/gcp_function_hook.py#L29
   
https://github.com/apache/airflow/blob/993779f93df172f5f774f1ea2634c55e9c1212c4/airflow/contrib/hooks/gcp_compute_hook.py#L28
   
https://github.com/apache/airflow/blob/49ade3834a46d9a12c0339a017d1807bf021c5e3/airflow/contrib/hooks/gcp_sql_hook.py#L54
   
https://github.com/apache/airflow/blob/85ea7e8575bb0f9a2fac989f26970809b36640a4/airflow/contrib/hooks/gcp_dataflow_hook.py#L51
   I'm afraid it's a good idea to give the user the option to configure this 
parameter in this way. Too many parameters will make it difficult to use the 
operator. In my opinion, this value should be configured in the connection 
settings instead of the specific use, but fixed value is good enough in this 
case.
   
   I working on similar mechanism for Google Cloud AI operators: 
https://github.com/PolideaInternal/airflow/pull/85/files 
   I want to support retrying in reaction to exceeding the temporary quota. I 
would like the use of all GCP operators to be similar, so it is important for 
me to elaborate one variant.
   
   CC: @potiuk 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj edited a comment on issue #5043: [AIRFLOW-4236] Add num_retries to MySqlToGoogleCloudStorageOperator

mik-laj edited a comment on issue #5043: [AIRFLOW-4236] Add num_retries to 
MySqlToGoogleCloudStorageOperator
URL: https://github.com/apache/airflow/pull/5043#issuecomment-480453668
 
 
   Other operators have a fixed number of retry attempts. 
   
https://github.com/apache/airflow/blob/e27950a75ce287c094e550fba07d1c8de5dc4143/airflow/contrib/hooks/gcp_transfer_hook.py#L92
   
https://github.com/apache/airflow/blob/383ec374ea148d04af94496dc9e3283263d67d5f/airflow/contrib/hooks/gcp_function_hook.py#L29
   
https://github.com/apache/airflow/blob/993779f93df172f5f774f1ea2634c55e9c1212c4/airflow/contrib/hooks/gcp_compute_hook.py#L28
   
https://github.com/apache/airflow/blob/49ade3834a46d9a12c0339a017d1807bf021c5e3/airflow/contrib/hooks/gcp_sql_hook.py#L54
   
https://github.com/apache/airflow/blob/85ea7e8575bb0f9a2fac989f26970809b36640a4/airflow/contrib/hooks/gcp_dataflow_hook.py#L51
   I'm afraid
it's a good idea to give the user the option to configure this parameter in 
this way. Too many parameters will make it difficult to use the operator. In my 
opinion, this value should be configured in the connection settings instead of 
the specific use, but fixed value is good enough in this case.
   
   I working on similar mechanism for Google Cloud AI operators: 
https://github.com/PolideaInternal/airflow/pull/85/files 
   I want to support retrying in reaction to exceeding the temporary quota. I 
would like the use of all GCP operators to be similar, so it is important for 
me to elaborate one variant.
   
   CC: @potiuk 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #5043: [AIRFLOW-4236] Add num_retries to MySqlToGoogleCloudStorageOperator

codecov-io edited a comment on issue #5043: [AIRFLOW-4236] Add num_retries to 
MySqlToGoogleCloudStorageOperator
URL: https://github.com/apache/airflow/pull/5043#issuecomment-480259600
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5043?src=pr=h1) 
Report
   > Merging 
[#5043](https://codecov.io/gh/apache/airflow/pull/5043?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/b93f2649aed7fe566b987ec4cb135a2ec643acd6?src=pr=desc)
 will **decrease** coverage by `0.12%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5043/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5043?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#5043  +/-   ##
   ==
   - Coverage   76.36%   76.24%   -0.13% 
   ==
 Files 471  466   -5 
 Lines   3029030102 -188 
   ==
   - Hits2313022950 -180 
   + Misses   7160 7152   -8
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/5043?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/contrib/operators/mysql\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/5043/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9teXNxbF90b19nY3MucHk=)
 | `90.14% <100%> (+0.06%)` | :arrow_up: |
   | 
[airflow/contrib/hooks/cloudant\_hook.py](https://codecov.io/gh/apache/airflow/pull/5043/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2Nsb3VkYW50X2hvb2sucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/utils/helpers.py](https://codecov.io/gh/apache/airflow/pull/5043/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9oZWxwZXJzLnB5)
 | `82.51% <0%> (-0.36%)` | :arrow_down: |
   | 
[airflow/contrib/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/5043/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4ZWN1dG9ycy9rdWJlcm5ldGVzX2V4ZWN1dG9yLnB5)
 | `63.17% <0%> (-0.21%)` | :arrow_down: |
   | 
[airflow/contrib/kubernetes/worker\_configuration.py](https://codecov.io/gh/apache/airflow/pull/5043/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2t1YmVybmV0ZXMvd29ya2VyX2NvbmZpZ3VyYXRpb24ucHk=)
 | `95.72% <0%> (-0.18%)` | :arrow_down: |
   | 
[airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/5043/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==)
 | `90.29% <0%> (-0.1%)` | :arrow_down: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/5043/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `98.59% <0%> (-0.04%)` | :arrow_down: |
   | 
[.../kubernetes\_request\_factory/pod\_request\_factory.py](https://codecov.io/gh/apache/airflow/pull/5043/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2t1YmVybmV0ZXMva3ViZXJuZXRlc19yZXF1ZXN0X2ZhY3RvcnkvcG9kX3JlcXVlc3RfZmFjdG9yeS5weQ==)
 | `100% <0%> (ø)` | :arrow_up: |
   | 
[...example\_dags/example\_kubernetes\_executor\_config.py](https://codecov.io/gh/apache/airflow/pull/5043/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2t1YmVybmV0ZXNfZXhlY3V0b3JfY29uZmlnLnB5)
 | `0% <0%> (ø)` | :arrow_up: |
   | 
[airflow/contrib/hooks/gcp\_api\_base\_hook.py](https://codecov.io/gh/apache/airflow/pull/5043/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF9hcGlfYmFzZV9ob29rLnB5)
 | `84.76% <0%> (ø)` | :arrow_up: |
   | ... and [7 
more](https://codecov.io/gh/apache/airflow/pull/5043/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5043?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5043?src=pr=footer). 
Last update 
[b93f264...d32beb9](https://codecov.io/gh/apache/airflow/pull/5043?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #5007: [AIRFLOW-XXX] Remove duplicate information

mik-laj commented on issue #5007: [AIRFLOW-XXX] Remove duplicate information
URL: https://github.com/apache/airflow/pull/5007#issuecomment-480461716
 
 
   I rebased PR.  
   New preview: http://tall-smile.surge.sh/integration.html
   In addition, I corrected the names of the services to be consistent with the 
official naming and sorted alphabetically.
   
   PTAL @kaxil  @ashb 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #5049: [AIRFLOW-3063] - Make DockerOperator support all docker arguments

mik-laj commented on a change in pull request #5049: [AIRFLOW-3063] - Make 
DockerOperator support all docker arguments
URL: https://github.com/apache/airflow/pull/5049#discussion_r272773755
 
 

 ##
 File path: airflow/operators/docker_operator.py
 ##
 @@ -235,17 +244,9 @@ def execute(self, context):
 if result['StatusCode'] != 0:
 raise AirflowException('docker container failed: ' + 
repr(result))
 
-# duplicated conditional logic because of expensive operation
-if self.do_xcom_push:
+if self.xcom_push_flag:
 
 Review comment:
   Can you keep a comment?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #5049: [AIRFLOW-3063] - Make DockerOperator support all docker arguments

mik-laj commented on a change in pull request #5049: [AIRFLOW-3063] - Make 
DockerOperator support all docker arguments
URL: https://github.com/apache/airflow/pull/5049#discussion_r272773523
 
 

 ##
 File path: airflow/operators/docker_operator.py
 ##
 @@ -93,86 +72,128 @@ class DockerOperator(BaseOperator):
 The path is also made available via the environment variable
 ``AIRFLOW_TMP_DIR`` inside the container.
 :type tmp_dir: str
-:param user: Default user inside the docker container.
-:type user: int or str
-:param volumes: List of volumes to mount into the container, e.g.
-``['/host/path:/container/path', '/host/path2:/container/path2:ro']``.
-:param working_dir: Working directory to
-set on the container (equivalent to the -w switch the docker client)
-:type working_dir: str
+:param xcom_push: Does the stdout will be pushed to the next step using 
XCom.
+The default is False.
+:type xcom_push: bool
 :param xcom_all: Push all the stdout or just the last line.
 The default is False (last line).
 :type xcom_all: bool
 :param docker_conn_id: ID of the Airflow connection to use
 :type docker_conn_id: str
-:param shm_size: Size of ``/dev/shm`` in bytes. The size must be
-greater than 0. If omitted uses system default.
-:type shm_size: int
+:param container_config: a json that define all the config of the 
container .
+:type container_config: json object
+** Use of container_config**
+Takes a json object only, you can provide all the supported arguments 
in
+create_container method. Check create_container method documentation 
for
+more details: https://goo.gl/u5PQqE
+Note: the image ``image`` attribute is mandatory.
+
+.. code-block:: python
+
+container_config = {'image':'ubuntu:latest',
+'command': 'echo hello world!'
+'environment':{'SOME_ENV_VAR':'SOME_VALUE'},
+'working_dir':'/container/some_path'
+}
+:param host_config: a json that define all host config of the container
+:type host_config: json object
+**Use of host_config**
 
 Review comment:
   ```suggestion
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #5049: [AIRFLOW-3063] - Make DockerOperator support all docker arguments

mik-laj commented on a change in pull request #5049: [AIRFLOW-3063] - Make 
DockerOperator support all docker arguments
URL: https://github.com/apache/airflow/pull/5049#discussion_r272773305
 
 

 ##
 File path: airflow/operators/docker_operator.py
 ##
 @@ -93,86 +72,128 @@ class DockerOperator(BaseOperator):
 The path is also made available via the environment variable
 ``AIRFLOW_TMP_DIR`` inside the container.
 :type tmp_dir: str
-:param user: Default user inside the docker container.
-:type user: int or str
-:param volumes: List of volumes to mount into the container, e.g.
-``['/host/path:/container/path', '/host/path2:/container/path2:ro']``.
-:param working_dir: Working directory to
-set on the container (equivalent to the -w switch the docker client)
-:type working_dir: str
+:param xcom_push: Does the stdout will be pushed to the next step using 
XCom.
+The default is False.
+:type xcom_push: bool
 :param xcom_all: Push all the stdout or just the last line.
 The default is False (last line).
 :type xcom_all: bool
 :param docker_conn_id: ID of the Airflow connection to use
 :type docker_conn_id: str
-:param shm_size: Size of ``/dev/shm`` in bytes. The size must be
-greater than 0. If omitted uses system default.
-:type shm_size: int
+:param container_config: a json that define all the config of the 
container .
+:type container_config: json object
+** Use of container_config**
+Takes a json object only, you can provide all the supported arguments 
in
+create_container method. Check create_container method documentation 
for
+more details: https://goo.gl/u5PQqE
+Note: the image ``image`` attribute is mandatory.
+
+.. code-block:: python
+
+container_config = {'image':'ubuntu:latest',
+'command': 'echo hello world!'
+'environment':{'SOME_ENV_VAR':'SOME_VALUE'},
+'working_dir':'/container/some_path'
+}
+:param host_config: a json that define all host config of the container
+:type host_config: json object
+**Use of host_config**
+Takes a json object only, you can provide all the supported arguments 
in
+create_host_config method. Check create_host_config method 
documentation for
+more details: https://goo.gl/65NcDi
 
 Review comment:
   Can you use the full address?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #5049: [AIRFLOW-3063] - Make DockerOperator support all docker arguments

mik-laj commented on a change in pull request #5049: [AIRFLOW-3063] - Make 
DockerOperator support all docker arguments
URL: https://github.com/apache/airflow/pull/5049#discussion_r272773351
 
 

 ##
 File path: airflow/operators/docker_operator.py
 ##
 @@ -93,86 +72,128 @@ class DockerOperator(BaseOperator):
 The path is also made available via the environment variable
 ``AIRFLOW_TMP_DIR`` inside the container.
 :type tmp_dir: str
-:param user: Default user inside the docker container.
-:type user: int or str
-:param volumes: List of volumes to mount into the container, e.g.
-``['/host/path:/container/path', '/host/path2:/container/path2:ro']``.
-:param working_dir: Working directory to
-set on the container (equivalent to the -w switch the docker client)
-:type working_dir: str
+:param xcom_push: Does the stdout will be pushed to the next step using 
XCom.
+The default is False.
+:type xcom_push: bool
 :param xcom_all: Push all the stdout or just the last line.
 The default is False (last line).
 :type xcom_all: bool
 :param docker_conn_id: ID of the Airflow connection to use
 :type docker_conn_id: str
-:param shm_size: Size of ``/dev/shm`` in bytes. The size must be
-greater than 0. If omitted uses system default.
-:type shm_size: int
+:param container_config: a json that define all the config of the 
container .
+:type container_config: json object
 
 Review comment:
   ```suggestion
   :type container_config: dict
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Assigned] (AIRFLOW-2289) Add additional quick start to INSTALL

2019-04-05 Thread siyuan guo (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

siyuan guo reassigned AIRFLOW-2289:
---

Assignee: (was: siyuan guo)

> Add additional quick start to INSTALL
> -
>
> Key: AIRFLOW-2289
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2289
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Bolke de Bruin
>Priority: Blocker
> Fix For: 1.10.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] mik-laj commented on issue #5049: [AIRFLOW-3063] - Make DockerOperator support all docker arguments

mik-laj commented on issue #5049: [AIRFLOW-3063] - Make DockerOperator support 
all docker arguments
URL: https://github.com/apache/airflow/pull/5049#issuecomment-480455326
 
 
   Is your change fully backwards compatible? If it's compatible then I'm very 
happy.
   If it is not I have another question. Is it possible to provide backward 
compatibility?  e.g. by adding support for two ways at the same time. If this 
is not possible, please add a note in the file `UPDATING.md`.
   
   I saw that the operator does not have the full documentation described, but 
it would be useful. Using this operator is a common scenario. Would you like to 
write a guide on this topic? 
   Example of a guide: 
   
https://airflow.readthedocs.io/en/latest/howto/operator/gcp/natural_language.html


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Assigned] (AIRFLOW-2289) Add additional quick start to INSTALL

2019-04-05 Thread siyuan guo (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

siyuan guo reassigned AIRFLOW-2289:
---

Assignee: siyuan guo

> Add additional quick start to INSTALL
> -
>
> Key: AIRFLOW-2289
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2289
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Bolke de Bruin
>Assignee: siyuan guo
>Priority: Blocker
> Fix For: 1.10.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] milton0825 commented on issue #5050: [AIRFLOW-4251] Instrument DagRun schedule delay

milton0825 commented on issue #5050: [AIRFLOW-4251] Instrument DagRun schedule 
delay
URL: https://github.com/apache/airflow/pull/5050#issuecomment-480454740
 
 
   @feng-tao 
   What I am measuring here is the delay between the expected start date and 
the actual start date of a `DagRun`. Both expected start date and actual start 
date are dependent on the  `schedule_interval` because that is how Airflow 
works (scheduled start_date = execution_date + schedule_interval). The 
`schedule_delay` itself is not dependent on the `schedule_interval` tho.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #5043: [AIRFLOW-4236] Add num_retries to MySqlToGoogleCloudStorageOperator

mik-laj commented on issue #5043: [AIRFLOW-4236] Add num_retries to 
MySqlToGoogleCloudStorageOperator
URL: https://github.com/apache/airflow/pull/5043#issuecomment-480453668
 
 
   Other operators have a fixed number of retry attempts. 
   
https://github.com/apache/airflow/blob/e27950a75ce287c094e550fba07d1c8de5dc4143/airflow/contrib/hooks/gcp_transfer_hook.py#L92
   
https://github.com/apache/airflow/blob/383ec374ea148d04af94496dc9e3283263d67d5f/airflow/contrib/hooks/gcp_function_hook.py#L29
   
https://github.com/apache/airflow/blob/993779f93df172f5f774f1ea2634c55e9c1212c4/airflow/contrib/hooks/gcp_compute_hook.py#L28
   
https://github.com/apache/airflow/blob/49ade3834a46d9a12c0339a017d1807bf021c5e3/airflow/contrib/hooks/gcp_sql_hook.py#L54
   
https://github.com/apache/airflow/blob/85ea7e8575bb0f9a2fac989f26970809b36640a4/airflow/contrib/hooks/gcp_dataflow_hook.py#L51
   I do not know if it's a good idea to give the user the option to configure 
this parameter in this way. Too many parameters will make it difficult to use 
the operator. In my opinion, this value should be configured in the connection 
settings instead of the specific use, but fixed value is good enough in this 
case.
   
   I working on similar mechanism for Google Cloud Vision: 
https://github.com/PolideaInternal/airflow/pull/85/files 
   I want to support retrying in reaction to exceeding the temporary quota. I 
would like the use of all GCP operators to be similar, so it is important for 
me to elaborate one variant.
   
   CC: @potiuk 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj edited a comment on issue #5043: [AIRFLOW-4236] Add num_retries to MySqlToGoogleCloudStorageOperator

mik-laj edited a comment on issue #5043: [AIRFLOW-4236] Add num_retries to 
MySqlToGoogleCloudStorageOperator
URL: https://github.com/apache/airflow/pull/5043#issuecomment-480453668
 
 
   Other operators have a fixed number of retry attempts. 
   
https://github.com/apache/airflow/blob/e27950a75ce287c094e550fba07d1c8de5dc4143/airflow/contrib/hooks/gcp_transfer_hook.py#L92
   
https://github.com/apache/airflow/blob/383ec374ea148d04af94496dc9e3283263d67d5f/airflow/contrib/hooks/gcp_function_hook.py#L29
   
https://github.com/apache/airflow/blob/993779f93df172f5f774f1ea2634c55e9c1212c4/airflow/contrib/hooks/gcp_compute_hook.py#L28
   
https://github.com/apache/airflow/blob/49ade3834a46d9a12c0339a017d1807bf021c5e3/airflow/contrib/hooks/gcp_sql_hook.py#L54
   
https://github.com/apache/airflow/blob/85ea7e8575bb0f9a2fac989f26970809b36640a4/airflow/contrib/hooks/gcp_dataflow_hook.py#L51
   I do not know if it's a good idea to give the user the option to configure 
this parameter in this way. Too many parameters will make it difficult to use 
the operator. In my opinion, this value should be configured in the connection 
settings instead of the specific use, but fixed value is good enough in this 
case.
   
   I working on similar mechanism for Google Cloud AI operators: 
https://github.com/PolideaInternal/airflow/pull/85/files 
   I want to support retrying in reaction to exceeding the temporary quota. I 
would like the use of all GCP operators to be similar, so it is important for 
me to elaborate one variant.
   
   CC: @potiuk 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feng-tao commented on issue #5050: [AIRFLOW-4251] Instrument DagRun schedule delay

feng-tao commented on issue #5050: [AIRFLOW-4251] Instrument DagRun schedule 
delay
URL: https://github.com/apache/airflow/pull/5050#issuecomment-480453274
 
 
   correct me if I am wrong, but could you help me understand why we need this 
stat given it is highly dependent on the DAG's schedule interval?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] spinus commented on issue #4268: [AIRFLOW-3412] Fix kubernetes executor to delete pods after termination

spinus commented on issue #4268: [AIRFLOW-3412] Fix kubernetes executor to 
delete pods after termination
URL: https://github.com/apache/airflow/pull/4268#issuecomment-480449541
 
 
   Looks like another PR was merged with similar function 
(https://github.com/apache/airflow/pull/4163/files), probably that can be 
closed, correct?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ryanyuan commented on issue #5043: [AIRFLOW-4236] Add num_retries to MySqlToGoogleCloudStorageOperator

2019-04-05 Thread ASF subversion and git services (JIRA)

ryanyuan commented on issue #5043: [AIRFLOW-4236] Add num_retries to 
MySqlToGoogleCloudStorageOperator
URL: https://github.com/apache/airflow/pull/5043#issuecomment-480446313
 
 
   @OmerJog Cool. I will get it done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3274) Add run_as_user and fs_group security context options for KubernetesExecutor



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811312#comment-16811312
 ] 

ASF subversion and git services commented on AIRFLOW-3274:
--

Commit be20160f88d8d4ea858d989cd0868004eb9e65de in airflow's branch 
refs/heads/v1-10-test from Philippe Gagnon
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=be20160 ]

[AIRFLOW-3274] Add run_as_user and fs_group options for Kubernetes (#4648)



> Add run_as_user and fs_group security context options for KubernetesExecutor
> 
>
> Key: AIRFLOW-3274
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3274
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: kubernetes, scheduler
>Reporter: Philippe Gagnon
>Priority: Major
> Fix For: 1.10.3
>
>
> At this time it is not possible to add `run_as_user` or `fs_group` 
> securityContext options to worker pods when using KubernetesExecutor. This 
> makes it harder to use KubernetesExecutor on clusters with pod security 
> policies which do not allow containers to run as root.
> I have already implemented this functionality for my internal use and will 
> propose a PR soon.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4247) Template the region of the DataprocOperator

2019-04-05 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811309#comment-16811309
 ] 

ASF subversion and git services commented on AIRFLOW-4247:
--

Commit d4d47f0615b8732998f9d95195b396ef5088cdf8 in airflow's branch 
refs/heads/v1-10-test from Fokko Driesprong
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=d4d47f0 ]

[AIRFLOW-4247] Template Region on the DataprocOperators (#5046)



> Template the region of the DataprocOperator
> ---
>
> Key: AIRFLOW-4247
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4247
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: gcp
>Affects Versions: 1.10.2
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
> Fix For: 1.10.3
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4220) Add tests for CloudantHook

2019-04-05 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811319#comment-16811319
 ] 

ASF subversion and git services commented on AIRFLOW-4220:
--

Commit b93f2649aed7fe566b987ec4cb135a2ec643acd6 in airflow's branch 
refs/heads/master from Felix Uellendall
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=b93f264 ]

[AIRFLOW-4220] Change CloudantHook to a new major version and add tests (#5023)

- upgrade cloudant version from `>=0.5.9,<2.0` to `>=2.0`
- remove the use of the `schema` attribute in the connection
- remove `db` function since the database object can also be retrieved by 
calling `cloudant_session['database_name']`
- update docs
- refactor code

> Add tests for CloudantHook
> --
>
> Key: AIRFLOW-4220
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4220
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4122) Remove chain function in airflow.utils

2019-04-05 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811318#comment-16811318
 ] 

ASF subversion and git services commented on AIRFLOW-4122:
--

Commit e732006fdd716898411d07a36004b2b3e57be18c in airflow's branch 
refs/heads/master from Kaxil Naik
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=e732006 ]

revert [AIRFLOW-4122] Remove chain function

Reverts 2 commits:
- ee71a8bb102dcc3a591c5c175ab88a9043cffb0f
- 430efc9afb23d7fe2f88e2bac2e3c45825218410


> Remove chain function in airflow.utils
> --
>
> Key: AIRFLOW-4122
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4122
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: utils
>Affects Versions: 1.10.2
>Reporter: zhongjiajie
>Assignee: zhongjiajie
>Priority: Major
>  Labels: patch
> Fix For: 1.10.3
>
>
> I had submit [pr|[https://github.com/apache/airflow/pull/4779]] and it 
> related to Jira ticket . 
> In PR comment, I was suggest close this PR and submit new one to remove 
> function `chain` in `airflow.utils.heplers`, due to Airflower think use bit 
> operation like `>>` or `<<` visual and easier to explain, and have multiple 
> ways is confusion to users. BTW, this function was created 3 year ago and 
> never get updated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4248) FileExistsError when creating log dir in file_processor_handler.py

2019-04-05 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811311#comment-16811311
 ] 

ASF subversion and git services commented on AIRFLOW-4248:
--

Commit 45270cd3cfbec9d6f61794c4ddd647d6237d8603 in airflow's branch 
refs/heads/v1-10-test from Kevin Pullin
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=45270cd ]

[AIRFLOW-4248] Fix 'FileExistsError' makedirs race in file_processor_handler 
(#5047)



> FileExistsError when creating log dir in file_processor_handler.py
> --
>
> Key: AIRFLOW-4248
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4248
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Kevin Pullin
>Assignee: Kevin Pullin
>Priority: Major
>
> Similar to AIRFLOW-2065, the `_init_file` method in `file_processor_handler` 
> does not handle the race condition of multiple processes creating the same 
> directory on a shared logging filesystem.
> If the error occurs in the scheduler process, tasks do not execute (at least 
> when using the Kubernetes executor). However the process remains alive and 
> requires a restart to resume running tasks.
>  
> Example log message:
>  
> {noformat}
> [2019-04-03 23:59:55,091] {{sqlalchemy.py:79}} WARNING - DB connection 
> invalidated. Reconnecting...
> Process DagFileProcessor41527-Process:
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.6/multiprocessing/process.py", line 258, in 
> _bootstrap
>     self.run()
>   File "/usr/local/lib/python3.6/multiprocessing/process.py", line 93, in run
>     self._target(*self._args, **self._kwargs)
>   File "/usr/local/lib/python3.6/site-packages/airflow/jobs.py", line 374, in 
> helper
>     set_context(log, file_path)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/utils/log/logging_mixin.py", 
> line 170, in set_context
>     handler.set_context(value)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/utils/log/file_processor_handler.py",
>  line 66, in set_context
>     local_loc = self._init_file(filename)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/utils/log/file_processor_handler.py",
>  line 141, in _init_file
>     os.makedirs(directory)
>   File "/usr/local/lib/python3.6/os.py", line 220, in makedirs
>     mkdir(name, mode)
> FileExistsError: [Errno 17] File exists: 
> '/usr/local/airflow/logs/scheduler/2019-04-04'{noformat}
>  
> The path `/usr/local/airflow/logs` is on a shared NFS volume.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4240) State changing actions shouldn't be GET requests

2019-04-05 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811310#comment-16811310
 ] 

ASF subversion and git services commented on AIRFLOW-4240:
--

Commit eb4d3ecc23c520ebb0ed32532b1669081c362e9b in airflow's branch 
refs/heads/v1-10-test from Ash Berlin-Taylor
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=eb4d3ec ]

[AIRFLOW-4240] State-changing actions should be POST requests (#5039)

To make the requests POSTs and to follow the redirect that the backend
issue I turned the "toggle" buttons in to an actual form, which makes
there much less logic needed to build up the URL - the browser handles
it all for us. The only thing we have to do is set the "action" on the
URL.

For the "link" ones (delete,trigger,refresh) I wrote a short
`postAsForm` which takes the URL and submits a form. A little bit messy,
but it works.

> State changing actions shouldn't be GET requests
> 
>
> Key: AIRFLOW-4240
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4240
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Ash Berlin-Taylor
>Assignee: Ash Berlin-Taylor
>Priority: Major
> Fix For: 1.10.3
>
>
> We have a number of actions which perform actions (trigger, clear, etc) that 
> are performed over GET requests.
> That should be avoided as browsers/corporate proxies might prefetch the URLs 
> causing things to behave oddly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4122) Remove chain function in airflow.utils

2019-04-05 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811313#comment-16811313
 ] 

ASF subversion and git services commented on AIRFLOW-4122:
--

Commit 9988fdb1bd7b97e0e23396ba3fa373af419da7d9 in airflow's branch 
refs/heads/v1-10-test from Ash Berlin-Taylor
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=9988fdb ]

Revert "[AIRFLOW-4122] Deprecate chain function (#4940)"

This reverts commit c1a23e62f4c92b64208de4029497af5df48bff69.

This is still useful for larger/more complex DAGs


> Remove chain function in airflow.utils
> --
>
> Key: AIRFLOW-4122
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4122
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: utils
>Affects Versions: 1.10.2
>Reporter: zhongjiajie
>Assignee: zhongjiajie
>Priority: Major
>  Labels: patch
> Fix For: 1.10.3
>
>
> I had submit [pr|[https://github.com/apache/airflow/pull/4779]] and it 
> related to Jira ticket . 
> In PR comment, I was suggest close this PR and submit new one to remove 
> function `chain` in `airflow.utils.heplers`, due to Airflower think use bit 
> operation like `>>` or `<<` visual and easier to explain, and have multiple 
> ways is confusion to users. BTW, this function was created 3 year ago and 
> never get updated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4246) Flask-OAuthlib has breaking changes in dependencies.

2019-04-05 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811308#comment-16811308
 ] 

ASF subversion and git services commented on AIRFLOW-4246:
--

Commit b24489f75c6eb86ea6186a3483ee09f2fb3926a1 in airflow's branch 
refs/heads/v1-10-test from Paul Bramhall
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=b24489f ]

[AIRFLOW-4246] Flask-Oauthlib needs downstream dependencies pinning due to 
breaking changes (#5045)


> Flask-OAuthlib has breaking changes in dependencies.
> 
>
> Key: AIRFLOW-4246
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4246
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: dependencies
>Affects Versions: 1.10.2
>Reporter: Paul Bramhall
>Priority: Major
>  Labels: build
>
> When specifying either 'google_auth' or 'github_enterprise' as a requirement 
> when building airflow, this pulls in a pinned version of the following 
> package:
> Flask-OAuthlib>=0.9.1
> However, downstream dependencies have since been updated, which causes 
> breaking changes to this version of Flask-OAuthlib:
> [https://github.com/lepture/flask-oauthlib/issues/390]
> Specifically, we need to also lock the following packages when installing 
> this dependency:
> oauthlib!=2.0.3,!=2.0.4,!=2.0.5,<3.0.0,>=1.1.2
> requests-oauthlib==1.1.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4069) Add Opsgenie Alert Hook and Operator

2019-04-05 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811314#comment-16811314
 ] 

ASF subversion and git services commented on AIRFLOW-4069:
--

Commit d949340429fc2917e24f977a7842c8a6c9aa5605 in airflow's branch 
refs/heads/v1-10-test from Nathaniel Ritholtz
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=d949340 ]

[AIRFLOW-4069] Add Opsgenie Alert Hook and Operator (#4903)



> Add Opsgenie Alert Hook and Operator
> 
>
> Key: AIRFLOW-4069
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4069
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: hooks, operators
>Affects Versions: 1.10.2
>Reporter: Nathaniel Ritholtz
>Assignee: Nathaniel Ritholtz
>Priority: Minor
> Fix For: 1.10.3
>
>
> Add a hook and operator for the [Opsgenie Alert API 
> |https://app.opsgenie.com/alert#/alert-genie] to create alerts.
> We are using Opsgenie internally for alerting, and would optimally want to 
> use a hook or operator within our *on_failure_callback* to create an Opsgenie 
> alert for DAG failures.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-3996) Include full source code examples in the documentation

2019-04-05 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811315#comment-16811315
 ] 

ASF subversion and git services commented on AIRFLOW-3996:
--

Commit 96fbc6141c1dd133741ef674dbd9e580a5ee81a9 in airflow's branch 
refs/heads/v1-10-test from Kamil Breguła
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=96fbc61 ]

[AIRFLOW-3996] Add view source link to included fragments


> Include full source code examples in the documentation
> --
>
> Key: AIRFLOW-3996
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3996
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Kamil Bregula
>Priority: Major
> Fix For: 1.10.3
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-3971) Google Cloud Natural Language Operators

2019-04-05 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811316#comment-16811316
 ] 

ASF subversion and git services commented on AIRFLOW-3971:
--

Commit 91b856fce05e4ec73ceef19892ac2fa5606fa51d in airflow's branch 
refs/heads/v1-10-test from Kamil Breguła
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=91b856f ]

[AIRFLOW-3971] Add Google Cloud Natural Language operators (#4980)


> Google Cloud Natural Language Operators
> ---
>
> Key: AIRFLOW-3971
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3971
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Kamil Bregula
>Assignee: Kamil Bregula
>Priority: Major
> Fix For: 1.10.3
>
>
> Hello, 
> I want to create a new set of operors:
>  
> |*Operator name*|*API URL /description*|
> |CloudLanguageAnalyzeEntities|[Docs|https://googleapis.github.io/google-cloud-python/latest/language/gapic/v1/api.html#google.cloud.language_v1.LanguageServiceClient.analyze_entities]|
> |CloudLanguageAnalyzeEntitySentiment|[Docs|https://googleapis.github.io/google-cloud-python/latest/language/gapic/v1/api.html#google.cloud.language_v1.LanguageServiceClient.analyze_entity_sentiment]|
> |CloudLanguageAnalyzeSentiment|[Docs|https://googleapis.github.io/google-cloud-python/latest/language/gapic/v1/api.html#google.cloud.language_v1.LanguageServiceClient.analyze_sentiment]|
> |CloudLanguageClassifyTextOperator|[Docs|https://googleapis.github.io/google-cloud-python/latest/language/gapic/v1/api.html#google.cloud.language_v1.LanguageServiceClient.classify_text]|
> Greetings
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] cmdoptesc commented on issue #5032: [AIRFLOW-4232] Add `none_skipped` trigger rule

cmdoptesc commented on issue #5032: [AIRFLOW-4232] Add `none_skipped` trigger 
rule
URL: https://github.com/apache/airflow/pull/5032#issuecomment-480436002
 
 
   Thanks a million @ashb  
   
   Yeah, all the other tests were just using strings, which would probably 
error out if the `flag_upstream_failed` flag was set to true.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-3996) Include full source code examples in the documentation

2019-04-05 Thread Kaxil Naik (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik resolved AIRFLOW-3996.
-
   Resolution: Fixed
Fix Version/s: 1.10.3

> Include full source code examples in the documentation
> --
>
> Key: AIRFLOW-3996
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3996
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Kamil Bregula
>Priority: Major
> Fix For: 1.10.3
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] ashb commented on issue #5032: [AIRFLOW-4232] Add `none_skipped` trigger rule

2019-04-05 Thread ASF subversion and git services (JIRA)

ashb commented on issue #5032: [AIRFLOW-4232] Add `none_skipped` trigger rule
URL: https://github.com/apache/airflow/pull/5032#issuecomment-480435069
 
 
   I've made an update to this branch to try and fix those tests @cmdoptesc - I 
didn't realise that none of the other paths hit the upstream_failed path :/


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3971) Google Cloud Natural Language Operators



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811304#comment-16811304
 ] 

ASF subversion and git services commented on AIRFLOW-3971:
--

Commit b789afd667e07d14beffd46978ff4a48d069789a in airflow's branch 
refs/heads/v1-10-stable from Kamil Breguła
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=b789afd ]

[AIRFLOW-3971] Add Google Cloud Natural Language operators (#4980)


> Google Cloud Natural Language Operators
> ---
>
> Key: AIRFLOW-3971
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3971
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Kamil Bregula
>Assignee: Kamil Bregula
>Priority: Major
> Fix For: 1.10.3
>
>
> Hello, 
> I want to create a new set of operors:
>  
> |*Operator name*|*API URL /description*|
> |CloudLanguageAnalyzeEntities|[Docs|https://googleapis.github.io/google-cloud-python/latest/language/gapic/v1/api.html#google.cloud.language_v1.LanguageServiceClient.analyze_entities]|
> |CloudLanguageAnalyzeEntitySentiment|[Docs|https://googleapis.github.io/google-cloud-python/latest/language/gapic/v1/api.html#google.cloud.language_v1.LanguageServiceClient.analyze_entity_sentiment]|
> |CloudLanguageAnalyzeSentiment|[Docs|https://googleapis.github.io/google-cloud-python/latest/language/gapic/v1/api.html#google.cloud.language_v1.LanguageServiceClient.analyze_sentiment]|
> |CloudLanguageClassifyTextOperator|[Docs|https://googleapis.github.io/google-cloud-python/latest/language/gapic/v1/api.html#google.cloud.language_v1.LanguageServiceClient.classify_text]|
> Greetings
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AIRFLOW-3971) Google Cloud Natural Language Operators

2019-04-05 Thread Kaxil Naik (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik resolved AIRFLOW-3971.
-
   Resolution: Fixed
Fix Version/s: 1.10.3

> Google Cloud Natural Language Operators
> ---
>
> Key: AIRFLOW-3971
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3971
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Kamil Bregula
>Assignee: Kamil Bregula
>Priority: Major
> Fix For: 1.10.3
>
>
> Hello, 
> I want to create a new set of operors:
>  
> |*Operator name*|*API URL /description*|
> |CloudLanguageAnalyzeEntities|[Docs|https://googleapis.github.io/google-cloud-python/latest/language/gapic/v1/api.html#google.cloud.language_v1.LanguageServiceClient.analyze_entities]|
> |CloudLanguageAnalyzeEntitySentiment|[Docs|https://googleapis.github.io/google-cloud-python/latest/language/gapic/v1/api.html#google.cloud.language_v1.LanguageServiceClient.analyze_entity_sentiment]|
> |CloudLanguageAnalyzeSentiment|[Docs|https://googleapis.github.io/google-cloud-python/latest/language/gapic/v1/api.html#google.cloud.language_v1.LanguageServiceClient.analyze_sentiment]|
> |CloudLanguageClassifyTextOperator|[Docs|https://googleapis.github.io/google-cloud-python/latest/language/gapic/v1/api.html#google.cloud.language_v1.LanguageServiceClient.classify_text]|
> Greetings
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4220) Add tests for CloudantHook

2019-04-05 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811298#comment-16811298
 ] 

ASF GitHub Bot commented on AIRFLOW-4220:
-

kaxil commented on pull request #5023: [AIRFLOW-4220] Change CloudantHook and 
add tests
URL: https://github.com/apache/airflow/pull/5023
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add tests for CloudantHook
> --
>
> Key: AIRFLOW-4220
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4220
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AIRFLOW-4220) Add tests for CloudantHook

2019-04-05 Thread Kaxil Naik (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik resolved AIRFLOW-4220.
-
   Resolution: Fixed
Fix Version/s: 2.0.0

> Add tests for CloudantHook
> --
>
> Key: AIRFLOW-4220
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4220
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4220) Add tests for CloudantHook



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811299#comment-16811299
 ] 

ASF subversion and git services commented on AIRFLOW-4220:
--

Commit f74064ecda7f835c7026faf5f27ba4b801dadf0e in airflow's branch 
refs/heads/master from Felix Uellendall
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f74064e ]

[AIRFLOW-4220] Change CloudantHook to a new major version and add tests (#5023)

- upgrade cloudant version from `>=0.5.9,<2.0` to `>=2.0`
- remove the use of the `schema` attribute in the connection
- remove `db` function since the database object can also be retrieved by 
calling `cloudant_session['database_name']`
- update docs
- refactor code

> Add tests for CloudantHook
> --
>
> Key: AIRFLOW-4220
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4220
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] kaxil merged pull request #5023: [AIRFLOW-4220] Change CloudantHook and add tests

2019-04-05 Thread ASF subversion and git services (JIRA)

kaxil merged pull request #5023: [AIRFLOW-4220] Change CloudantHook and add 
tests
URL: https://github.com/apache/airflow/pull/5023
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3996) Include full source code examples in the documentation



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811295#comment-16811295
 ] 

ASF subversion and git services commented on AIRFLOW-3996:
--

Commit 2200339c91f2d13a1b67530a09d1269bf876e8cb in airflow's branch 
refs/heads/v1-10-stable from Kamil Breguła
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=2200339 ]

[AIRFLOW-3996] Add view source link to included fragments


> Include full source code examples in the documentation
> --
>
> Key: AIRFLOW-3996
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3996
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Kamil Bregula
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] ashb commented on issue #5032: [AIRFLOW-4232] Add `none_skipped` trigger rule

ashb commented on issue #5032: [AIRFLOW-4232] Add `none_skipped` trigger rule
URL: https://github.com/apache/airflow/pull/5032#issuecomment-480431547
 
 
   `NOT NULL constraint failed: task_instance.execution_date` now


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-4069) Add Opsgenie Alert Hook and Operator



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor resolved AIRFLOW-4069.

   Resolution: Fixed
Fix Version/s: 1.10.3

> Add Opsgenie Alert Hook and Operator
> 
>
> Key: AIRFLOW-4069
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4069
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: hooks, operators
>Affects Versions: 1.10.2
>Reporter: Nathaniel Ritholtz
>Assignee: Nathaniel Ritholtz
>Priority: Minor
> Fix For: 1.10.3
>
>
> Add a hook and operator for the [Opsgenie Alert API 
> |https://app.opsgenie.com/alert#/alert-genie] to create alerts.
> We are using Opsgenie internally for alerting, and would optimally want to 
> use a hook or operator within our *on_failure_callback* to create an Opsgenie 
> alert for DAG failures.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4069) Add Opsgenie Alert Hook and Operator



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811289#comment-16811289
 ] 

ASF GitHub Bot commented on AIRFLOW-4069:
-

ashb commented on pull request #4903: [AIRFLOW-4069] Add Opsgenie Alert Hook 
and Operator
URL: https://github.com/apache/airflow/pull/4903
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Opsgenie Alert Hook and Operator
> 
>
> Key: AIRFLOW-4069
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4069
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: hooks, operators
>Affects Versions: 1.10.2
>Reporter: Nathaniel Ritholtz
>Assignee: Nathaniel Ritholtz
>Priority: Minor
>
> Add a hook and operator for the [Opsgenie Alert API 
> |https://app.opsgenie.com/alert#/alert-genie] to create alerts.
> We are using Opsgenie internally for alerting, and would optimally want to 
> use a hook or operator within our *on_failure_callback* to create an Opsgenie 
> alert for DAG failures.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] ashb merged pull request #4903: [AIRFLOW-4069] Add Opsgenie Alert Hook and Operator

2019-04-05 Thread ASF subversion and git services (JIRA)

ashb merged pull request #4903: [AIRFLOW-4069] Add Opsgenie Alert Hook and 
Operator
URL: https://github.com/apache/airflow/pull/4903
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3971) Google Cloud Natural Language Operators



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811287#comment-16811287
 ] 

ASF subversion and git services commented on AIRFLOW-3971:
--

Commit 99c8a6f1258e5fc8a9d4223b0203a0a76611bc78 in airflow's branch 
refs/heads/master from Kamil Breguła
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=99c8a6f ]

[AIRFLOW-3971] Add Google Cloud Natural Language operators (#4980)



> Google Cloud Natural Language Operators
> ---
>
> Key: AIRFLOW-3971
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3971
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Kamil Bregula
>Assignee: Kamil Bregula
>Priority: Major
>
> Hello, 
> I want to create a new set of operors:
>  
> |*Operator name*|*API URL /description*|
> |CloudLanguageAnalyzeEntities|[Docs|https://googleapis.github.io/google-cloud-python/latest/language/gapic/v1/api.html#google.cloud.language_v1.LanguageServiceClient.analyze_entities]|
> |CloudLanguageAnalyzeEntitySentiment|[Docs|https://googleapis.github.io/google-cloud-python/latest/language/gapic/v1/api.html#google.cloud.language_v1.LanguageServiceClient.analyze_entity_sentiment]|
> |CloudLanguageAnalyzeSentiment|[Docs|https://googleapis.github.io/google-cloud-python/latest/language/gapic/v1/api.html#google.cloud.language_v1.LanguageServiceClient.analyze_sentiment]|
> |CloudLanguageClassifyTextOperator|[Docs|https://googleapis.github.io/google-cloud-python/latest/language/gapic/v1/api.html#google.cloud.language_v1.LanguageServiceClient.classify_text]|
> Greetings
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-3971) Google Cloud Natural Language Operators



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811286#comment-16811286
 ] 

ASF GitHub Bot commented on AIRFLOW-3971:
-

kaxil commented on pull request #4980: [AIRFLOW-3971] Add Google Cloud Natural 
Language operators
URL: https://github.com/apache/airflow/pull/4980
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Google Cloud Natural Language Operators
> ---
>
> Key: AIRFLOW-3971
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3971
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Kamil Bregula
>Assignee: Kamil Bregula
>Priority: Major
>
> Hello, 
> I want to create a new set of operors:
>  
> |*Operator name*|*API URL /description*|
> |CloudLanguageAnalyzeEntities|[Docs|https://googleapis.github.io/google-cloud-python/latest/language/gapic/v1/api.html#google.cloud.language_v1.LanguageServiceClient.analyze_entities]|
> |CloudLanguageAnalyzeEntitySentiment|[Docs|https://googleapis.github.io/google-cloud-python/latest/language/gapic/v1/api.html#google.cloud.language_v1.LanguageServiceClient.analyze_entity_sentiment]|
> |CloudLanguageAnalyzeSentiment|[Docs|https://googleapis.github.io/google-cloud-python/latest/language/gapic/v1/api.html#google.cloud.language_v1.LanguageServiceClient.analyze_sentiment]|
> |CloudLanguageClassifyTextOperator|[Docs|https://googleapis.github.io/google-cloud-python/latest/language/gapic/v1/api.html#google.cloud.language_v1.LanguageServiceClient.classify_text]|
> Greetings
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] kaxil merged pull request #4980: [AIRFLOW-3971] Add Google Cloud Natural Language operators

kaxil merged pull request #4980: [AIRFLOW-3971] Add Google Cloud Natural 
Language operators
URL: https://github.com/apache/airflow/pull/4980
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Closed] (AIRFLOW-3274) Add run_as_user and fs_group security context options for KubernetesExecutor

2019-04-05 Thread Philippe Gagnon (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philippe Gagnon closed AIRFLOW-3274.


> Add run_as_user and fs_group security context options for KubernetesExecutor
> 
>
> Key: AIRFLOW-3274
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3274
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: kubernetes, scheduler
>Reporter: Philippe Gagnon
>Priority: Major
> Fix For: 1.10.3
>
>
> At this time it is not possible to add `run_as_user` or `fs_group` 
> securityContext options to worker pods when using KubernetesExecutor. This 
> makes it harder to use KubernetesExecutor on clusters with pod security 
> policies which do not allow containers to run as root.
> I have already implemented this functionality for my internal use and will 
> propose a PR soon.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] kaxil commented on issue #5043: [AIRFLOW-4236] Add num_retries to MySqlToGoogleCloudStorageOperator

kaxil commented on issue #5043: [AIRFLOW-4236] Add num_retries to 
MySqlToGoogleCloudStorageOperator
URL: https://github.com/apache/airflow/pull/5043#issuecomment-480428604
 
 
   Better to address similar issues in one PR then to split it across many


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3274) Add run_as_user and fs_group security context options for KubernetesExecutor



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811256#comment-16811256
 ] 

ASF GitHub Bot commented on AIRFLOW-3274:
-

ashb commented on pull request #4648: [AIRFLOW-3274] Add run_as_user and 
fs_group options for Kubernetes
URL: https://github.com/apache/airflow/pull/4648
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add run_as_user and fs_group security context options for KubernetesExecutor
> 
>
> Key: AIRFLOW-3274
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3274
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: kubernetes, scheduler
>Reporter: Philippe Gagnon
>Priority: Major
>
> At this time it is not possible to add `run_as_user` or `fs_group` 
> securityContext options to worker pods when using KubernetesExecutor. This 
> makes it harder to use KubernetesExecutor on clusters with pod security 
> policies which do not allow containers to run as root.
> I have already implemented this functionality for my internal use and will 
> propose a PR soon.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] pgagnon commented on issue #4648: [AIRFLOW-3274] Add run_as_user and fs_group options for Kubernetes

pgagnon commented on issue #4648: [AIRFLOW-3274] Add run_as_user and fs_group 
options for Kubernetes
URL: https://github.com/apache/airflow/pull/4648#issuecomment-480424185
 
 
   @ashb Thanks for your patience and guidance. ☺️


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-3274) Add run_as_user and fs_group security context options for KubernetesExecutor

2019-04-05 Thread ASF subversion and git services (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor resolved AIRFLOW-3274.

   Resolution: Fixed
Fix Version/s: 1.10.3

> Add run_as_user and fs_group security context options for KubernetesExecutor
> 
>
> Key: AIRFLOW-3274
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3274
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: kubernetes, scheduler
>Reporter: Philippe Gagnon
>Priority: Major
> Fix For: 1.10.3
>
>
> At this time it is not possible to add `run_as_user` or `fs_group` 
> securityContext options to worker pods when using KubernetesExecutor. This 
> makes it harder to use KubernetesExecutor on clusters with pod security 
> policies which do not allow containers to run as root.
> I have already implemented this functionality for my internal use and will 
> propose a PR soon.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-3274) Add run_as_user and fs_group security context options for KubernetesExecutor



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811257#comment-16811257
 ] 

ASF subversion and git services commented on AIRFLOW-3274:
--

Commit 9b33f9b8ec0705f354df039abf37686baeeea679 in airflow's branch 
refs/heads/master from Philippe Gagnon
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=9b33f9b ]

[AIRFLOW-3274] Add run_as_user and fs_group options for Kubernetes (#4648)



> Add run_as_user and fs_group security context options for KubernetesExecutor
> 
>
> Key: AIRFLOW-3274
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3274
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: kubernetes, scheduler
>Reporter: Philippe Gagnon
>Priority: Major
> Fix For: 1.10.3
>
>
> At this time it is not possible to add `run_as_user` or `fs_group` 
> securityContext options to worker pods when using KubernetesExecutor. This 
> makes it harder to use KubernetesExecutor on clusters with pod security 
> policies which do not allow containers to run as root.
> I have already implemented this functionality for my internal use and will 
> propose a PR soon.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] ashb merged pull request #4648: [AIRFLOW-3274] Add run_as_user and fs_group options for Kubernetes

ashb merged pull request #4648: [AIRFLOW-3274] Add run_as_user and fs_group 
options for Kubernetes
URL: https://github.com/apache/airflow/pull/4648
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nritholtz commented on issue #4903: [AIRFLOW-4069] Add Opsgenie Alert Hook and Operator

nritholtz commented on issue #4903: [AIRFLOW-4069] Add Opsgenie Alert Hook and 
Operator
URL: https://github.com/apache/airflow/pull/4903#issuecomment-480421779
 
 
   @ashb Thanks for the rebuild, looks all green now.  


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] pgagnon commented on issue #4648: [AIRFLOW-3274] Add run_as_user and fs_group options for Kubernetes

pgagnon commented on issue #4648: [AIRFLOW-3274] Add run_as_user and fs_group 
options for Kubernetes
URL: https://github.com/apache/airflow/pull/4648#issuecomment-480414653
 
 
   @ashb All green.  


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #4903: [AIRFLOW-4069] Add Opsgenie Alert Hook and Operator

codecov-io edited a comment on issue #4903: [AIRFLOW-4069] Add Opsgenie Alert 
Hook and Operator
URL: https://github.com/apache/airflow/pull/4903#issuecomment-471783078
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/4903?src=pr=h1) 
Report
   > Merging 
[#4903](https://codecov.io/gh/apache/airflow/pull/4903?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/dbed51e702bf8177800183d2c4f595073aa2339d?src=pr=desc)
 will **increase** coverage by `0.05%`.
   > The diff coverage is `96.49%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/4903/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/4903?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4903  +/-   ##
   ==
   + Coverage   76.23%   76.29%   +0.05% 
   ==
 Files 466  468   +2 
 Lines   3010130188  +87 
   ==
   + Hits2294923031  +82 
   - Misses   7152 7157   +5
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/4903?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/contrib/hooks/opsgenie\_alert\_hook.py](https://codecov.io/gh/apache/airflow/pull/4903/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL29wc2dlbmllX2FsZXJ0X2hvb2sucHk=)
 | `100% <100%> (ø)` | |
   | 
[airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/4903/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==)
 | `90.38% <100%> (+0.09%)` | :arrow_up: |
   | 
[...rflow/contrib/operators/opsgenie\_alert\_operator.py](https://codecov.io/gh/apache/airflow/pull/4903/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9vcHNnZW5pZV9hbGVydF9vcGVyYXRvci5weQ==)
 | `93.93% <93.93%> (ø)` | |
   | 
[airflow/utils/log/file\_processor\_handler.py](https://codecov.io/gh/apache/airflow/pull/4903/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV9wcm9jZXNzb3JfaGFuZGxlci5weQ==)
 | `87.25% <0%> (+1.14%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/4903?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/4903?src=pr=footer). 
Last update 
[dbed51e...778c919](https://codecov.io/gh/apache/airflow/pull/4903?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #4648: [AIRFLOW-3274] Add run_as_user and fs_group options for Kubernetes

codecov-io edited a comment on issue #4648: [AIRFLOW-3274] Add run_as_user and 
fs_group options for Kubernetes
URL: https://github.com/apache/airflow/pull/4648#issuecomment-460139330
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/4648?src=pr=h1) 
Report
   > Merging 
[#4648](https://codecov.io/gh/apache/airflow/pull/4648?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/dbed51e702bf8177800183d2c4f595073aa2339d?src=pr=desc)
 will **decrease** coverage by `<.01%`.
   > The diff coverage is `86.66%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/4648/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/4648?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4648  +/-   ##
   ==
   - Coverage   76.23%   76.23%   -0.01% 
   ==
 Files 466  466  
 Lines   3010130116  +15 
   ==
   + Hits2294922958   +9 
   - Misses   7152 7158   +6
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/4648?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[...example\_dags/example\_kubernetes\_executor\_config.py](https://codecov.io/gh/apache/airflow/pull/4648/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2t1YmVybmV0ZXNfZXhlY3V0b3JfY29uZmlnLnB5)
 | `0% <0%> (ø)` | :arrow_up: |
   | 
[.../kubernetes\_request\_factory/pod\_request\_factory.py](https://codecov.io/gh/apache/airflow/pull/4648/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2t1YmVybmV0ZXMva3ViZXJuZXRlc19yZXF1ZXN0X2ZhY3RvcnkvcG9kX3JlcXVlc3RfZmFjdG9yeS5weQ==)
 | `100% <100%> (ø)` | :arrow_up: |
   | 
[...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/4648/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==)
 | `98.63% <100%> (+0.03%)` | :arrow_up: |
   | 
[airflow/contrib/kubernetes/worker\_configuration.py](https://codecov.io/gh/apache/airflow/pull/4648/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2t1YmVybmV0ZXMvd29ya2VyX2NvbmZpZ3VyYXRpb24ucHk=)
 | `95.9% <100%> (+0.17%)` | :arrow_up: |
   | 
[airflow/contrib/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/4648/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4ZWN1dG9ycy9rdWJlcm5ldGVzX2V4ZWN1dG9yLnB5)
 | `63.38% <100%> (+0.2%)` | :arrow_up: |
   | 
[airflow/utils/log/file\_processor\_handler.py](https://codecov.io/gh/apache/airflow/pull/4648/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV9wcm9jZXNzb3JfaGFuZGxlci5weQ==)
 | `82.66% <0%> (-3.45%)` | :arrow_down: |
   | 
[airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4648/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=)
 | `92.95% <0%> (-0.05%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/4648?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/4648?src=pr=footer). 
Last update 
[dbed51e...91f94af](https://codecov.io/gh/apache/airflow/pull/4648?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-4251) Instrument DagRun schedule delay

2019-04-05 Thread Chao-Han Tsai (JIRA)

Chao-Han Tsai created AIRFLOW-4251:
--

 Summary: Instrument DagRun schedule delay
 Key: AIRFLOW-4251
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4251
 Project: Apache Airflow
  Issue Type: New Feature
Reporter: Chao-Han Tsai
Assignee: Chao-Han Tsai


Instrument DagRun schedule delay - time between expected DagRun start date and 
the actual DagRun start date.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] cmdoptesc commented on a change in pull request #5032: [AIRFLOW-4232] Add `none_skipped` trigger rule

cmdoptesc commented on a change in pull request #5032: [AIRFLOW-4232] Add 
`none_skipped` trigger rule
URL: https://github.com/apache/airflow/pull/5032#discussion_r272731994
 
 

 ##
 File path: tests/ti_deps/deps/test_trigger_rule_dep.py
 ##
 @@ -275,6 +276,72 @@ def test_all_done_tr_failure(self):
 self.assertEqual(len(dep_statuses), 1)
 self.assertFalse(dep_statuses[0].passed)
 
+def test_none_skipped_tr_success(self):
+"""
+None-skipped trigger rule success
+"""
+
+ti = self._get_task_instance(TriggerRule.NONE_SKIPPED,
+ upstream_task_ids=["FakeTaskID",
+"OtherFakeTaskID",
+"FailedFakeTaskID"])
+dep_statuses = tuple(TriggerRuleDep()._evaluate_trigger_rule(
+ti=ti,
+successes=2,
+skipped=0,
+failed=1,
+upstream_failed=0,
+done=3,
+flag_upstream_failed=False,
+session="Fake Session"))
+self.assertEqual(len(dep_statuses), 0)
+
+# with `flag_upstream_failed` set to True
+dep_statuses = tuple(TriggerRuleDep()._evaluate_trigger_rule(
+ti=ti,
+successes=0,
+skipped=0,
+failed=3,
+upstream_failed=0,
+done=3,
+flag_upstream_failed=True,
+session="Fake Session"))
+self.assertEqual(len(dep_statuses), 0)
+
+def test_none_skipped_tr_failure(self):
+"""
+None-skipped trigger rule failure
+"""
+ti = self._get_task_instance(TriggerRule.NONE_SKIPPED,
+ upstream_task_ids=["FakeTaskID",
+"SkippedTaskID"])
+
+with create_session() as session:
+dep_statuses = tuple(TriggerRuleDep()._evaluate_trigger_rule(
+ti=ti,
+successes=1,
+skipped=1,
+failed=0,
+upstream_failed=0,
+done=2,
+flag_upstream_failed=False,
+session=session))
 
 Review comment:
   I assume there was the `AttributeError: 'str' object has no attribute 
'merge'` because we were just passing a str instead of an actual session.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] cmdoptesc commented on a change in pull request #5032: [AIRFLOW-4232] Add `none_skipped` trigger rule

cmdoptesc commented on a change in pull request #5032: [AIRFLOW-4232] Add 
`none_skipped` trigger rule
URL: https://github.com/apache/airflow/pull/5032#discussion_r272731994
 
 

 ##
 File path: tests/ti_deps/deps/test_trigger_rule_dep.py
 ##
 @@ -275,6 +276,72 @@ def test_all_done_tr_failure(self):
 self.assertEqual(len(dep_statuses), 1)
 self.assertFalse(dep_statuses[0].passed)
 
+def test_none_skipped_tr_success(self):
+"""
+None-skipped trigger rule success
+"""
+
+ti = self._get_task_instance(TriggerRule.NONE_SKIPPED,
+ upstream_task_ids=["FakeTaskID",
+"OtherFakeTaskID",
+"FailedFakeTaskID"])
+dep_statuses = tuple(TriggerRuleDep()._evaluate_trigger_rule(
+ti=ti,
+successes=2,
+skipped=0,
+failed=1,
+upstream_failed=0,
+done=3,
+flag_upstream_failed=False,
+session="Fake Session"))
+self.assertEqual(len(dep_statuses), 0)
+
+# with `flag_upstream_failed` set to True
+dep_statuses = tuple(TriggerRuleDep()._evaluate_trigger_rule(
+ti=ti,
+successes=0,
+skipped=0,
+failed=3,
+upstream_failed=0,
+done=3,
+flag_upstream_failed=True,
+session="Fake Session"))
+self.assertEqual(len(dep_statuses), 0)
+
+def test_none_skipped_tr_failure(self):
+"""
+None-skipped trigger rule failure
+"""
+ti = self._get_task_instance(TriggerRule.NONE_SKIPPED,
+ upstream_task_ids=["FakeTaskID",
+"SkippedTaskID"])
+
+with create_session() as session:
+dep_statuses = tuple(TriggerRuleDep()._evaluate_trigger_rule(
+ti=ti,
+successes=1,
+skipped=1,
+failed=0,
+upstream_failed=0,
+done=2,
+flag_upstream_failed=False,
+session=session))
 
 Review comment:
   I assume there was the `AttributeError: 'str' object has no attribute 
'merge' because we were just passing a str instead of an actual session.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4251) Instrument DagRun schedule delay



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811216#comment-16811216
 ] 

ASF GitHub Bot commented on AIRFLOW-4251:
-

milton0825 commented on pull request #5050: [AIRFLOW-4251] Instrument DagRun 
schedule delay
URL: https://github.com/apache/airflow/pull/5050
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [X] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW4251
   
   ### Description
   
   - [X] Here are some details about my PR, including screenshots of any UI 
changes:
   
   Instrument DagRun schedule delay - time between expected DagRun start date 
and the actual DagRun start date.
   
   ### Tests
   
   - [X] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   Tests not required as we did not change any airflow logic
   
   ### Commits
   
   - [X] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [ ] Passes `flake8`
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Instrument DagRun schedule delay
> 
>
> Key: AIRFLOW-4251
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4251
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
>
> Instrument DagRun schedule delay - time between expected DagRun start date 
> and the actual DagRun start date.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] galuszkak commented on issue #4932: [AIRFLOW-3611] Simplified development workflow [Depends on multi-staging]

galuszkak commented on issue #4932: [AIRFLOW-3611] Simplified development 
workflow [Depends on multi-staging]
URL: https://github.com/apache/airflow/pull/4932#issuecomment-480406199
 
 
   @potiuk that's all from my testing. This is an amazing improvement. My first 
experience as a contributor, to figure out how to run locally tests was really 
tragic. This from contributor perspective is such a superior experience even if 
it's not working, it was really easy to figure it out what to do.
   
   Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] milton0825 opened a new pull request #5050: [AIRFLOW-4251] Instrument DagRun schedule delay

milton0825 opened a new pull request #5050: [AIRFLOW-4251] Instrument DagRun 
schedule delay
URL: https://github.com/apache/airflow/pull/5050
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [X] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW4251
   
   ### Description
   
   - [X] Here are some details about my PR, including screenshots of any UI 
changes:
   
   Instrument DagRun schedule delay - time between expected DagRun start date 
and the actual DagRun start date.
   
   ### Tests
   
   - [X] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   Tests not required as we did not change any airflow logic
   
   ### Commits
   
   - [X] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [ ] Passes `flake8`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] pgagnon commented on issue #4648: [AIRFLOW-3274] Add run_as_user and fs_group options for Kubernetes

pgagnon commented on issue #4648: [AIRFLOW-3274] Add run_as_user and fs_group 
options for Kubernetes
URL: https://github.com/apache/airflow/pull/4648#issuecomment-480404382
 
 
   @ashb Failed again but I think it's a flake. 
https://travis-ci.org/apache/airflow/builds/516345620


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nritholtz commented on issue #4903: [AIRFLOW-4069] Add Opsgenie Alert Hook and Operator

nritholtz commented on issue #4903: [AIRFLOW-4069] Add Opsgenie Alert Hook and 
Operator
URL: https://github.com/apache/airflow/pull/4903#issuecomment-480397936
 
 
   @ashb looks like unrelated test failures again on 2 of the jobs. Otherwise 
made all suggested changes.
   
   Thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on issue #4648: [AIRFLOW-3274] Add run_as_user and fs_group options for Kubernetes

2019-04-05 Thread ASF subversion and git services (JIRA)

ashb commented on issue #4648: [AIRFLOW-3274] Add run_as_user and fs_group 
options for Kubernetes
URL: https://github.com/apache/airflow/pull/4648#issuecomment-480395880
 
 
   Test failures:
   
   ```
   3) ERROR: test_make_pod_git_sync_ssh_without_known_hosts 
(tests.contrib.executors.test_kubernetes_executor.TestKubernetesWorkerConfiguration)
   --
  Traceback (most recent call last):
   tests/contrib/executors/test_kubernetes_executor.py line 421 in 
test_make_pod_git_sync_ssh_without_known_hosts
 self.assertEqual(65533, pod.security_context['fs_group'])
  KeyError: 'fs_group'
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4248) FileExistsError when creating log dir in file_processor_handler.py



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811186#comment-16811186
 ] 

ASF subversion and git services commented on AIRFLOW-4248:
--

Commit 45270cd3cfbec9d6f61794c4ddd647d6237d8603 in airflow's branch 
refs/heads/v1-10-stable from Kevin Pullin
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=45270cd ]

[AIRFLOW-4248] Fix 'FileExistsError' makedirs race in file_processor_handler 
(#5047)



> FileExistsError when creating log dir in file_processor_handler.py
> --
>
> Key: AIRFLOW-4248
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4248
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Kevin Pullin
>Assignee: Kevin Pullin
>Priority: Major
>
> Similar to AIRFLOW-2065, the `_init_file` method in `file_processor_handler` 
> does not handle the race condition of multiple processes creating the same 
> directory on a shared logging filesystem.
> If the error occurs in the scheduler process, tasks do not execute (at least 
> when using the Kubernetes executor). However the process remains alive and 
> requires a restart to resume running tasks.
>  
> Example log message:
>  
> {noformat}
> [2019-04-03 23:59:55,091] {{sqlalchemy.py:79}} WARNING - DB connection 
> invalidated. Reconnecting...
> Process DagFileProcessor41527-Process:
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.6/multiprocessing/process.py", line 258, in 
> _bootstrap
>     self.run()
>   File "/usr/local/lib/python3.6/multiprocessing/process.py", line 93, in run
>     self._target(*self._args, **self._kwargs)
>   File "/usr/local/lib/python3.6/site-packages/airflow/jobs.py", line 374, in 
> helper
>     set_context(log, file_path)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/utils/log/logging_mixin.py", 
> line 170, in set_context
>     handler.set_context(value)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/utils/log/file_processor_handler.py",
>  line 66, in set_context
>     local_loc = self._init_file(filename)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/utils/log/file_processor_handler.py",
>  line 141, in _init_file
>     os.makedirs(directory)
>   File "/usr/local/lib/python3.6/os.py", line 220, in makedirs
>     mkdir(name, mode)
> FileExistsError: [Errno 17] File exists: 
> '/usr/local/airflow/logs/scheduler/2019-04-04'{noformat}
>  
> The path `/usr/local/airflow/logs` is on a shared NFS volume.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] ashb commented on issue #5032: [AIRFLOW-4232] Add `none_skipped` trigger rule

2019-04-05 Thread ASF subversion and git services (JIRA)

ashb commented on issue #5032: [AIRFLOW-4232] Add `none_skipped` trigger rule
URL: https://github.com/apache/airflow/pull/5032#issuecomment-480394634
 
 
   Looks like you have a test failure:
   
   ```49) ERROR: test_none_skipped_tr_failure 
(tests.ti_deps.deps.test_trigger_rule_dep.TriggerRuleDepTest)
   --
  Traceback (most recent call last):
   tests/ti_deps/deps/test_trigger_rule_dep.py line 337 in 
test_none_skipped_tr_failure
 session="Fake Session"))
   airflow/ti_deps/deps/trigger_rule_dep.py line 162 in 
_evaluate_trigger_rule
 ti.set_state(State.SKIPPED, session)
   airflow/utils/db.py line 69 in wrapper
 return func(*args, **kwargs)
   airflow/models/__init__.py line 990 in set_state
 session.merge(self)
  AttributeError: 'str' object has no attribute 'merge'```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4248) FileExistsError when creating log dir in file_processor_handler.py



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811185#comment-16811185
 ] 

ASF subversion and git services commented on AIRFLOW-4248:
--

Commit 8195ea90fce953d605aa61b46bd7cd6032c5ee3e in airflow's branch 
refs/heads/master from Kevin Pullin
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=8195ea9 ]

[AIRFLOW-4248] Fix 'FileExistsError' makedirs race in file_processor_handler 
(#5047)



> FileExistsError when creating log dir in file_processor_handler.py
> --
>
> Key: AIRFLOW-4248
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4248
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Kevin Pullin
>Assignee: Kevin Pullin
>Priority: Major
>
> Similar to AIRFLOW-2065, the `_init_file` method in `file_processor_handler` 
> does not handle the race condition of multiple processes creating the same 
> directory on a shared logging filesystem.
> If the error occurs in the scheduler process, tasks do not execute (at least 
> when using the Kubernetes executor). However the process remains alive and 
> requires a restart to resume running tasks.
>  
> Example log message:
>  
> {noformat}
> [2019-04-03 23:59:55,091] {{sqlalchemy.py:79}} WARNING - DB connection 
> invalidated. Reconnecting...
> Process DagFileProcessor41527-Process:
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.6/multiprocessing/process.py", line 258, in 
> _bootstrap
>     self.run()
>   File "/usr/local/lib/python3.6/multiprocessing/process.py", line 93, in run
>     self._target(*self._args, **self._kwargs)
>   File "/usr/local/lib/python3.6/site-packages/airflow/jobs.py", line 374, in 
> helper
>     set_context(log, file_path)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/utils/log/logging_mixin.py", 
> line 170, in set_context
>     handler.set_context(value)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/utils/log/file_processor_handler.py",
>  line 66, in set_context
>     local_loc = self._init_file(filename)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/utils/log/file_processor_handler.py",
>  line 141, in _init_file
>     os.makedirs(directory)
>   File "/usr/local/lib/python3.6/os.py", line 220, in makedirs
>     mkdir(name, mode)
> FileExistsError: [Errno 17] File exists: 
> '/usr/local/airflow/logs/scheduler/2019-04-04'{noformat}
>  
> The path `/usr/local/airflow/logs` is on a shared NFS volume.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4248) FileExistsError when creating log dir in file_processor_handler.py



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811184#comment-16811184
 ] 

ASF GitHub Bot commented on AIRFLOW-4248:
-

ashb commented on pull request #5047: [AIRFLOW-4248] Fix 'FileExistsError' 
makedirs() race condition
URL: https://github.com/apache/airflow/pull/5047
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> FileExistsError when creating log dir in file_processor_handler.py
> --
>
> Key: AIRFLOW-4248
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4248
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Kevin Pullin
>Assignee: Kevin Pullin
>Priority: Major
>
> Similar to AIRFLOW-2065, the `_init_file` method in `file_processor_handler` 
> does not handle the race condition of multiple processes creating the same 
> directory on a shared logging filesystem.
> If the error occurs in the scheduler process, tasks do not execute (at least 
> when using the Kubernetes executor). However the process remains alive and 
> requires a restart to resume running tasks.
>  
> Example log message:
>  
> {noformat}
> [2019-04-03 23:59:55,091] {{sqlalchemy.py:79}} WARNING - DB connection 
> invalidated. Reconnecting...
> Process DagFileProcessor41527-Process:
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.6/multiprocessing/process.py", line 258, in 
> _bootstrap
>     self.run()
>   File "/usr/local/lib/python3.6/multiprocessing/process.py", line 93, in run
>     self._target(*self._args, **self._kwargs)
>   File "/usr/local/lib/python3.6/site-packages/airflow/jobs.py", line 374, in 
> helper
>     set_context(log, file_path)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/utils/log/logging_mixin.py", 
> line 170, in set_context
>     handler.set_context(value)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/utils/log/file_processor_handler.py",
>  line 66, in set_context
>     local_loc = self._init_file(filename)
>   File 
> "/usr/local/lib/python3.6/site-packages/airflow/utils/log/file_processor_handler.py",
>  line 141, in _init_file
>     os.makedirs(directory)
>   File "/usr/local/lib/python3.6/os.py", line 220, in makedirs
>     mkdir(name, mode)
> FileExistsError: [Errno 17] File exists: 
> '/usr/local/airflow/logs/scheduler/2019-04-04'{noformat}
>  
> The path `/usr/local/airflow/logs` is on a shared NFS volume.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] ashb merged pull request #5047: [AIRFLOW-4248] Fix 'FileExistsError' makedirs() race condition

ashb merged pull request #5047: [AIRFLOW-4248] Fix 'FileExistsError' makedirs() 
race condition
URL: https://github.com/apache/airflow/pull/5047
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #5047: [AIRFLOW-4248] Fix 'FileExistsError' makedirs() race condition

codecov-io commented on issue #5047: [AIRFLOW-4248] Fix 'FileExistsError' 
makedirs() race condition
URL: https://github.com/apache/airflow/pull/5047#issuecomment-480392687
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5047?src=pr=h1) 
Report
   > Merging 
[#5047](https://codecov.io/gh/apache/airflow/pull/5047?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/4991c34f47d4c00ff9591aeef3c2d2a74b62f1a9?src=pr=desc)
 will **decrease** coverage by `<.01%`.
   > The diff coverage is `28.57%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5047/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5047?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#5047  +/-   ##
   ==
   - Coverage   76.23%   76.22%   -0.01% 
   ==
 Files 466  466  
 Lines   3010130104   +3 
   ==
 Hits2294822948  
   - Misses   7153 7156   +3
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/5047?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/utils/log/file\_processor\_handler.py](https://codecov.io/gh/apache/airflow/pull/5047/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV9wcm9jZXNzb3JfaGFuZGxlci5weQ==)
 | `82.66% <28.57%> (-3.45%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5047?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5047?src=pr=footer). 
Last update 
[4991c34...edd76d6](https://codecov.io/gh/apache/airflow/pull/5047?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #4932: [AIRFLOW-3611] Simplified development workflow [Depends on multi-staging]

potiuk commented on issue #4932: [AIRFLOW-3611] Simplified development workflow 
[Depends on multi-staging]
URL: https://github.com/apache/airflow/pull/4932#issuecomment-480391941
 
 
   I understand @galuszkak that you are talking about tests that you run 
manually? Some more information about the freeeze ? Maybe we can switch to 
https://apache-airflow-slack.herokuapp.com/  and chat there? I am there as 
"Jarek Potiuk"


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3063) Support all docker options to create containers with DockerOperator



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811177#comment-16811177
 ] 

ASF GitHub Bot commented on AIRFLOW-3063:
-

MerouaneBen commented on pull request #5049: [AIRFLOW-3063] - Make 
DockerOperator support all docker arguments
URL: https://github.com/apache/airflow/pull/5049
 
 
   A flexible implementation of dockerOperator class that takes all the
   containers config arguments in json kwargs dynamically.
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-3063
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   The default docker compose operator is great solution to isolate and scale 
the execution of dag tasks, but when it comes to use it in real complex tasks, 
we quickly notice that we can not use all the arguments and parameters to 
customise the containers available in the official Docker python lib. In this 
PR, we provide a flexible approach that allows to provide all the possible 
arguments for create_container method in kwargs object, instead of having only 
some of them listed in the constructor of the class.
   
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   The original unitest of docker operator lib where updated to work with the 
modifications implemented in the class, and some new unit tests have been added 
as well.
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [x] Passes `flake8`
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Support all docker options to create containers with DockerOperator
> ---
>
> Key: AIRFLOW-3063
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3063
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: docker
>Affects Versions: 1.10.0
>Reporter: Victor
>Priority: Major
>
> The original need is to run a container in a predefined docker network, but I 
> suppose the operator should support setting any of the options supported by 
> the docker-py API.
> What is not totally clear is if it should be possible to define any of the 
> options that can be passed to the docker API in a templated fashion or not.
> In my case, a temporary network could for example have been created in a 
> previous operator and pushed to xcom.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] MerouaneBen opened a new pull request #5049: [AIRFLOW-3063] - Make DockerOperator support all docker arguments

MerouaneBen opened a new pull request #5049: [AIRFLOW-3063] - Make
DockerOperator support all docker arguments
URL: https://github.com/apache/airflow/pull/5049

A flexible implementation of dockerOperator class that takes all the
containers config arguments in json kwargs dynamically.

Make sure you have checked _all_ steps below.

### Jira

- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3063
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).

### Description

- [x] Here are some details about my PR, including screenshots of any UI
changes:
The default docker compose operator is great solution to isolate and scale
the execution of dag tasks, but when it comes to use it in real complex tasks,
we quickly notice that we can not use all the arguments and parameters to
customise the containers available in the official Docker python lib. In this
PR, we provide a flexible approach that allows to provide all the possible
arguments for create_container method in kwargs object, instead of having only
some of them listed in the constructor of the class.

### Tests

- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
The original unitest of docker operator lib where updated to work with the
modifications implemented in the class, and some new unit tests have been added
as well.

### Commits

- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"

### Documentation

- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release

### Code Quality

- [x] Passes `flake8`

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-3855) Incorrect type of query_params being required in BigQueryOperator



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor resolved AIRFLOW-3855.

Resolution: Duplicate

> Incorrect type of query_params being required in BigQueryOperator
> -
>
> Key: AIRFLOW-3855
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3855
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: api, gcp
>Affects Versions: 1.10.2
>Reporter: Ivan Ukhov
>Priority: Major
> Fix For: 1.10.3
>
>
> BigQueryOperator has been requiring query_params to be a dict for quite some 
> time:
> [https://github.com/apache/airflow/blob/1.10.2/airflow/contrib/hooks/bigquery_hook.py#L678]
> However, according to Google’s API is has to be a list:
> [https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query]
> See configuration.query.queryParameters:
> {quote}
> configuration.query.queryParameters[] list Query parameters for standard SQL 
> queries.
> {quote}
> Prior to 1.10.2, it was not actually checked, and passing a list was working 
> fine. Since 1.10.2, it no longer works, as a check is in place and raises an 
> exception.
> If one tries to pass a dict, as suggested by Airflow’s documentation, an 
> exception is raised on BigQuery’s side, as query parameters do not get 
> through, which is likely to be due to queryParameters (as a dict) being 
> translated into something that BigQuery does not understand.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AIRFLOW-3855) Incorrect type of query_params being required in BigQueryOperator



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor resolved AIRFLOW-3855.

Resolution: Duplicate

> Incorrect type of query_params being required in BigQueryOperator
> -
>
> Key: AIRFLOW-3855
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3855
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: api, gcp
>Affects Versions: 1.10.2
>Reporter: Ivan Ukhov
>Priority: Major
> Fix For: 1.10.3
>
>
> BigQueryOperator has been requiring query_params to be a dict for quite some 
> time:
> [https://github.com/apache/airflow/blob/1.10.2/airflow/contrib/hooks/bigquery_hook.py#L678]
> However, according to Google’s API is has to be a list:
> [https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query]
> See configuration.query.queryParameters:
> {quote}
> configuration.query.queryParameters[] list Query parameters for standard SQL 
> queries.
> {quote}
> Prior to 1.10.2, it was not actually checked, and passing a list was working 
> fine. Since 1.10.2, it no longer works, as a check is in place and raises an 
> exception.
> If one tries to pass a dict, as suggested by Airflow’s documentation, an 
> exception is raised on BigQuery’s side, as query parameters do not get 
> through, which is likely to be due to queryParameters (as a dict) being 
> translated into something that BigQuery does not understand.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (AIRFLOW-3855) Incorrect type of query_params being required in BigQueryOperator



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor reopened AIRFLOW-3855:


> Incorrect type of query_params being required in BigQueryOperator
> -
>
> Key: AIRFLOW-3855
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3855
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: api, gcp
>Affects Versions: 1.10.2
>Reporter: Ivan Ukhov
>Priority: Major
> Fix For: 1.10.3
>
>
> BigQueryOperator has been requiring query_params to be a dict for quite some 
> time:
> [https://github.com/apache/airflow/blob/1.10.2/airflow/contrib/hooks/bigquery_hook.py#L678]
> However, according to Google’s API is has to be a list:
> [https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query]
> See configuration.query.queryParameters:
> {quote}
> configuration.query.queryParameters[] list Query parameters for standard SQL 
> queries.
> {quote}
> Prior to 1.10.2, it was not actually checked, and passing a list was working 
> fine. Since 1.10.2, it no longer works, as a check is in place and raises an 
> exception.
> If one tries to pass a dict, as suggested by Airflow’s documentation, an 
> exception is raised on BigQuery’s side, as query parameters do not get 
> through, which is likely to be due to queryParameters (as a dict) being 
> translated into something that BigQuery does not understand.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (AIRFLOW-3855) Incorrect type of query_params being required in BigQueryOperator



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor reopened AIRFLOW-3855:


> Incorrect type of query_params being required in BigQueryOperator
> -
>
> Key: AIRFLOW-3855
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3855
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: api, gcp
>Affects Versions: 1.10.2
>Reporter: Ivan Ukhov
>Priority: Major
> Fix For: 1.10.3
>
>
> BigQueryOperator has been requiring query_params to be a dict for quite some 
> time:
> [https://github.com/apache/airflow/blob/1.10.2/airflow/contrib/hooks/bigquery_hook.py#L678]
> However, according to Google’s API is has to be a list:
> [https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query]
> See configuration.query.queryParameters:
> {quote}
> configuration.query.queryParameters[] list Query parameters for standard SQL 
> queries.
> {quote}
> Prior to 1.10.2, it was not actually checked, and passing a list was working 
> fine. Since 1.10.2, it no longer works, as a check is in place and raises an 
> exception.
> If one tries to pass a dict, as suggested by Airflow’s documentation, an 
> exception is raised on BigQuery’s side, as query parameters do not get 
> through, which is likely to be due to queryParameters (as a dict) being 
> translated into something that BigQuery does not understand.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-3855) Incorrect type of query_params being required in BigQueryOperator



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811157#comment-16811157
 ] 

Ash Berlin-Taylor commented on AIRFLOW-3855:


Changing resolution

> Incorrect type of query_params being required in BigQueryOperator
> -
>
> Key: AIRFLOW-3855
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3855
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: api, gcp
>Affects Versions: 1.10.2
>Reporter: Ivan Ukhov
>Priority: Major
> Fix For: 1.10.3
>
>
> BigQueryOperator has been requiring query_params to be a dict for quite some 
> time:
> [https://github.com/apache/airflow/blob/1.10.2/airflow/contrib/hooks/bigquery_hook.py#L678]
> However, according to Google’s API is has to be a list:
> [https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query]
> See configuration.query.queryParameters:
> {quote}
> configuration.query.queryParameters[] list Query parameters for standard SQL 
> queries.
> {quote}
> Prior to 1.10.2, it was not actually checked, and passing a list was working 
> fine. Since 1.10.2, it no longer works, as a check is in place and raises an 
> exception.
> If one tries to pass a dict, as suggested by Airflow’s documentation, an 
> exception is raised on BigQuery’s side, as query parameters do not get 
> through, which is likely to be due to queryParameters (as a dict) being 
> translated into something that BigQuery does not understand.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] nritholtz commented on a change in pull request #4903: [AIRFLOW-4069] Add Opsgenie Alert Hook and Operator

nritholtz commented on a change in pull request #4903: [AIRFLOW-4069] Add 
Opsgenie Alert Hook and Operator
URL: https://github.com/apache/airflow/pull/4903#discussion_r272703256
 
 

 ##
 File path: tests/contrib/operators/test_opsgenie_alert_operator.py
 ##
 @@ -0,0 +1,124 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+import unittest
+
+from airflow import DAG, configuration
+
+from airflow.contrib.operators.opsgenie_alert_operator import 
OpsgenieAlertOperator
+from airflow.utils import timezone
+
+DEFAULT_DATE = timezone.datetime(2017, 1, 1)
+
+
+class TestOpsgenieAlertOperator(unittest.TestCase):
+_config = {
+'message': 'An example alert message',
+'alias': 'Life is too short for no alias',
+'description': 'Every alert needs a description',
+'responders': [
+{'id': '4513b7ea-3b91-438f-b7e4-e3e54af9147c', 'type': 'team'},
+{'name': 'NOC', 'type': 'team'},
+{'id': 'bb4d9938-c3c2-455d-aaab-727aa701c0d8', 'type': 'user'},
+{'username': 'trin...@opsgenie.com', 'type': 'user'},
+{'id': 'aee8a0de-c80f-4515-a232-501c0bc9d715', 'type': 
'escalation'},
+{'name': 'Nightwatch Escalation', 'type': 'escalation'},
+{'id': '80564037-1984-4f38-b98e-8a1f662df552', 'type': 'schedule'},
+{'name': 'First Responders Schedule', 'type': 'schedule'}
+],
+'visibleTo': [
+{'id': '4513b7ea-3b91-438f-b7e4-e3e54af9147c', 'type': 'team'},
+{'name': 'rocket_team', 'type': 'team'},
+{'id': 'bb4d9938-c3c2-455d-aaab-727aa701c0d8', 'type': 'user'},
+{'username': 'trin...@opsgenie.com', 'type': 'user'}
+],
+'actions': ['Restart', 'AnExampleAction'],
+'tags': ['OverwriteQuietHours', 'Critical'],
+'details': {'key1': 'value1', 'key2': 'value2'},
+'entity': 'An example entity',
+'source': 'Airflow',
+'priority': 'P1',
+'user': 'Jesse',
+'note': 'Write this down'
+}
+
+expected_payload_dict = {
+'message': _config['message'],
+'alias': _config['alias'],
+'description': _config['description'],
+'responders': _config['responders'],
+'visibleTo': _config['visibleTo'],
+'actions': _config['actions'],
+'tags': _config['tags'],
+'details': _config['details'],
+'entity': _config['entity'],
+'source': _config['source'],
+'priority': _config['priority'],
+'user': _config['user'],
+'note': _config['note']
+}
+
+def setUp(self):
+configuration.load_test_config()
+args = {
+'owner': 'airflow',
+'start_date': DEFAULT_DATE
+}
+self.dag = DAG('test_dag_id', default_args=args)
+
+def test_build_opsgenie_payload(self):
+# Given / When
+operator = OpsgenieAlertOperator(
+task_id='opsgenie_alert_job',
+dag=self.dag,
+**self._config
+)
+
+payload = operator._build_opsgenie_payload()
+
+# Then
+self.assertEqual(self.expected_payload_dict, payload)
+
+def test_execute(self):
 
 Review comment:
   Fixed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nritholtz commented on a change in pull request #4903: [AIRFLOW-4069] Add Opsgenie Alert Hook and Operator