date:20190408

XD-DENG edited a comment on issue #5064: [AIRFLOW-4260] Fix sphinx deprecation 
warning for autodoc_default_flags
URL: https://github.com/apache/airflow/pull/5064#issuecomment-481094394
 
 
   Thanks @feluelle . I noticed that there is another doc error due to 
deprecation 
`/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/sphinx/events.py:76:
 RemovedInSphinx30Warning: viewcode_import was renamed to 
viewcode_follow_imported_members. Please update your configuration.` 
(https://travis-ci.org/apache/airflow/jobs/517586747)
   
   Would you like to fix it together here? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] XD-DENG commented on a change in pull request #5064: [AIRFLOW-4260] Fix sphinx deprecation warning for autodoc_default_flags

XD-DENG commented on a change in pull request #5064: [AIRFLOW-4260] Fix sphinx 
deprecation warning for autodoc_default_flags
URL: https://github.com/apache/airflow/pull/5064#discussion_r273318702
 
 

 ##
 File path: docs/conf.py
 ##
 @@ -122,7 +122,10 @@
 'docroles'
 ]
 
-autodoc_default_flags = ['show-inheritance', 'members']
+autodoc_default_options = {
+'show-inheritance': True,
+'members': True
+}
 
 viewcode_import = True
 
 Review comment:
   This line is what I was talking about.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] bryanyang0528 commented on issue #5067: [AIRFLOW-4265] Lineage backend did not work normally

2019-04-08 Thread ASF subversion and git services (JIRA)

bryanyang0528 commented on issue #5067: [AIRFLOW-4265] Lineage backend did not 
work normally
URL: https://github.com/apache/airflow/pull/5067#issuecomment-481094243
 
 
   @feng-tao  thx!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-4265) Lineage backend did not work normally

2019-04-08 Thread Tao Feng (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tao Feng resolved AIRFLOW-4265.
---
   Resolution: Fixed
Fix Version/s: 2.0.0

> Lineage backend did not work normally
> -
>
> Key: AIRFLOW-4265
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4265
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Bryan Yang
>Assignee: Bryan Yang
>Priority: Major
> Fix For: 2.0.0
>
>
> 1. Lineage backend comes from {{import_string}} function, and 
> {{import_string}} will try to seperate module_path and class_name by dot.
> {quote}def import_string(dotted_path):
>  """
>  Import a dotted module path and return the attribute/class designated by the
>  last name in the path. Raise ImportError if the import failed.
>  """
>  try:
>  module_path, class_name = dotted_path.rsplit('.', 1)
>  except ValueError:
>  raise ImportError("{} doesn't look like a module path".format(dotted_path))
> module = import_module(module_path)
> {quote}
> But in the current document: {{backend = airflow.lineage.backend.atlas}} 
> which is just a module path without class name \{{AtlasBackend}}.
> 2. Continuously, this function will return a module or class instead of an 
> instance. So that when {{apply_lineage}} want to call 
> {{backend.send_lineage}}, it will fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4265) Lineage backend did not work normally



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812986#comment-16812986
 ] 

ASF subversion and git services commented on AIRFLOW-4265:
--

Commit 6ddf05598a9eb6970b313733e11d22cdc5aa611d in airflow's branch 
refs/heads/master from Bryan Yang
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=6ddf055 ]

[AIRFLOW-4265] Lineage backend did not work normally (#5067)

* add debug log

* change SendMessage to staticmethod

* updated doc


> Lineage backend did not work normally
> -
>
> Key: AIRFLOW-4265
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4265
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Bryan Yang
>Assignee: Bryan Yang
>Priority: Major
>
> 1. Lineage backend comes from {{import_string}} function, and 
> {{import_string}} will try to seperate module_path and class_name by dot.
> {quote}def import_string(dotted_path):
>  """
>  Import a dotted module path and return the attribute/class designated by the
>  last name in the path. Raise ImportError if the import failed.
>  """
>  try:
>  module_path, class_name = dotted_path.rsplit('.', 1)
>  except ValueError:
>  raise ImportError("{} doesn't look like a module path".format(dotted_path))
> module = import_module(module_path)
> {quote}
> But in the current document: {{backend = airflow.lineage.backend.atlas}} 
> which is just a module path without class name \{{AtlasBackend}}.
> 2. Continuously, this function will return a module or class instead of an 
> instance. So that when {{apply_lineage}} want to call 
> {{backend.send_lineage}}, it will fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] feng-tao commented on issue #5067: [AIRFLOW-4265] Lineage backend did not work normally

feng-tao commented on issue #5067: [AIRFLOW-4265] Lineage backend did not work 
normally
URL: https://github.com/apache/airflow/pull/5067#issuecomment-481090404
 
 
   lgtm


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feng-tao merged pull request #5067: [AIRFLOW-4265] Lineage backend did not work normally

feng-tao merged pull request #5067: [AIRFLOW-4265] Lineage backend did not work 
normally
URL: https://github.com/apache/airflow/pull/5067
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-4261) To fix some minor issues in airflow/jobs.py

2019-04-08 Thread Tao Feng (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tao Feng resolved AIRFLOW-4261.
---
   Resolution: Fixed
Fix Version/s: 2.0.0

> To fix some minor issues in airflow/jobs.py
> ---
>
> Key: AIRFLOW-4261
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4261
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: scheduler
>Affects Versions: 1.10.2
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Minor
> Fix For: 2.0.0
>
>
> There are some minor issues in airflow/jobs.py. Clearing them can help 1. 
> make code cleaner; 2. introduce minor performance improvement.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] feng-tao commented on a change in pull request #5065: [AIRFLOW-4261] Minor refactoring on jobs.py

feng-tao commented on a change in pull request #5065: [AIRFLOW-4261] Minor 
refactoring on jobs.py
URL: https://github.com/apache/airflow/pull/5065#discussion_r273286654
 
 

 ##
 File path: airflow/jobs.py
 ##
 @@ -675,10 +674,11 @@ def manage_slas(self, dag, session=None):
 qry = (
 session
 .query(TI)
-.filter(TI.state != State.SUCCESS)
-.filter(TI.execution_date.in_(sla_dates))
-.filter(TI.dag_id == dag.dag_id)
-.all()
+.filter(
+TI.state != State.SUCCESS,
+TI.execution_date.in_(sla_dates),
+TI.dag_id == dag.dag_id
+).all()
 
 Review comment:
   It is fine, we could keep those as it doesn’t harm like you said. Thanks for 
the info.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feng-tao merged pull request #5065: [AIRFLOW-4261] Minor refactoring on jobs.py

2019-04-08 Thread ASF subversion and git services (JIRA)

feng-tao merged pull request #5065: [AIRFLOW-4261] Minor refactoring on jobs.py
URL: https://github.com/apache/airflow/pull/5065
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4261) To fix some minor issues in airflow/jobs.py



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812960#comment-16812960
 ] 

ASF subversion and git services commented on AIRFLOW-4261:
--

Commit 851b0d93dad78141a2822b86eb01174afc2fc068 in airflow's branch 
refs/heads/master from Xiaodong
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=851b0d9 ]

[AIRFLOW-4261] Minor refactoring on jobs.py (#5065)



> To fix some minor issues in airflow/jobs.py
> ---
>
> Key: AIRFLOW-4261
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4261
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: scheduler
>Affects Versions: 1.10.2
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Minor
>
> There are some minor issues in airflow/jobs.py. Clearing them can help 1. 
> make code cleaner; 2. introduce minor performance improvement.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4261) To fix some minor issues in airflow/jobs.py



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812959#comment-16812959
 ] 

ASF GitHub Bot commented on AIRFLOW-4261:
-

feng-tao commented on pull request #5065: [AIRFLOW-4261] Minor refactoring on 
jobs.py
URL: https://github.com/apache/airflow/pull/5065
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> To fix some minor issues in airflow/jobs.py
> ---
>
> Key: AIRFLOW-4261
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4261
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: scheduler
>Affects Versions: 1.10.2
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Minor
>
> There are some minor issues in airflow/jobs.py. Clearing them can help 1. 
> make code cleaner; 2. introduce minor performance improvement.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] bryanyang0528 commented on issue #5067: [AIRFLOW-4265] Lineage backend did not work normally

bryanyang0528 commented on issue #5067: [AIRFLOW-4265] Lineage backend did not 
work normally
URL: https://github.com/apache/airflow/pull/5067#issuecomment-481083240
 
 
   import_string will fail if backend = `airflow.lineage.backend.atlas`
   
   ```
   >>> from airflow.utils.module_loading import import_string
   /airflow/airflow/configuration.py:557: DeprecationWarning: Specifying 
airflow_home in the config file is deprecated. As you have left it at the 
default value you should remove the setting from your airflow.cfg and suffer no 
change in behaviour.
 category=DeprecationWarning,
   >>> backend = import_string('airflow.lineage.backend.atlas')
   Traceback (most recent call last):
 File "/airflow/airflow/utils/module_loading.py", line 36, in import_string
   return getattr(module, class_name)
   AttributeError: module 'airflow.lineage.backend' has no attribute 'atlas'
   
   During handling of the above exception, another exception occurred:
   
   Traceback (most recent call last):
 File "", line 1, in 
 File "/airflow/airflow/utils/module_loading.py", line 39, in import_string
   module_path, class_name)
   ImportError: Module "airflow.lineage.backend" does not define a "atlas" 
attribute/class
   >>>
   ```
   
   if backend = `airflow.lineage.backend.atlas.AtlasBackend`. it passed 
import_string, but backend is a class, can not call send_message directly.
   ```
   >>> backend = import_string('airflow.lineage.backend.atlas.AtlasBackend')
   >>> backend
   
   >>> backend.send_lineage(operator='', inlets='', outlets='', context='')
   Traceback (most recent call last):
 File "", line 1, in 
   TypeError: send_lineage() missing 1 required positional argument: 'self'
   >>>
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] bryanyang0528 edited a comment on issue #5067: [AIRFLOW-4265] Lineage backend did not work normally

bryanyang0528 edited a comment on issue #5067: [AIRFLOW-4265] Lineage backend 
did not work normally
URL: https://github.com/apache/airflow/pull/5067#issuecomment-481083240
 
 
   import_string will fail if backend = `airflow.lineage.backend.atlas`
   
   ```
   >>> from airflow.utils.module_loading import import_string
   /airflow/airflow/configuration.py:557: DeprecationWarning: Specifying 
airflow_home in the config file is deprecated. As you have left it at the 
default value you should remove the setting from your airflow.cfg and suffer no 
change in behaviour.
 category=DeprecationWarning,
   >>> backend = import_string('airflow.lineage.backend.atlas')
   Traceback (most recent call last):
 File "/airflow/airflow/utils/module_loading.py", line 36, in import_string
   return getattr(module, class_name)
   AttributeError: module 'airflow.lineage.backend' has no attribute 'atlas'
   
   During handling of the above exception, another exception occurred:
   
   Traceback (most recent call last):
 File "", line 1, in 
 File "/airflow/airflow/utils/module_loading.py", line 39, in import_string
   module_path, class_name)
   ImportError: Module "airflow.lineage.backend" does not define a "atlas" 
attribute/class
   >>>
   ```
   
   if backend = `airflow.lineage.backend.atlas.AtlasBackend`, it passed 
import_string, but backend is a class, can not call send_message directly.
   ```
   >>> backend = import_string('airflow.lineage.backend.atlas.AtlasBackend')
   >>> backend
   
   >>> backend.send_lineage(operator='', inlets='', outlets='', context='')
   Traceback (most recent call last):
 File "", line 1, in 
   TypeError: send_lineage() missing 1 required positional argument: 'self'
   >>>
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4265) Lineage backend did not work normally



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812958#comment-16812958
 ] 

ASF GitHub Bot commented on AIRFLOW-4265:
-

bryanyang0528 commented on pull request #5067: [AIRFLOW-4265] Lineage backend 
did not work normally
URL: https://github.com/apache/airflow/pull/5067
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-4265
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [x] Passes `flake8`
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Lineage backend did not work normally
> -
>
> Key: AIRFLOW-4265
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4265
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Bryan Yang
>Assignee: Bryan Yang
>Priority: Major
>
> 1. Lineage backend comes from {{import_string}} function, and 
> {{import_string}} will try to seperate module_path and class_name by dot.
> {quote}def import_string(dotted_path):
>  """
>  Import a dotted module path and return the attribute/class designated by the
>  last name in the path. Raise ImportError if the import failed.
>  """
>  try:
>  module_path, class_name = dotted_path.rsplit('.', 1)
>  except ValueError:
>  raise ImportError("{} doesn't look like a module path".format(dotted_path))
> module = import_module(module_path)
> {quote}
> But in the current document: {{backend = airflow.lineage.backend.atlas}} 
> which is just a module path without class name \{{AtlasBackend}}.
> 2. Continuously, this function will return a module or class instead of an 
> instance. So that when {{apply_lineage}} want to call 
> {{backend.send_lineage}}, it will fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] bryanyang0528 opened a new pull request #5067: [AIRFLOW-4265] Lineage backend did not work normally

bryanyang0528 opened a new pull request #5067: [AIRFLOW-4265] Lineage backend 
did not work normally
URL: https://github.com/apache/airflow/pull/5067
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-4265
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [x] Passes `flake8`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] XD-DENG merged pull request #5063: [AIRFLOW-XXX] Ignore python files under node_modules when building docs

XD-DENG merged pull request #5063: [AIRFLOW-XXX] Ignore python files under 
node_modules when building docs
URL: https://github.com/apache/airflow/pull/5063
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-4265) Lineage backend did not work normally

2019-04-08 Thread Bryan Yang (JIRA)

Bryan Yang created AIRFLOW-4265:
---

 Summary: Lineage backend did not work normally
 Key: AIRFLOW-4265
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4265
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Bryan Yang
Assignee: Bryan Yang


1. Lineage backend comes from {{import_string}} function, and {{import_string}} 
will try to seperate module_path and class_name by dot.
{quote}def import_string(dotted_path):
 """
 Import a dotted module path and return the attribute/class designated by the
 last name in the path. Raise ImportError if the import failed.
 """
 try:
 module_path, class_name = dotted_path.rsplit('.', 1)
 except ValueError:
 raise ImportError("{} doesn't look like a module path".format(dotted_path))

module = import_module(module_path)
{quote}
But in the current document: {{backend = airflow.lineage.backend.atlas}} which 
is just a module path without class name \{{AtlasBackend}}.

2. Continuously, this function will return a module or class instead of an 
instance. So that when {{apply_lineage}} want to call {{backend.send_lineage}}, 
it will fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] feng-tao commented on a change in pull request #5065: [AIRFLOW-4261] Minor refactoring on jobs.py

feng-tao commented on a change in pull request #5065: [AIRFLOW-4261] Minor 
refactoring on jobs.py
URL: https://github.com/apache/airflow/pull/5065#discussion_r273286650
 
 

 ##
 File path: airflow/jobs.py
 ##
 @@ -675,10 +674,11 @@ def manage_slas(self, dag, session=None):
 qry = (
 session
 .query(TI)
-.filter(TI.state != State.SUCCESS)
-.filter(TI.execution_date.in_(sla_dates))
-.filter(TI.dag_id == dag.dag_id)
-.all()
+.filter(
+TI.state != State.SUCCESS,
+TI.execution_date.in_(sla_dates),
+TI.dag_id == dag.dag_id
+).all()
 
 Review comment:
   It is fine, we could keep those as it doesn’t harm like you said. Thanks for 
the info.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feng-tao commented on a change in pull request #5065: [AIRFLOW-4261] Minor refactoring on jobs.py

feng-tao commented on a change in pull request #5065: [AIRFLOW-4261] Minor 
refactoring on jobs.py
URL: https://github.com/apache/airflow/pull/5065#discussion_r273286654
 
 

 ##
 File path: airflow/jobs.py
 ##
 @@ -675,10 +674,11 @@ def manage_slas(self, dag, session=None):
 qry = (
 session
 .query(TI)
-.filter(TI.state != State.SUCCESS)
-.filter(TI.execution_date.in_(sla_dates))
-.filter(TI.dag_id == dag.dag_id)
-.all()
+.filter(
+TI.state != State.SUCCESS,
+TI.execution_date.in_(sla_dates),
+TI.dag_id == dag.dag_id
+).all()
 
 Review comment:
   It is fine, we could keep those as it doesn’t harm like you said. Thanks for 
the info.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] XD-DENG commented on a change in pull request #5065: [AIRFLOW-4261] Minor refactoring on jobs.py

XD-DENG commented on a change in pull request #5065: [AIRFLOW-4261] Minor 
refactoring on jobs.py
URL: https://github.com/apache/airflow/pull/5065#discussion_r273283711
 
 

 ##
 File path: airflow/jobs.py
 ##
 @@ -675,10 +674,11 @@ def manage_slas(self, dag, session=None):
 qry = (
 session
 .query(TI)
-.filter(TI.state != State.SUCCESS)
-.filter(TI.execution_date.in_(sla_dates))
-.filter(TI.dag_id == dag.dag_id)
-.all()
+.filter(
+TI.state != State.SUCCESS,
+TI.execution_date.in_(sla_dates),
+TI.dag_id == dag.dag_id
+).all()
 
 Review comment:
   ### SqlAlchemy
   
   `Multiple criteria may be specified as comma separated; the effect is that 
they will be joined together using the and_() function`
   
   Reference: 
https://docs.sqlalchemy.org/en/latest/orm/query.html#sqlalchemy.orm.query.Query.filter
   
   ### SQL Statements Under the Hood
   I have checked the sql statements below:
   ```python
   from airflow.settings import Session
   from airflow import models
   s = Session()
   DM = models.DagModel
   
   print(s.query(DM).filter(DM.is_active).filter(DM.dag_id == 
'tutorial').statement)
   print(s.query(DM).filter(DM.is_active, DM.dag_id == 'tutorial').statement)
   ```
   
   Result:
   ```
   SELECT dag.dag_id, dag.is_paused, dag.is_subdag, dag.is_active, 
dag.last_scheduler_run, dag.last_pickled, dag.last_expired, dag.scheduler_lock, 
dag.pickle_id, dag.fileloc, dag.owners, dag.description, dag.default_view, 
dag.schedule_interval
   FROM dag
   WHERE dag.is_active AND dag.dag_id = :dag_id_1
   
   SELECT dag.dag_id, dag.is_paused, dag.is_subdag, dag.is_active, 
dag.last_scheduler_run, dag.last_pickled, dag.last_expired, dag.scheduler_lock, 
dag.pickle_id, dag.fileloc, dag.owners, dag.description, dag.default_view, 
dag.schedule_interval
   FROM dag
   WHERE dag.is_active AND dag.dag_id = :dag_id_1
   ```
   
   So there is NO fundamental differences between them. But there is minor 
difference in terms performance (please refer to 
https://github.com/apache/airflow/pull/4433), and I suspect it comes from the 
compiling process from python code to sql statement (not sure).
   
   ### Suggestion
   This change doesn't break or harm anything, even though the improvement it 
brings is very minor.
   
   @feng-tao Please let me know if you don't think it's necessary to make these 
two `sqlalchemy filter`-related changes. I'm ok to remove them and only keep 
another two changes in this PR.
   
   Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] pgagnon commented on issue #4636: [AIRFLOW-3737] Kubernetes executor cannot handle long dag/task names

pgagnon commented on issue #4636: [AIRFLOW-3737] Kubernetes executor cannot 
handle long dag/task names
URL: https://github.com/apache/airflow/pull/4636#issuecomment-481048844
 
 
   Thanks for your work @PaulW and @darrenhaken !


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #5054: [AIRFLOW-4255] Replace Discovery based api with client based for GCS

kaxil commented on a change in pull request #5054: [AIRFLOW-4255] Replace 
Discovery based api with client based for GCS
URL: https://github.com/apache/airflow/pull/5054#discussion_r273269585
 
 

 ##
 File path: airflow/contrib/hooks/gcs_hook.py
 ##
 @@ -193,16 +187,7 @@ def upload(self, bucket, object, filename,
 :type mime_type: str
 :param gzip: Option to compress file for upload
 :type gzip: bool
-:param multipart: If True, the upload will be split into multiple HTTP 
requests. The
 
 Review comment:
   @Fokko Added a comment on `Multipart`  in Updating.md.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-4264) Would it be possible to make DummyOperator a fairly distinct color then other types of operators?

2019-04-08 Thread joyce chan (JIRA)

joyce chan created AIRFLOW-4264:
---

 Summary: Would it be possible to make DummyOperator a fairly 
distinct color then other types of operators?
 Key: AIRFLOW-4264
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4264
 Project: Apache Airflow
  Issue Type: Improvement
  Components: ui
Reporter: joyce chan


Sometimes, it's hard to tell at a quick glance on the graph view, unless I've 
named a task of a dummy operator with a certain name, that it's a task of type 
DummyOperator, but if it uses another color that would be very helpful.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AIRFLOW-4262) SlackWebhookOperator is not working properly

2019-04-08 Thread Tao Feng (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tao Feng resolved AIRFLOW-4262.
---
   Resolution: Fixed
Fix Version/s: 2.0.0

> SlackWebhookOperator is not working properly
> 
>
> Key: AIRFLOW-4262
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4262
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Junda Yang
>Assignee: Junda Yang
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] kaxil commented on a change in pull request #5054: [AIRFLOW-4255] Replace Discovery based api with client based for GCS

kaxil commented on a change in pull request #5054: [AIRFLOW-4255] Replace 
Discovery based api with client based for GCS
URL: https://github.com/apache/airflow/pull/5054#discussion_r273253790
 
 

 ##
 File path: airflow/contrib/hooks/gcs_hook.py
 ##
 @@ -253,8 +253,8 @@ def is_updated_after(self, bucket, object, ts):
 
 if blob_update_time > ts:
 return True
-else:
-return False
+
+return False
 
 Review comment:
   Can you approve this PR if you are ok with it :) ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4262) SlackWebhookOperator is not working properly

2019-04-08 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812819#comment-16812819
 ] 

ASF GitHub Bot commented on AIRFLOW-4262:
-

feng-tao commented on pull request #5066: [AIRFLOW-4262] Ensure 
SlackWebhookHook can take either http_conn_id or webhook_token, or both
URL: https://github.com/apache/airflow/pull/5066
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> SlackWebhookOperator is not working properly
> 
>
> Key: AIRFLOW-4262
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4262
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Junda Yang
>Assignee: Junda Yang
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4262) SlackWebhookOperator is not working properly



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812820#comment-16812820
 ] 

ASF subversion and git services commented on AIRFLOW-4262:
--

Commit 7ac82196b2a2eac43feb3060cffb4e6092a3afb7 in airflow's branch 
refs/heads/master from Junda Yang
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=7ac8219 ]

[AIRFLOW-4262] Ensure SlackWebhookHook can take either http_conn_id or 
webhook_token, or both (#5066)

* fix SlackWebhookHook

* update docstring


> SlackWebhookOperator is not working properly
> 
>
> Key: AIRFLOW-4262
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4262
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Junda Yang
>Assignee: Junda Yang
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] feng-tao merged pull request #5066: [AIRFLOW-4262] Ensure SlackWebhookHook can take either http_conn_id or webhook_token, or both

feng-tao merged pull request #5066: [AIRFLOW-4262] Ensure SlackWebhookHook can 
take either http_conn_id or webhook_token, or both
URL: https://github.com/apache/airflow/pull/5066
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feng-tao commented on a change in pull request #5066: [AIRFLOW-4262] Ensure SlackWebhookHook can take either http_conn_id or webhook_token, or both

feng-tao commented on a change in pull request #5066: [AIRFLOW-4262] Ensure 
SlackWebhookHook can take either http_conn_id or webhook_token, or both
URL: https://github.com/apache/airflow/pull/5066#discussion_r273251207
 
 

 ##
 File path: airflow/hooks/http_hook.py
 ##
 @@ -57,25 +57,26 @@ def get_conn(self, headers=None):
 :param headers: additional headers to be passed through as a dictionary
 :type headers: dict
 """
-conn = self.get_connection(self.http_conn_id)
 session = requests.Session()
-
-if "://" in conn.host:
-self.base_url = conn.host
-else:
-# schema defaults to HTTP
-schema = conn.schema if conn.schema else "http"
-self.base_url = schema + "://" + conn.host
-
-if conn.port:
-self.base_url = self.base_url + ":" + str(conn.port)
-if conn.login:
-session.auth = (conn.login, conn.password)
-if conn.extra:
-try:
-session.headers.update(conn.extra_dejson)
-except TypeError:
-self.log.warn('Connection to %s has invalid extra field.', 
conn.host)
+if self.http_conn_id:
+conn = self.get_connection(self.http_conn_id)
 
 Review comment:
   make sense. Thanks for the info.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] youngyjd commented on a change in pull request #5066: [AIRFLOW-4262] Ensure SlackWebhookHook can take either http_conn_id or webhook_token, or both

youngyjd commented on a change in pull request #5066: [AIRFLOW-4262] Ensure 
SlackWebhookHook can take either http_conn_id or webhook_token, or both
URL: https://github.com/apache/airflow/pull/5066#discussion_r273250884
 
 

 ##
 File path: airflow/hooks/http_hook.py
 ##
 @@ -57,25 +57,26 @@ def get_conn(self, headers=None):
 :param headers: additional headers to be passed through as a dictionary
 :type headers: dict
 """
-conn = self.get_connection(self.http_conn_id)
 session = requests.Session()
-
-if "://" in conn.host:
-self.base_url = conn.host
-else:
-# schema defaults to HTTP
-schema = conn.schema if conn.schema else "http"
-self.base_url = schema + "://" + conn.host
-
-if conn.port:
-self.base_url = self.base_url + ":" + str(conn.port)
-if conn.login:
-session.auth = (conn.login, conn.password)
-if conn.extra:
-try:
-session.headers.update(conn.extra_dejson)
-except TypeError:
-self.log.warn('Connection to %s has invalid extra field.', 
conn.host)
+if self.http_conn_id:
+conn = self.get_connection(self.http_conn_id)
 
 Review comment:
   If http_conn_id is provided but does not exist in Connection table, it will 
error out here: 
https://github.com/apache/airflow/blob/ab58eb67f63fa53b68bf50382f5eaf9303e3bdeb/airflow/hooks/base_hook.py#L58
   
   If user does not specify http_conn_id, `http_default` will be used 
https://github.com/apache/airflow/blob/ab58eb67f63fa53b68bf50382f5eaf9303e3bdeb/airflow/hooks/http_hook.py#L44.
 If user specifies `http_conn_id = None`, then `endpoint` must be the full path 
to make request work 
https://github.com/apache/airflow/blob/cc06fefce6b25ade0b04cc4ee20e51dd5b6d418b/airflow/hooks/http_hook.py#L84.
 
   
   which means, either we provide full url path in `http_conn_id` or 
`endpoint`, or use `http_conn_id` as base_url and `endpoint` as relative path 
to make HttpHook work.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feng-tao commented on a change in pull request #5066: [AIRFLOW-4262] Ensure SlackWebhookHook can take either http_conn_id or webhook_token, or both

feng-tao commented on a change in pull request #5066: [AIRFLOW-4262] Ensure 
SlackWebhookHook can take either http_conn_id or webhook_token, or both
URL: https://github.com/apache/airflow/pull/5066#discussion_r273248061
 
 

 ##
 File path: airflow/hooks/http_hook.py
 ##
 @@ -57,25 +57,26 @@ def get_conn(self, headers=None):
 :param headers: additional headers to be passed through as a dictionary
 :type headers: dict
 """
-conn = self.get_connection(self.http_conn_id)
 session = requests.Session()
-
-if "://" in conn.host:
-self.base_url = conn.host
-else:
-# schema defaults to HTTP
-schema = conn.schema if conn.schema else "http"
-self.base_url = schema + "://" + conn.host
-
-if conn.port:
-self.base_url = self.base_url + ":" + str(conn.port)
-if conn.login:
-session.auth = (conn.login, conn.password)
-if conn.extra:
-try:
-session.headers.update(conn.extra_dejson)
-except TypeError:
-self.log.warn('Connection to %s has invalid extra field.', 
conn.host)
+if self.http_conn_id:
+conn = self.get_connection(self.http_conn_id)
 
 Review comment:
   what happens if user provides http_conn_id while it is undefined in the 
connection table? Will it throw exception?
   
   Another question would be whether we should raise the exception if user use 
the hook without providing the connection id?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ArgentFalcon commented on issue #4957: [AIRFLOW-161] New redirect route and extra links

ArgentFalcon commented on issue #4957: [AIRFLOW-161] New redirect route and 
extra links
URL: https://github.com/apache/airflow/pull/4957#issuecomment-481008827
 
 
   Yeah, go ahead


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-4263) SSL error bad handshake while calling databrick job from airflow Dag

2019-04-08 Thread danish shadab (JIRA)

danish shadab created AIRFLOW-4263:
--

 Summary: SSL error bad handshake while calling databrick job from 
airflow Dag
 Key: AIRFLOW-4263
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4263
 Project: Apache Airflow
  Issue Type: Bug
  Components: DAG
Affects Versions: 1.10.2
Reporter: danish shadab


I am running an airflow container in which My airflow dags fails to connect to 
Databrick job with the error log below.

failed with reason: 
HTTPSConnectionPool(host='Mycompany-dev.cloud.databricks.com', port=443): Max 
retries exceeded with url: /api/2.0/jobs/runs/submit (Caused by 
SSLError(SSLError("bad handshake: Error([('SSL routines', 
'tls_process_server_certificate', 'certificate verify failed')],)",),))

more Information: Initially to install docker or java it was giving same error 
in which i rewrite the pip install code as below, however i am not sure how to 
include it while connecting to a server from airflow UI.

pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org

Please suggest



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] BasPH commented on issue #5056: [AIRFLOW-4259] Move remaining models out of models.py

BasPH commented on issue #5056: [AIRFLOW-4259] Move remaining models out of 
models.py
URL: https://github.com/apache/airflow/pull/5056#issuecomment-481004203
 
 
   @ashb all green now, ready for review :) 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feng-tao commented on issue #5066: [AIRFLOW-4262] fix SlackWebhookHook

feng-tao commented on issue #5066: [AIRFLOW-4262] fix SlackWebhookHook
URL: https://github.com/apache/airflow/pull/5066#issuecomment-480998357
 
 
   the commit message could be more descriptive, e.g like ```Ensure that 
SlackWebhookHook can take either http_conn_id or webhook_token, or both```.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4262) SlackWebhookOperator is not working properly



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812750#comment-16812750
 ] 

ASF GitHub Bot commented on AIRFLOW-4262:
-

youngyjd commented on pull request #5066: [AIRFLOW-4262] fix SlackWebhookHook
URL: https://github.com/apache/airflow/pull/5066
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [X] My PR addresses the following 
[AIRFLOW-4262](https://issues.apache.org/jira/browse/AIRFLOW-4262) issues and 
references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-4262
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [X] Here are some details about my PR, including screenshots of any UI 
changes:
   This PR is to fix SlackWebhookHook malfunction. It ensures that 
SlackWebhookHook can take either `http_conn_id` or `webhook_token`, or both, to 
POST messages into a slack channel, as said in the 
[docstring](https://github.com/apache/airflow/blob/4d34632fafe2f0bb8fcd67bb0964315eeb5c4fa9/airflow/contrib/hooks/slack_webhook_hook.py#L28-L30)
   
   ### Tests
   
   - [X] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
 1. My PR adds test to ensure correct url and http method is hit by 
SlackWebhookHook if only `http_conn_id` is supplied.
 2. My PR adds test to ensure correct url and http method is hit by 
SlackWebhookHook if only `webhook_token` is supplied.
 3. My PR adds test to ensure correct url and http method is hit by 
SlackWebhookHook if both `http_conn_id` and `webhook_token` is supplied.
   
   ### Commits
   
   - [X] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [X] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [X] Passes `flake8`
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> SlackWebhookOperator is not working properly
> 
>
> Key: AIRFLOW-4262
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4262
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Junda Yang
>Assignee: Junda Yang
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] youngyjd opened a new pull request #5066: [AIRFLOW-4262] fix SlackWebhookHook

youngyjd opened a new pull request #5066: [AIRFLOW-4262] fix SlackWebhookHook
URL: https://github.com/apache/airflow/pull/5066

Make sure you have checked _all_ steps below.

### Jira

- [X] My PR addresses the following
[AIRFLOW-4262](https://issues.apache.org/jira/browse/AIRFLOW-4262) issues and
references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-4262
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).

### Description

- [X] Here are some details about my PR, including screenshots of any UI
changes:
This PR is to fix SlackWebhookHook malfunction. It ensures that
SlackWebhookHook can take either `http_conn_id` or `webhook_token`, or both, to
POST messages into a slack channel, as said in the
[docstring](https://github.com/apache/airflow/blob/4d34632fafe2f0bb8fcd67bb0964315eeb5c4fa9/airflow/contrib/hooks/slack_webhook_hook.py#L28-L30)

### Tests

- [X] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
1. My PR adds test to ensure correct url and http method is hit by
SlackWebhookHook if only `http_conn_id` is supplied.
2. My PR adds test to ensure correct url and http method is hit by
SlackWebhookHook if only `webhook_token` is supplied.
3. My PR adds test to ensure correct url and http method is hit by
SlackWebhookHook if both `http_conn_id` and `webhook_token` is supplied.

### Commits

- [X] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"

### Documentation

- [X] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release

### Code Quality

- [X] Passes `flake8`

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

With regards,
Apache Git Services

[GitHub] [airflow] feng-tao commented on a change in pull request #5065: [AIRFLOW-4261] Minor refactoring on jobs.py

feng-tao commented on a change in pull request #5065: [AIRFLOW-4261] Minor 
refactoring on jobs.py
URL: https://github.com/apache/airflow/pull/5065#discussion_r273185923
 
 

 ##
 File path: airflow/jobs.py
 ##
 @@ -675,10 +674,11 @@ def manage_slas(self, dag, session=None):
 qry = (
 session
 .query(TI)
-.filter(TI.state != State.SUCCESS)
-.filter(TI.execution_date.in_(sla_dates))
-.filter(TI.dag_id == dag.dag_id)
-.all()
+.filter(
+TI.state != State.SUCCESS,
+TI.execution_date.in_(sla_dates),
+TI.dag_id == dag.dag_id
+).all()
 
 Review comment:
   do we know the actual sql difference between these two approach?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feluelle commented on issue #5065: [AIRFLOW-4261] Minor refactoring on jobs.py

feluelle commented on issue #5065: [AIRFLOW-4261] Minor refactoring on jobs.py
URL: https://github.com/apache/airflow/pull/5065#issuecomment-480948845
 
 
   LGTM overall  


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-4262) SlackWebhookOperator is not working properly

2019-04-08 Thread Junda Yang (JIRA)

Junda Yang created AIRFLOW-4262:
---

 Summary: SlackWebhookOperator is not working properly
 Key: AIRFLOW-4262
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4262
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Junda Yang
Assignee: Junda Yang






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] feng-tao edited a comment on issue #5010: [AIRFLOW-4192] Remove obsolete/derived task context variables

feng-tao edited a comment on issue #5010: [AIRFLOW-4192] Remove 
obsolete/derived task context variables
URL: https://github.com/apache/airflow/pull/5010#issuecomment-480515401
 
 
   @BasPH , I understand that we could compute the removed macro with some 
expressions. But I assume someone created those for convenience access. I am ok 
from my side, but could you send an email to the mailing list get some 
feedbacks from user as macros are used in users' DAG? Once there are no concern 
from the mailing list,  we could commit the change.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] BasPH commented on issue #5056: [WIP][AIRFLOW-4259] Move remaining models out of models.py

BasPH commented on issue #5056: [WIP][AIRFLOW-4259] Move remaining models out 
of models.py
URL: https://github.com/apache/airflow/pull/5056#issuecomment-480872981
 
 
   Thanks.
   
   1. Fokko placed all issues under a parent issue so I'll reference that.
   2. Will import the other classes in this PR too.
   3. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] XD-DENG commented on issue #5058: [AIRFLOW-XXX] Add newline docstring for doc generation

XD-DENG commented on issue #5058: [AIRFLOW-XXX] Add newline docstring for doc 
generation
URL: https://github.com/apache/airflow/pull/5058#issuecomment-480851237
 
 
   @zhongjiajie welcome ;-)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] XD-DENG commented on a change in pull request #5065: [AIRFLOW-4261] Minor refactoring on jobs.py

XD-DENG commented on a change in pull request #5065: [AIRFLOW-4261] Minor 
refactoring on jobs.py
URL: https://github.com/apache/airflow/pull/5065#discussion_r273068593
 
 

 ##
 File path: airflow/jobs.py
 ##
 @@ -723,7 +723,7 @@ def manage_slas(self, dag, session=None):
 emails |= set(get_email_address_list(task.email))
 elif isinstance(task.email, (list, tuple)):
 emails |= set(task.email)
-if emails and len(slas):
+if emails:
 
 Review comment:
   This line is in the if-check block of `if slas:` 
https://github.com/apache/airflow/blob/fc3b45a61ac51441c3b9e4d99a20473cd3664056/airflow/jobs.py#L673
   
   It's impossible for `len(slas)` to be zero here (correct me if I missed 
anything here)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] XD-DENG commented on a change in pull request #5065: [AIRFLOW-4261] Minor refactoring on jobs.py

XD-DENG commented on a change in pull request #5065: [AIRFLOW-4261] Minor 
refactoring on jobs.py
URL: https://github.com/apache/airflow/pull/5065#discussion_r273069300
 
 

 ##
 File path: airflow/jobs.py
 ##
 @@ -1428,14 +1428,14 @@ def _process_dags(self, dagbag, dags, tis_out):
 """
 for dag in dags:
 dag = dagbag.get_dag(dag.dag_id)
-if dag.is_paused:
-self.log.info("Not processing DAG %s since it's paused", 
dag.dag_id)
-continue
-
 if not dag:
 self.log.error("DAG ID %s was not found in the DagBag", 
dag.dag_id)
 continue
 
+if dag.is_paused:
+self.log.info("Not processing DAG %s since it's paused", 
dag.dag_id)
+continue
+
 
 Review comment:
   The order of checking `if not dag:` and `if dag.is_paused:` was improper 
here.
   
   If dag is `None`, `if dag.is_paused:` will throw exception, and check `if 
not dag:` will never be invoked.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] XD-DENG commented on a change in pull request #5065: [AIRFLOW-4261] Minor refactoring on jobs.py

XD-DENG commented on a change in pull request #5065: [AIRFLOW-4261] Minor 
refactoring on jobs.py
URL: https://github.com/apache/airflow/pull/5065#discussion_r273068593
 
 

 ##
 File path: airflow/jobs.py
 ##
 @@ -723,7 +723,7 @@ def manage_slas(self, dag, session=None):
 emails |= set(get_email_address_list(task.email))
 elif isinstance(task.email, (list, tuple)):
 emails |= set(task.email)
-if emails and len(slas):
+if emails:
 
 Review comment:
   This line is in the if-check block of `if slas:` 
(https://github.com/apache/airflow/blob/fc3b45a61ac51441c3b9e4d99a20473cd3664056/airflow/jobs.py#L673).
 It's impossible for `len(slas)` to be zero here (correct me if I missed 
anything here)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] XD-DENG commented on a change in pull request #5065: [AIRFLOW-4261] Minor refactoring on jobs.py

XD-DENG commented on a change in pull request #5065: [AIRFLOW-4261] Minor 
refactoring on jobs.py
URL: https://github.com/apache/airflow/pull/5065#discussion_r273067758
 
 

 ##
 File path: airflow/jobs.py
 ##
 @@ -675,10 +674,11 @@ def manage_slas(self, dag, session=None):
 qry = (
 session
 .query(TI)
-.filter(TI.state != State.SUCCESS)
-.filter(TI.execution_date.in_(sla_dates))
-.filter(TI.dag_id == dag.dag_id)
-.all()
+.filter(
+TI.state != State.SUCCESS,
+TI.execution_date.in_(sla_dates),
+TI.dag_id == dag.dag_id
+).all()
 
 Review comment:
   This change can bring minor performance improvement. 
   
   Benchmarking for this sort of change was done in 
https://github.com/apache/airflow/pull/4433


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] XD-DENG commented on a change in pull request #5065: [AIRFLOW-4261] Minor refactoring on jobs.py

XD-DENG commented on a change in pull request #5065: [AIRFLOW-4261] Minor 
refactoring on jobs.py
URL: https://github.com/apache/airflow/pull/5065#discussion_r273067697
 
 

 ##
 File path: airflow/jobs.py
 ##
 @@ -665,8 +665,7 @@ def manage_slas(self, dag, session=None):
 slas = (
 session
 .query(SlaMiss)
-.filter(SlaMiss.notification_sent == False)  # noqa: E712
-.filter(SlaMiss.dag_id == dag.dag_id)
+.filter(SlaMiss.notification_sent == False, SlaMiss.dag_id == 
dag.dag_id)  # noqa: E712
 
 Review comment:
   This change can bring minor performance improvement. 
   
   Benchmarking for this sort of change was done in 
https://github.com/apache/airflow/pull/4433


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4261) To fix some minor issues in airflow/jobs.py



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812456#comment-16812456
 ] 

ASF GitHub Bot commented on AIRFLOW-4261:
-

XD-DENG commented on pull request #5065: [AIRFLOW-4261] Minor refactoring on 
jobs.py
URL: https://github.com/apache/airflow/pull/5065
 
 
   ### Jira
   
 - https://issues.apache.org/jira/browse/AIRFLOW-4261
   
   ### Description
   
   Inside this PR, I tried to address a few minor issues, detailed description 
will be added in the code change.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> To fix some minor issues in airflow/jobs.py
> ---
>
> Key: AIRFLOW-4261
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4261
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: scheduler
>Affects Versions: 1.10.2
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Minor
>
> There are some minor issues in airflow/jobs.py. Clearing them can help 1. 
> make code cleaner; 2. introduce minor performance improvement.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] XD-DENG opened a new pull request #5065: [AIRFLOW-4261] Minor refactoring on jobs.py

XD-DENG opened a new pull request #5065: [AIRFLOW-4261] Minor refactoring on 
jobs.py
URL: https://github.com/apache/airflow/pull/5065
 
 
   ### Jira
   
 - https://issues.apache.org/jira/browse/AIRFLOW-4261
   
   ### Description
   
   Inside this PR, I tried to address a few minor issues, detailed description 
will be added in the code change.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-4261) To fix some minor issues in airflow/jobs.py

2019-04-08 Thread Xiaodong DENG (JIRA)

Xiaodong DENG created AIRFLOW-4261:
--

 Summary: To fix some minor issues in airflow/jobs.py
 Key: AIRFLOW-4261
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4261
 Project: Apache Airflow
  Issue Type: Improvement
  Components: scheduler
Affects Versions: 1.10.2
Reporter: Xiaodong DENG
Assignee: Xiaodong DENG


There are some minor issues in airflow/jobs.py. Clearing them can help 1. make 
code cleaner; 2. introduce minor performance improvement.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] codecov-io edited a comment on issue #5054: [AIRFLOW-4255] Replace Discovery based api with client based for GCS

codecov-io edited a comment on issue #5054: [AIRFLOW-4255] Replace Discovery 
based api with client based for GCS
URL: https://github.com/apache/airflow/pull/5054#issuecomment-480762291
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5054?src=pr=h1) 
Report
   > Merging 
[#5054](https://codecov.io/gh/apache/airflow/pull/5054?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/bd08034811f6019dfbf8da2b37c76c5696912340?src=pr=desc)
 will **increase** coverage by `0.55%`.
   > The diff coverage is `43.6%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5054/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5054?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#5054  +/-   ##
   ==
   + Coverage   76.36%   76.91%   +0.55% 
   ==
 Files 471  455  -16 
 Lines   3029529655 -640 
   ==
   - Hits2313522810 -325 
   + Misses   7160 6845 -315
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/5054?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/models/xcom.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMveGNvbS5weQ==)
 | `80% <100%> (+0.2%)` | :arrow_up: |
   | 
[airflow/contrib/hooks/gcs\_hook.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djc19ob29rLnB5)
 | `53.64% <43.07%> (-1.02%)` | :arrow_down: |
   | 
[airflow/contrib/operators/gcs\_download\_operator.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfZG93bmxvYWRfb3BlcmF0b3IucHk=)
 | `88.46% <50%> (+0.46%)` | :arrow_up: |
   | 
[airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=)
 | `92.95% <0%> (-0.05%)` | :arrow_down: |
   | 
[airflow/contrib/hooks/dingding\_hook.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2RpbmdkaW5nX2hvb2sucHk=)
 | `66.66% <0%> (ø)` | :arrow_up: |
   | 
[airflow/\_vendor/nvd3/scatterChart.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9fdmVuZG9yL252ZDMvc2NhdHRlckNoYXJ0LnB5)
 | | |
   | 
[airflow/\_vendor/nvd3/lineChart.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9fdmVuZG9yL252ZDMvbGluZUNoYXJ0LnB5)
 | | |
   | 
[airflow/\_vendor/nvd3/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9fdmVuZG9yL252ZDMvX19pbml0X18ucHk=)
 | | |
   | 
[airflow/\_vendor/nvd3/discreteBarChart.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9fdmVuZG9yL252ZDMvZGlzY3JldGVCYXJDaGFydC5weQ==)
 | | |
   | 
[airflow/\_vendor/nvd3/linePlusBarChart.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9fdmVuZG9yL252ZDMvbGluZVBsdXNCYXJDaGFydC5weQ==)
 | | |
   | ... and [11 
more](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5054?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5054?src=pr=footer). 
Last update 
[bd08034...772944b](https://codecov.io/gh/apache/airflow/pull/5054?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #5054: [AIRFLOW-4255] Replace Discovery based api with client based for GCS

potiuk commented on a change in pull request #5054: [AIRFLOW-4255] Replace 
Discovery based api with client based for GCS
URL: https://github.com/apache/airflow/pull/5054#discussion_r273028362
 
 

 ##
 File path: airflow/contrib/hooks/gcs_hook.py
 ##
 @@ -253,8 +253,8 @@ def is_updated_after(self, bucket, object, ts):
 
 if blob_update_time > ts:
 return True
-else:
-return False
+
+return False
 
 Review comment:
    


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #5054: [AIRFLOW-4255] Replace Discovery based api with client based for GCS

potiuk commented on a change in pull request #5054: [AIRFLOW-4255] Replace 
Discovery based api with client based for GCS
URL: https://github.com/apache/airflow/pull/5054#discussion_r273028272
 
 

 ##
 File path: airflow/contrib/hooks/gcs_hook.py
 ##
 @@ -212,44 +197,15 @@ def upload(self, bucket, object, filename,
 shutil.copyfileobj(f_in, f_out)
 filename = filename_gz
 
-try:
-if multipart:
-if multipart is True:
-chunksize = 256 * 1024 * 1024
-else:
-chunksize = multipart
-
-if chunksize % (256 * 1024) > 0 or chunksize < 0:
-raise ValueError("Multipart size is not a multiple of 
262144 (256KiB)")
-
-media = MediaFileUpload(filename, mimetype=mime_type,
-chunksize=chunksize, resumable=True)
-
-request = service.objects().insert(bucket=bucket, name=object, 
media_body=media)
-response = None
-while response is None:
-status, response = 
request.next_chunk(num_retries=num_retries)
-if status:
-self.log.info("Upload progress %.1f%%", 
status.progress() * 100)
-
-else:
-media = MediaFileUpload(filename, mime_type)
-
-service \
-.objects() \
-.insert(bucket=bucket, name=object, media_body=media) \
-.execute(num_retries=num_retries)
-
-except HttpError as ex:
-if ex.resp['status'] == '404':
-return False
-raise
-
-finally:
-if gzip:
-os.remove(filename)
+client = self.get_conn()
+bucket = client.get_bucket(bucket_name=bucket)
+blob = bucket.blob(blob_name=object)
+blob.upload_from_filename(filename=filename,
+  content_type=mime_type)
 
-return True
+if gzip:
+os.remove(filename)
+self.log.info('File %s uploaded to %s in %s bucket', filename, object, 
bucket)
 
 # pylint:disable=redefined-builtin
 def exists(self, bucket, object):
 
 Review comment:
   Makes perfect sense @kaxil  . Thanks for explanation. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #5054: [AIRFLOW-4255] Replace Discovery based api with client based for GCS

kaxil commented on a change in pull request #5054: [AIRFLOW-4255] Replace 
Discovery based api with client based for GCS
URL: https://github.com/apache/airflow/pull/5054#discussion_r273018225
 
 

 ##
 File path: airflow/contrib/hooks/gcs_hook.py
 ##
 @@ -212,44 +197,15 @@ def upload(self, bucket, object, filename,
 shutil.copyfileobj(f_in, f_out)
 filename = filename_gz
 
-try:
-if multipart:
-if multipart is True:
-chunksize = 256 * 1024 * 1024
-else:
-chunksize = multipart
-
-if chunksize % (256 * 1024) > 0 or chunksize < 0:
-raise ValueError("Multipart size is not a multiple of 
262144 (256KiB)")
-
-media = MediaFileUpload(filename, mimetype=mime_type,
-chunksize=chunksize, resumable=True)
-
-request = service.objects().insert(bucket=bucket, name=object, 
media_body=media)
-response = None
-while response is None:
-status, response = 
request.next_chunk(num_retries=num_retries)
-if status:
-self.log.info("Upload progress %.1f%%", 
status.progress() * 100)
-
-else:
-media = MediaFileUpload(filename, mime_type)
-
-service \
-.objects() \
-.insert(bucket=bucket, name=object, media_body=media) \
-.execute(num_retries=num_retries)
-
-except HttpError as ex:
-if ex.resp['status'] == '404':
-return False
-raise
-
-finally:
-if gzip:
-os.remove(filename)
+client = self.get_conn()
+bucket = client.get_bucket(bucket_name=bucket)
+blob = bucket.blob(blob_name=object)
+blob.upload_from_filename(filename=filename,
+  content_type=mime_type)
 
-return True
+if gzip:
+os.remove(filename)
+self.log.info('File %s uploaded to %s in %s bucket', filename, object, 
bucket)
 
 # pylint:disable=redefined-builtin
 def exists(self, bucket, object):
 
 Review comment:
   The maintain intention is so that we can cherry-pick this one in 1.10.4.
   
   If you look at this PR and check for "breaking-changes" - the one's that are 
there are not widely used (also in Updating.md). 
   
   I wouldn't want to change the name of something like the `object` parameter 
(or even `bucket`) and just put a note in Updating.md. We wont cherry-pick the 
2nd PR for 1.10.4 and would target 2.0 instead.
   
   They are fundamentally 2 separate pieces: This PR focuses on "Replacing 
discovery api with client api" and not on "updating parameter name".  Also more 
readable in Changelog.
   
   None of the changes in this PR remove or change any **required parameter** 
of any method.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] BasPH commented on issue #5010: [AIRFLOW-4192] Remove obsolete/derived task context variables

BasPH commented on issue #5010: [AIRFLOW-4192] Remove obsolete/derived task 
context variables
URL: https://github.com/apache/airflow/pull/5010#issuecomment-480814116
 
 
   @feng-tao I started this discussion: 
https://lists.apache.org/thread.html/a81b32efc95db41a5bf847ed03b64ae4d3e9c8a2cbeda1bdcf0eb2f5@%3Cdev.airflow.apache.org%3E


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #5054: [AIRFLOW-4255] Replace Discovery based api with client based for GCS

kaxil commented on a change in pull request #5054: [AIRFLOW-4255] Replace 
Discovery based api with client based for GCS
URL: https://github.com/apache/airflow/pull/5054#discussion_r273018225
 
 

 ##
 File path: airflow/contrib/hooks/gcs_hook.py
 ##
 @@ -212,44 +197,15 @@ def upload(self, bucket, object, filename,
 shutil.copyfileobj(f_in, f_out)
 filename = filename_gz
 
-try:
-if multipart:
-if multipart is True:
-chunksize = 256 * 1024 * 1024
-else:
-chunksize = multipart
-
-if chunksize % (256 * 1024) > 0 or chunksize < 0:
-raise ValueError("Multipart size is not a multiple of 
262144 (256KiB)")
-
-media = MediaFileUpload(filename, mimetype=mime_type,
-chunksize=chunksize, resumable=True)
-
-request = service.objects().insert(bucket=bucket, name=object, 
media_body=media)
-response = None
-while response is None:
-status, response = 
request.next_chunk(num_retries=num_retries)
-if status:
-self.log.info("Upload progress %.1f%%", 
status.progress() * 100)
-
-else:
-media = MediaFileUpload(filename, mime_type)
-
-service \
-.objects() \
-.insert(bucket=bucket, name=object, media_body=media) \
-.execute(num_retries=num_retries)
-
-except HttpError as ex:
-if ex.resp['status'] == '404':
-return False
-raise
-
-finally:
-if gzip:
-os.remove(filename)
+client = self.get_conn()
+bucket = client.get_bucket(bucket_name=bucket)
+blob = bucket.blob(blob_name=object)
+blob.upload_from_filename(filename=filename,
+  content_type=mime_type)
 
-return True
+if gzip:
+os.remove(filename)
+self.log.info('File %s uploaded to %s in %s bucket', filename, object, 
bucket)
 
 # pylint:disable=redefined-builtin
 def exists(self, bucket, object):
 
 Review comment:
   The maintain intention is so that we can cherry-pick this one in 1.10.4.
   
   If you look at this PR and check for "breaking-changes" - the one's that are 
there are not widely used (also in Updating.md). 
   
   I wouldn't want to change the name of something like the `object` parameter 
(or even `bucket`) and put a note in Updating.md. We should not be added the 
2nd PR for 1.10.4 and would target 2.0.
   
   They are fundamentally 2 separate pieces: This PR focuses on "Replacing 
discovery api with client api" and not on "updating parameter name".  Also more 
readable in Changelog.
   
   None of the changes in this PR remove or change any **required parameter** 
of any method.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #5054: [AIRFLOW-4255] Replace Discovery based api with client based for GCS

kaxil commented on a change in pull request #5054: [AIRFLOW-4255] Replace 
Discovery based api with client based for GCS
URL: https://github.com/apache/airflow/pull/5054#discussion_r273018225
 
 

 ##
 File path: airflow/contrib/hooks/gcs_hook.py
 ##
 @@ -212,44 +197,15 @@ def upload(self, bucket, object, filename,
 shutil.copyfileobj(f_in, f_out)
 filename = filename_gz
 
-try:
-if multipart:
-if multipart is True:
-chunksize = 256 * 1024 * 1024
-else:
-chunksize = multipart
-
-if chunksize % (256 * 1024) > 0 or chunksize < 0:
-raise ValueError("Multipart size is not a multiple of 
262144 (256KiB)")
-
-media = MediaFileUpload(filename, mimetype=mime_type,
-chunksize=chunksize, resumable=True)
-
-request = service.objects().insert(bucket=bucket, name=object, 
media_body=media)
-response = None
-while response is None:
-status, response = 
request.next_chunk(num_retries=num_retries)
-if status:
-self.log.info("Upload progress %.1f%%", 
status.progress() * 100)
-
-else:
-media = MediaFileUpload(filename, mime_type)
-
-service \
-.objects() \
-.insert(bucket=bucket, name=object, media_body=media) \
-.execute(num_retries=num_retries)
-
-except HttpError as ex:
-if ex.resp['status'] == '404':
-return False
-raise
-
-finally:
-if gzip:
-os.remove(filename)
+client = self.get_conn()
+bucket = client.get_bucket(bucket_name=bucket)
+blob = bucket.blob(blob_name=object)
+blob.upload_from_filename(filename=filename,
+  content_type=mime_type)
 
-return True
+if gzip:
+os.remove(filename)
+self.log.info('File %s uploaded to %s in %s bucket', filename, object, 
bucket)
 
 # pylint:disable=redefined-builtin
 def exists(self, bucket, object):
 
 Review comment:
   The maintain intention is so that we can cherry-pick this one in 1.10.4.
   
   If you look at this PR and check for "breaking-changes" - the one's that are 
there are not widely used (also in Updating.md). 
   
   I wouldn't want to change the name of something like the `object` parameter 
(or even `bucket`) and put a note in Updating.md. We should not be added the 
2nd PR for 1.10.4 and would target 2.0.
   
   They are fundamentally 2 separate pieces: This PR focuses on "Replacing 
discovery api with client api" and not on "updating parameter name".  Also more 
readable in Changelog.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #5054: [AIRFLOW-4255] Replace Discovery based api with client based for GCS

mik-laj commented on a change in pull request #5054: [AIRFLOW-4255] Replace 
Discovery based api with client based for GCS
URL: https://github.com/apache/airflow/pull/5054#discussion_r273014652
 
 

 ##
 File path: airflow/contrib/hooks/gcs_hook.py
 ##
 @@ -212,44 +197,15 @@ def upload(self, bucket, object, filename,
 shutil.copyfileobj(f_in, f_out)
 filename = filename_gz
 
-try:
-if multipart:
-if multipart is True:
-chunksize = 256 * 1024 * 1024
-else:
-chunksize = multipart
-
-if chunksize % (256 * 1024) > 0 or chunksize < 0:
-raise ValueError("Multipart size is not a multiple of 
262144 (256KiB)")
-
-media = MediaFileUpload(filename, mimetype=mime_type,
-chunksize=chunksize, resumable=True)
-
-request = service.objects().insert(bucket=bucket, name=object, 
media_body=media)
-response = None
-while response is None:
-status, response = 
request.next_chunk(num_retries=num_retries)
-if status:
-self.log.info("Upload progress %.1f%%", 
status.progress() * 100)
-
-else:
-media = MediaFileUpload(filename, mime_type)
-
-service \
-.objects() \
-.insert(bucket=bucket, name=object, media_body=media) \
-.execute(num_retries=num_retries)
-
-except HttpError as ex:
-if ex.resp['status'] == '404':
-return False
-raise
-
-finally:
-if gzip:
-os.remove(filename)
+client = self.get_conn()
+bucket = client.get_bucket(bucket_name=bucket)
+blob = bucket.blob(blob_name=object)
+blob.upload_from_filename(filename=filename,
+  content_type=mime_type)
 
-return True
+if gzip:
+os.remove(filename)
+self.log.info('File %s uploaded to %s in %s bucket', filename, object, 
bucket)
 
 # pylint:disable=redefined-builtin
 def exists(self, bucket, object):
 
 Review comment:
   Can you explain what are the intentions of sharing one refactorization for a 
few PR's?  This makes changes much more difficult to review.  I see a reason if 
this change was backwards compatible, but it is not. We have a note in 
fle`Updating.md`


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #5054: [AIRFLOW-4255] Replace Discovery based api with client based for GCS

mik-laj commented on a change in pull request #5054: [AIRFLOW-4255] Replace 
Discovery based api with client based for GCS
URL: https://github.com/apache/airflow/pull/5054#discussion_r273014652
 
 

 ##
 File path: airflow/contrib/hooks/gcs_hook.py
 ##
 @@ -212,44 +197,15 @@ def upload(self, bucket, object, filename,
 shutil.copyfileobj(f_in, f_out)
 filename = filename_gz
 
-try:
-if multipart:
-if multipart is True:
-chunksize = 256 * 1024 * 1024
-else:
-chunksize = multipart
-
-if chunksize % (256 * 1024) > 0 or chunksize < 0:
-raise ValueError("Multipart size is not a multiple of 
262144 (256KiB)")
-
-media = MediaFileUpload(filename, mimetype=mime_type,
-chunksize=chunksize, resumable=True)
-
-request = service.objects().insert(bucket=bucket, name=object, 
media_body=media)
-response = None
-while response is None:
-status, response = 
request.next_chunk(num_retries=num_retries)
-if status:
-self.log.info("Upload progress %.1f%%", 
status.progress() * 100)
-
-else:
-media = MediaFileUpload(filename, mime_type)
-
-service \
-.objects() \
-.insert(bucket=bucket, name=object, media_body=media) \
-.execute(num_retries=num_retries)
-
-except HttpError as ex:
-if ex.resp['status'] == '404':
-return False
-raise
-
-finally:
-if gzip:
-os.remove(filename)
+client = self.get_conn()
+bucket = client.get_bucket(bucket_name=bucket)
+blob = bucket.blob(blob_name=object)
+blob.upload_from_filename(filename=filename,
+  content_type=mime_type)
 
-return True
+if gzip:
+os.remove(filename)
+self.log.info('File %s uploaded to %s in %s bucket', filename, object, 
bucket)
 
 # pylint:disable=redefined-builtin
 def exists(self, bucket, object):
 
 Review comment:
   Can you indicate what are the intentions of sharing one refactorization for 
a few PR's?  This makes changes much more difficult to review.  I see a reason 
if this change was backwards compatible, but it is not. We have a note in 
fle`Updating.md`


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-3876) Invalid distutil import

2019-04-08 Thread Ash Berlin-Taylor (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor updated AIRFLOW-3876:
---
Fix Version/s: (was: 2.0.0)
   1.10.4

> Invalid distutil import
> ---
>
> Key: AIRFLOW-3876
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3876
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: operators
>Affects Versions: 1.10.2
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
> Fix For: 1.10.4
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] kaxil commented on a change in pull request #5054: [AIRFLOW-4255] Replace Discovery based api with client based for GCS

kaxil commented on a change in pull request #5054: [AIRFLOW-4255] Replace 
Discovery based api with client based for GCS
URL: https://github.com/apache/airflow/pull/5054#discussion_r273010207
 
 

 ##
 File path: airflow/contrib/hooks/gcs_hook.py
 ##
 @@ -212,44 +197,15 @@ def upload(self, bucket, object, filename,
 shutil.copyfileobj(f_in, f_out)
 filename = filename_gz
 
-try:
-if multipart:
-if multipart is True:
-chunksize = 256 * 1024 * 1024
-else:
-chunksize = multipart
-
-if chunksize % (256 * 1024) > 0 or chunksize < 0:
-raise ValueError("Multipart size is not a multiple of 
262144 (256KiB)")
-
-media = MediaFileUpload(filename, mimetype=mime_type,
-chunksize=chunksize, resumable=True)
-
-request = service.objects().insert(bucket=bucket, name=object, 
media_body=media)
-response = None
-while response is None:
-status, response = 
request.next_chunk(num_retries=num_retries)
-if status:
-self.log.info("Upload progress %.1f%%", 
status.progress() * 100)
-
-else:
-media = MediaFileUpload(filename, mime_type)
-
-service \
-.objects() \
-.insert(bucket=bucket, name=object, media_body=media) \
-.execute(num_retries=num_retries)
-
-except HttpError as ex:
-if ex.resp['status'] == '404':
-return False
-raise
-
-finally:
-if gzip:
-os.remove(filename)
+client = self.get_conn()
+bucket = client.get_bucket(bucket_name=bucket)
+blob = bucket.blob(blob_name=object)
+blob.upload_from_filename(filename=filename,
+  content_type=mime_type)
 
-return True
+if gzip:
+os.remove(filename)
+self.log.info('File %s uploaded to %s in %s bucket', filename, object, 
bucket)
 
 # pylint:disable=redefined-builtin
 def exists(self, bucket, object):
 
 Review comment:
   Ya, I have that PR ready. I am trying to keep the changes in this PR to be 
more on a backwards-compatible side.
   
   The next PR will contain some breaking changes which will contain these name 
changes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #5054: [AIRFLOW-4255] Replace Discovery based api with client based for GCS

kaxil commented on a change in pull request #5054: [AIRFLOW-4255] Replace 
Discovery based api with client based for GCS
URL: https://github.com/apache/airflow/pull/5054#discussion_r273009775
 
 

 ##
 File path: airflow/contrib/hooks/gcs_hook.py
 ##
 @@ -287,57 +236,41 @@ def is_updated_after(self, bucket, object, ts):
 :param ts: The timestamp to check against.
 :type ts: datetime.datetime
 """
-service = self.get_conn()
-try:
-response = (service
-.objects()
-.get(bucket=bucket, object=object)
-.execute())
-
-if 'updated' in response:
-import dateutil.parser
-import dateutil.tz
+client = self.get_conn()
+bucket = storage.Bucket(client=client, name=bucket)
+blob = bucket.get_blob(blob_name=object)
+blob.reload()
 
-if not ts.tzinfo:
-ts = ts.replace(tzinfo=dateutil.tz.tzutc())
+blob_update_time = blob.updated
 
-updated = dateutil.parser.parse(response['updated'])
-self.log.info("Verify object date: %s > %s", updated, ts)
+if blob_update_time is not None:
+import dateutil.tz
 
-if updated > ts:
-return True
+if not ts.tzinfo:
+ts = ts.replace(tzinfo=dateutil.tz.tzutc())
 
-except HttpError as ex:
-if ex.resp['status'] != '404':
-raise
+self.log.info("Verify object date: %s > %s", blob_update_time, ts)
 
-return False
+if blob_update_time > ts:
+return True
+else:
+return False
 
 Review comment:
   Good call.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4260) Fix sphinx deprecation warning for autodoc_default_flags



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812357#comment-16812357
 ] 

ASF GitHub Bot commented on AIRFLOW-4260:
-

feluelle commented on pull request #5064: [AIRFLOW-4260] Fix sphinx deprecation 
warning for autodoc_default_flags
URL: https://github.com/apache/airflow/pull/5064
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-4260
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   Since sphinx version 1.8 `autodoc_default_flags` is deprecated and can be 
replaced by using `autodoc_default_options` instead.
   
   See: 
http://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html#confval-autodoc_default_flags
 (and right below this option is the new one)
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [x] Passes `flake8`
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Fix sphinx deprecation warning for autodoc_default_flags
> 
>
> Key: AIRFLOW-4260
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4260
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Trivial
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] feluelle opened a new pull request #5064: [AIRFLOW-4260] Fix sphinx deprecation warning for autodoc_default_flags

feluelle opened a new pull request #5064: [AIRFLOW-4260] Fix sphinx deprecation 
warning for autodoc_default_flags
URL: https://github.com/apache/airflow/pull/5064
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-4260
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   Since sphinx version 1.8 `autodoc_default_flags` is deprecated and can be 
replaced by using `autodoc_default_options` instead.
   
   See: 
http://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html#confval-autodoc_default_flags
 (and right below this option is the new one)
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [x] Passes `flake8`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-4260) Fix sphinx deprecation warning for autodoc_default_flags

2019-04-08 Thread Felix Uellendall (JIRA)

Felix Uellendall created AIRFLOW-4260:
-

 Summary: Fix sphinx deprecation warning for autodoc_default_flags
 Key: AIRFLOW-4260
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4260
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Felix Uellendall
Assignee: Felix Uellendall






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3470) Refactor: Move Variable out of models.py



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-3470:
--
Parent: AIRFLOW-4259
Issue Type: Sub-task  (was: Task)

> Refactor: Move Variable out of models.py
> 
>
> Key: AIRFLOW-3470
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3470
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: models
>Affects Versions: 1.10.1
>Reporter: Fokko Driesprong
>Assignee: Bas Harenslak
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] potiuk commented on a change in pull request #5054: [AIRFLOW-4255] Replace Discovery based api with client based for GCS

potiuk commented on a change in pull request #5054: [AIRFLOW-4255] Replace 
Discovery based api with client based for GCS
URL: https://github.com/apache/airflow/pull/5054#discussion_r272994038
 
 

 ##
 File path: airflow/contrib/hooks/gcs_hook.py
 ##
 @@ -212,44 +197,15 @@ def upload(self, bucket, object, filename,
 shutil.copyfileobj(f_in, f_out)
 filename = filename_gz
 
-try:
-if multipart:
-if multipart is True:
-chunksize = 256 * 1024 * 1024
-else:
-chunksize = multipart
-
-if chunksize % (256 * 1024) > 0 or chunksize < 0:
-raise ValueError("Multipart size is not a multiple of 
262144 (256KiB)")
-
-media = MediaFileUpload(filename, mimetype=mime_type,
-chunksize=chunksize, resumable=True)
-
-request = service.objects().insert(bucket=bucket, name=object, 
media_body=media)
-response = None
-while response is None:
-status, response = 
request.next_chunk(num_retries=num_retries)
-if status:
-self.log.info("Upload progress %.1f%%", 
status.progress() * 100)
-
-else:
-media = MediaFileUpload(filename, mime_type)
-
-service \
-.objects() \
-.insert(bucket=bucket, name=object, media_body=media) \
-.execute(num_retries=num_retries)
-
-except HttpError as ex:
-if ex.resp['status'] == '404':
-return False
-raise
-
-finally:
-if gzip:
-os.remove(filename)
+client = self.get_conn()
+bucket = client.get_bucket(bucket_name=bucket)
+blob = bucket.blob(blob_name=object)
+blob.upload_from_filename(filename=filename,
+  content_type=mime_type)
 
-return True
+if gzip:
+os.remove(filename)
+self.log.info('File %s uploaded to %s in %s bucket', filename, object, 
bucket)
 
 # pylint:disable=redefined-builtin
 def exists(self, bucket, object):
 
 Review comment:
   Suggestion: maybe we can change the "object" name in the signature of 
functions. Since we are introducing backwards-incompatible changes anyway, that 
might be good time to get rid of the "object" redefinition and remove the 
pylint disable warnings.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #5054: [AIRFLOW-4255] Replace Discovery based api with client based for GCS

potiuk commented on a change in pull request #5054: [AIRFLOW-4255] Replace 
Discovery based api with client based for GCS
URL: https://github.com/apache/airflow/pull/5054#discussion_r272993210
 
 

 ##
 File path: airflow/contrib/hooks/gcs_hook.py
 ##
 @@ -287,57 +236,41 @@ def is_updated_after(self, bucket, object, ts):
 :param ts: The timestamp to check against.
 :type ts: datetime.datetime
 """
-service = self.get_conn()
-try:
-response = (service
-.objects()
-.get(bucket=bucket, object=object)
-.execute())
-
-if 'updated' in response:
-import dateutil.parser
-import dateutil.tz
+client = self.get_conn()
+bucket = storage.Bucket(client=client, name=bucket)
+blob = bucket.get_blob(blob_name=object)
+blob.reload()
 
-if not ts.tzinfo:
-ts = ts.replace(tzinfo=dateutil.tz.tzutc())
+blob_update_time = blob.updated
 
-updated = dateutil.parser.parse(response['updated'])
-self.log.info("Verify object date: %s > %s", updated, ts)
+if blob_update_time is not None:
+import dateutil.tz
 
-if updated > ts:
-return True
+if not ts.tzinfo:
+ts = ts.replace(tzinfo=dateutil.tz.tzutc())
 
-except HttpError as ex:
-if ex.resp['status'] != '404':
-raise
+self.log.info("Verify object date: %s > %s", blob_update_time, ts)
 
-return False
+if blob_update_time > ts:
+return True
+else:
+return False
 
 Review comment:
   'return False' is missing for if blob_udate_time is None


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-3464) Refactor: Move SkipMixin out of models.py



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-3464:
--
Parent: AIRFLOW-4259
Issue Type: Sub-task  (was: Task)

> Refactor: Move SkipMixin out of models.py
> -
>
> Key: AIRFLOW-3464
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3464
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: models
>Affects Versions: 1.10.1
>Reporter: Fokko Driesprong
>Assignee: Bas Harenslak
>Priority: Major
> Fix For: 1.10.3
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3467) Refactor: Move Chart out of models.py



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-3467:
--
Parent: AIRFLOW-4259
Issue Type: Sub-task  (was: Task)

> Refactor: Move Chart out of models.py
> -
>
> Key: AIRFLOW-3467
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3467
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: models
>Affects Versions: 1.10.1
>Reporter: Fokko Driesprong
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3469) Refactor: Move KnownEvent out of models.py



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-3469:
--
Parent: AIRFLOW-4259
Issue Type: Sub-task  (was: Task)

> Refactor: Move KnownEvent out of models.py
> --
>
> Key: AIRFLOW-3469
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3469
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: models
>Affects Versions: 1.10.1
>Reporter: Fokko Driesprong
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3476) Refactor: Move KubeResourceVersion out of models.py



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-3476:
--
Parent: AIRFLOW-4259
Issue Type: Sub-task  (was: Task)

> Refactor: Move KubeResourceVersion out of models.py
> ---
>
> Key: AIRFLOW-3476
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3476
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: models
>Affects Versions: 1.10.1
>Reporter: Fokko Driesprong
>Assignee: Bas Harenslak
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3477) Refactor: Move KubeWorkerIdentifier out of models.py



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-3477:
--
Parent: AIRFLOW-4259
Issue Type: Sub-task  (was: Task)

> Refactor: Move KubeWorkerIdentifier out of models.py
> 
>
> Key: AIRFLOW-3477
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3477
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: models
>Affects Versions: 1.10.1
>Reporter: Fokko Driesprong
>Assignee: Bas Harenslak
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3468) Refactor: Move KnownEventType out of models.py



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-3468:
--
Parent: AIRFLOW-4259
Issue Type: Sub-task  (was: Task)

> Refactor: Move KnownEventType out of models.py
> --
>
> Key: AIRFLOW-3468
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3468
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: models
>Affects Versions: 1.10.1
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3458) Refactor: Move Connection out of models.py



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-3458:
--
Parent: AIRFLOW-4259
Issue Type: Sub-task  (was: Task)

> Refactor: Move Connection out of models.py
> --
>
> Key: AIRFLOW-3458
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3458
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: models
>Affects Versions: 1.10.1
>Reporter: Fokko Driesprong
>Assignee: Bas Harenslak
>Priority: Major
> Fix For: 2.0.0, 1.10.3
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3463) Refactor: Move Log out of models.py



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-3463:
--
Parent: AIRFLOW-4259
Issue Type: Sub-task  (was: Task)

> Refactor: Move Log out of models.py
> ---
>
> Key: AIRFLOW-3463
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3463
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: models
>Affects Versions: 1.10.1
>Reporter: Fokko Driesprong
>Assignee: Andy Huynh
>Priority: Major
> Fix For: 1.10.3
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3459) Refactor: Move DagPickle out of models.py



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-3459:
--
Parent: AIRFLOW-4259
Issue Type: Sub-task  (was: Task)

> Refactor: Move DagPickle out of models.py
> -
>
> Key: AIRFLOW-3459
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3459
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: models
>Affects Versions: 1.10.1
>Reporter: Fokko Driesprong
>Assignee: Bas Harenslak
>Priority: Major
> Fix For: 1.10.3
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3466) Refactor: Move DAG out of models.py



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-3466:
--
Parent: AIRFLOW-4259
Issue Type: Sub-task  (was: Task)

> Refactor: Move DAG out of models.py
> ---
>
> Key: AIRFLOW-3466
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3466
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: models
>Affects Versions: 1.10.1
>Reporter: Fokko Driesprong
>Assignee: Bas Harenslak
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3472) Refactor: Move DagStat out of models.py



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-3472:
--
Parent: AIRFLOW-4259
Issue Type: Sub-task  (was: Task)

> Refactor: Move DagStat out of models.py
> ---
>
> Key: AIRFLOW-3472
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3472
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: models
>Affects Versions: 1.10.1
>Reporter: Fokko Driesprong
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3456) Refactor: Move DagBag out of models.py



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-3456:
--
Parent: AIRFLOW-4259
Issue Type: Sub-task  (was: Task)

> Refactor: Move DagBag out of models.py
> --
>
> Key: AIRFLOW-3456
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3456
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: models
>Affects Versions: 1.10.1
>Reporter: Fokko Driesprong
>Assignee: Bas Harenslak
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3474) Refactor: Move SlaMiss out of models.py



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-3474:
--
Parent: AIRFLOW-4259
Issue Type: Sub-task  (was: Task)

> Refactor: Move SlaMiss out of models.py
> ---
>
> Key: AIRFLOW-3474
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3474
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: models
>Affects Versions: 1.10.1
>Reporter: Fokko Driesprong
>Assignee: Andy Huynh
>Priority: Major
> Fix For: 1.10.3
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3461) Refactor: Move TaskFail out of models.py



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-3461:
--
Parent: AIRFLOW-4259
Issue Type: Sub-task  (was: Task)

> Refactor: Move TaskFail out of models.py
> 
>
> Key: AIRFLOW-3461
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3461
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: models
>Affects Versions: 1.10.1
>Reporter: Fokko Driesprong
>Assignee: Stefan Seelmann
>Priority: Major
> Fix For: 1.10.3
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3457) Refactor: Move User out of models.py



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-3457:
--
Parent: AIRFLOW-4259
Issue Type: Sub-task  (was: Task)

> Refactor: Move User out of models.py
> 
>
> Key: AIRFLOW-3457
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3457
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: models
>Affects Versions: 1.10.1
>Reporter: Fokko Driesprong
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3579) Refactor: Move Pool out of models.py

2019-04-08 Thread Bas Harenslak (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bas Harenslak updated AIRFLOW-3579:
---
Issue Type: Sub-task  (was: Task)
Parent: AIRFLOW-4259

> Refactor: Move Pool out of models.py
> 
>
> Key: AIRFLOW-3579
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3579
> Project: Apache Airflow
>  Issue Type: Sub-task
>Reporter: Bas Harenslak
>Assignee: Bas Harenslak
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3465) Refactor: Move BaseOperator out of models.py



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-3465:
--
Parent: AIRFLOW-4259
Issue Type: Sub-task  (was: Task)

> Refactor: Move BaseOperator out of models.py
> 
>
> Key: AIRFLOW-3465
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3465
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: models
>Affects Versions: 1.10.1
>Reporter: Fokko Driesprong
>Assignee: Bas Harenslak
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3460) Refactor: Move TaskInstance out of models.py



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-3460:
--
Parent: AIRFLOW-4259
Issue Type: Sub-task  (was: Task)

> Refactor: Move TaskInstance out of models.py
> 
>
> Key: AIRFLOW-3460
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3460
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: models
>Affects Versions: 1.10.1
>Reporter: Fokko Driesprong
>Assignee: Bas Harenslak
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3473) Refactor: Move DagRun out of models.py



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-3473:
--
Parent: AIRFLOW-4259
Issue Type: Sub-task  (was: Task)

> Refactor: Move DagRun out of models.py
> --
>
> Key: AIRFLOW-3473
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3473
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: models
>Affects Versions: 1.10.1
>Reporter: Fokko Driesprong
>Assignee: Bas Harenslak
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3471) Refactor: Move XCom out of models.py



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-3471:
--
Parent: AIRFLOW-4259
Issue Type: Sub-task  (was: Task)

> Refactor: Move XCom out of models.py
> 
>
> Key: AIRFLOW-3471
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3471
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: models
>Affects Versions: 1.10.1
>Reporter: Fokko Driesprong
>Assignee: Andy Huynh
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3462) Refactor: Move TaskReschedule out of models.py



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-3462:
--
Parent: AIRFLOW-4259
Issue Type: Sub-task  (was: Task)

> Refactor: Move TaskReschedule out of models.py
> --
>
> Key: AIRFLOW-3462
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3462
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: models
>Affects Versions: 1.10.1
>Reporter: Fokko Driesprong
>Assignee: Stefan Seelmann
>Priority: Major
> Fix For: 1.10.3
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3475) Refactor: Move ImportError out of models.py



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-3475:
--
Parent: AIRFLOW-4259
Issue Type: Sub-task  (was: Task)

> Refactor: Move ImportError out of models.py
> ---
>
> Key: AIRFLOW-3475
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3475
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: models
>Affects Versions: 1.10.1
>Reporter: Fokko Driesprong
>Assignee: Bas Harenslak
>Priority: Major
> Fix For: 1.10.3
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (AIRFLOW-4259) Refactor: move all classes out of models.py



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong reassigned AIRFLOW-4259:
-

Assignee: Bas Harenslak

> Refactor: move all classes out of models.py
> ---
>
> Key: AIRFLOW-4259
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4259
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Bas Harenslak
>Assignee: Bas Harenslak
>Priority: Major
>
> The infamous airflow/models.py contains over 6000 lines and becomes 
> unmanageable. All classes should be placed in separate modules.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-4259) Refactor: move all classes out of models.py



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-4259:
--
Issue Type: Task  (was: Bug)

> Refactor: move all classes out of models.py
> ---
>
> Key: AIRFLOW-4259
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4259
> Project: Apache Airflow
>  Issue Type: Task
>Reporter: Bas Harenslak
>Assignee: Bas Harenslak
>Priority: Major
>
> The infamous airflow/models.py contains over 6000 lines and becomes 
> unmanageable. All classes should be placed in separate modules.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-4259) Refactor: move all classes out of models.py

2019-04-08 Thread Bas Harenslak (JIRA)

Bas Harenslak created AIRFLOW-4259:
--

 Summary: Refactor: move all classes out of models.py
 Key: AIRFLOW-4259
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4259
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Bas Harenslak


The infamous airflow/models.py contains over 6000 lines and becomes 
unmanageable. All classes should be placed in separate modules.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] codecov-io edited a comment on issue #5054: [AIRFLOW-4255] Replace Discovery based api with client based for GCS

codecov-io edited a comment on issue #5054: [AIRFLOW-4255] Replace Discovery 
based api with client based for GCS
URL: https://github.com/apache/airflow/pull/5054#issuecomment-480762291
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5054?src=pr=h1) 
Report
   > Merging 
[#5054](https://codecov.io/gh/apache/airflow/pull/5054?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/bd08034811f6019dfbf8da2b37c76c5696912340?src=pr=desc)
 will **increase** coverage by `0.55%`.
   > The diff coverage is `43.28%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5054/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5054?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#5054  +/-   ##
   ==
   + Coverage   76.36%   76.92%   +0.55% 
   ==
 Files 471  455  -16 
 Lines   3029529655 -640 
   ==
   - Hits2313522811 -324 
   + Misses   7160 6844 -316
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/5054?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/models/xcom.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMveGNvbS5weQ==)
 | `80% <100%> (+0.2%)` | :arrow_up: |
   | 
[airflow/contrib/hooks/gcs\_hook.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djc19ob29rLnB5)
 | `53.64% <42.74%> (-1.02%)` | :arrow_down: |
   | 
[airflow/contrib/operators/gcs\_download\_operator.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfZG93bmxvYWRfb3BlcmF0b3IucHk=)
 | `88.46% <50%> (+0.46%)` | :arrow_up: |
   | 
[airflow/contrib/hooks/dingding\_hook.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2RpbmdkaW5nX2hvb2sucHk=)
 | `66.66% <0%> (ø)` | :arrow_up: |
   | 
[airflow/\_vendor/nvd3/scatterChart.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9fdmVuZG9yL252ZDMvc2NhdHRlckNoYXJ0LnB5)
 | | |
   | 
[airflow/\_vendor/nvd3/lineChart.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9fdmVuZG9yL252ZDMvbGluZUNoYXJ0LnB5)
 | | |
   | 
[airflow/\_vendor/nvd3/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9fdmVuZG9yL252ZDMvX19pbml0X18ucHk=)
 | | |
   | 
[airflow/\_vendor/nvd3/discreteBarChart.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9fdmVuZG9yL252ZDMvZGlzY3JldGVCYXJDaGFydC5weQ==)
 | | |
   | 
[airflow/\_vendor/nvd3/linePlusBarChart.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9fdmVuZG9yL252ZDMvbGluZVBsdXNCYXJDaGFydC5weQ==)
 | | |
   | 
[airflow/\_vendor/nvd3/pieChart.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9fdmVuZG9yL252ZDMvcGllQ2hhcnQucHk=)
 | | |
   | ... and [10 
more](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5054?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5054?src=pr=footer). 
Last update 
[bd08034...81d8837](https://codecov.io/gh/apache/airflow/pull/5054?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #5054: [AIRFLOW-4255] Replace Discovery based api with client based for GCS

codecov-io edited a comment on issue #5054: [AIRFLOW-4255] Replace Discovery 
based api with client based for GCS
URL: https://github.com/apache/airflow/pull/5054#issuecomment-480762291
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5054?src=pr=h1) 
Report
   > Merging 
[#5054](https://codecov.io/gh/apache/airflow/pull/5054?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/bd08034811f6019dfbf8da2b37c76c5696912340?src=pr=desc)
 will **increase** coverage by `0.55%`.
   > The diff coverage is `43.28%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5054/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5054?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#5054  +/-   ##
   ==
   + Coverage   76.36%   76.92%   +0.55% 
   ==
 Files 471  455  -16 
 Lines   3029529655 -640 
   ==
   - Hits2313522811 -324 
   + Misses   7160 6844 -316
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/5054?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/models/xcom.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMveGNvbS5weQ==)
 | `80% <100%> (+0.2%)` | :arrow_up: |
   | 
[airflow/contrib/hooks/gcs\_hook.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djc19ob29rLnB5)
 | `53.64% <42.74%> (-1.02%)` | :arrow_down: |
   | 
[airflow/contrib/operators/gcs\_download\_operator.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfZG93bmxvYWRfb3BlcmF0b3IucHk=)
 | `88.46% <50%> (+0.46%)` | :arrow_up: |
   | 
[airflow/contrib/hooks/dingding\_hook.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2RpbmdkaW5nX2hvb2sucHk=)
 | `66.66% <0%> (ø)` | :arrow_up: |
   | 
[airflow/\_vendor/nvd3/scatterChart.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9fdmVuZG9yL252ZDMvc2NhdHRlckNoYXJ0LnB5)
 | | |
   | 
[airflow/\_vendor/nvd3/lineChart.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9fdmVuZG9yL252ZDMvbGluZUNoYXJ0LnB5)
 | | |
   | 
[airflow/\_vendor/nvd3/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9fdmVuZG9yL252ZDMvX19pbml0X18ucHk=)
 | | |
   | 
[airflow/\_vendor/nvd3/discreteBarChart.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9fdmVuZG9yL252ZDMvZGlzY3JldGVCYXJDaGFydC5weQ==)
 | | |
   | 
[airflow/\_vendor/nvd3/linePlusBarChart.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9fdmVuZG9yL252ZDMvbGluZVBsdXNCYXJDaGFydC5weQ==)
 | | |
   | 
[airflow/\_vendor/nvd3/pieChart.py](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree#diff-YWlyZmxvdy9fdmVuZG9yL252ZDMvcGllQ2hhcnQucHk=)
 | | |
   | ... and [10 
more](https://codecov.io/gh/apache/airflow/pull/5054/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5054?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5054?src=pr=footer). 
Last update 
[bd08034...81d8837](https://codecov.io/gh/apache/airflow/pull/5054?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] zhongjiajie commented on a change in pull request #4779: [WIP][AIRFLOW-3958] Support list tasks as upstream in chain

zhongjiajie commented on a change in pull request #4779: [WIP][AIRFLOW-3958] 
Support list tasks as upstream in chain
URL: https://github.com/apache/airflow/pull/4779#discussion_r272986625
 
 

 ##
 File path: airflow/utils/helpers.py
 ##
 @@ -154,19 +156,46 @@ def as_flattened_list(iterable):
 
 
 def chain(*tasks):
-"""
-Given a number of tasks, builds a dependency chain.
+r"""
+Given a number of tasks or list of tasks, builds a dependency chain.
 
-chain(task_1, task_2, task_3, task_4)
+chain(t1, [t2, t3], [t4, t5], t6)
 
 is equivalent to
 
-task_1.set_downstream(task_2)
-task_2.set_downstream(task_3)
-task_3.set_downstream(task_4)
+  / -> t2 -> t4 \
+t1   -> t6
+  \ -> t3 -> t5 /
+
+t1.set_downstream(t2)
+t1.set_downstream(t3)
+t2.set_downstream(t4)
+t3.set_downstream(t5)
+t4.set_downstream(t6)
+t5.set_downstream(t6)
+
+:param tasks: tasks or list[tasks] to set dependencies
+:type tasks: airflow.models.BaseOperator, List[airflow.models.BaseOperator]
 """
-for up_task, down_task in zip(tasks[:-1], tasks[1:]):
-up_task.set_downstream(down_task)
+for index, task in enumerate(tasks[:-1]):
+down_task = tasks[index + 1]
+if isinstance(task, BaseOperator):
+task.set_downstream(down_task)
+elif isinstance(down_task, BaseOperator):
+down_task.set_upstream(task)
+elif isinstance(task, list) and isinstance(down_task, list):
+# Parallel pipelines
+if len(task) == len(down_task):
+for a, b in zip(task, down_task):
+a.set_downstream(b)
+else:
+raise AirflowException(
 
 Review comment:
   Not really, that the `cross` mean, we already have `cross_downstream` in 
`airflow.utils.helper`.I am inspired in 
https://apache-airflow.slack.com/archives/CCPRP7943/p1554633832061000?thread_ts=1554374078.016800=CCPRP7943
 , thinking we should have a function to make parallel pipelines work
   
   ```
/  -> t1 -> t3
   t0
\  -> t2 -> t4
   ```
   If we do that, `chain` could do `chain(t0, [t1, t2, t3], [t4, t5, t6])` like
   
   ```
/  -> t1 -> t4
   t0  -> t2 -> t5
\  -> t3 -> t6
   ```
   and could `cross_downstream([t1, t2, t3], [t4, t5, t6])`
   
   ```
   t1   \ /  -> t4
   \ /
   t2-> X-> t5
   / \
   t3   /  \ -> t6
   ```
   
   make our dependent more possibilities.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] zhongjiajie commented on a change in pull request #4779: [WIP][AIRFLOW-3958] Support list tasks as upstream in chain

zhongjiajie commented on a change in pull request #4779: [WIP][AIRFLOW-3958] 
Support list tasks as upstream in chain
URL: https://github.com/apache/airflow/pull/4779#discussion_r272986625
 
 

 ##
 File path: airflow/utils/helpers.py
 ##
 @@ -154,19 +156,46 @@ def as_flattened_list(iterable):
 
 
 def chain(*tasks):
-"""
-Given a number of tasks, builds a dependency chain.
+r"""
+Given a number of tasks or list of tasks, builds a dependency chain.
 
-chain(task_1, task_2, task_3, task_4)
+chain(t1, [t2, t3], [t4, t5], t6)
 
 is equivalent to
 
-task_1.set_downstream(task_2)
-task_2.set_downstream(task_3)
-task_3.set_downstream(task_4)
+  / -> t2 -> t4 \
+t1   -> t6
+  \ -> t3 -> t5 /
+
+t1.set_downstream(t2)
+t1.set_downstream(t3)
+t2.set_downstream(t4)
+t3.set_downstream(t5)
+t4.set_downstream(t6)
+t5.set_downstream(t6)
+
+:param tasks: tasks or list[tasks] to set dependencies
+:type tasks: airflow.models.BaseOperator, List[airflow.models.BaseOperator]
 """
-for up_task, down_task in zip(tasks[:-1], tasks[1:]):
-up_task.set_downstream(down_task)
+for index, task in enumerate(tasks[:-1]):
+down_task = tasks[index + 1]
+if isinstance(task, BaseOperator):
+task.set_downstream(down_task)
+elif isinstance(down_task, BaseOperator):
+down_task.set_upstream(task)
+elif isinstance(task, list) and isinstance(down_task, list):
+# Parallel pipelines
+if len(task) == len(down_task):
+for a, b in zip(task, down_task):
+a.set_downstream(b)
+else:
+raise AirflowException(
 
 Review comment:
   Not really, that the `cross` mean, we already have `cross_downstream` in 
`airflow.utils.helper`.I am inspired in 
https://apache-airflow.slack.com/archives/CCPRP7943/p1554633832061000?thread_ts=1554374078.016800=CCPRP7943
 , thinking we should have a function to make parallel pipelines work
   
   ```
/  -> t1 -> t3
   t0
\  -> t2 -> t4
   ```
   If we do that, `chain` could do `chain(t0, [t1, t2, t3], [t4, t5, t6])` like
   
   ```
/  -> t1 -> t4
   t0--> t2 -> t5
\  -> t3 -> t6
   ```
   and could `cross_downstream([t1, t2, t3], [t4, t5, t6])`
   
   ```
   t1   \ /  -> t4
   \ /
   t2 -> X   -> t5
   / \
   t3   /  \ -> t6
   ```
   
   make our dependent more possibilities.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb opened a new pull request #5063: [AIRFLOW-XXX] Ignore python files under node_modules when building docs