[GitHub] [airflow] OmerJog commented on a change in pull request #2256: [AIRFLOW-1153] Fix issue ' params ' don't pass to HiveOperator execution context

2019-04-10 Thread GitBox
OmerJog commented on a change in pull request #2256: [AIRFLOW-1153] Fix issue ' 
params ' don't pass to HiveOperator execution context
URL: https://github.com/apache/airflow/pull/2256#discussion_r274266250
 
 

 ##
 File path: airflow/utils/operator_helpers.py
 ##
 @@ -43,7 +43,8 @@ def context_to_airflow_vars(context, 
in_env_var_format=False):
 :type in_env_var_format: bool
 :return task_instance context as dict.
 """
-params = dict()
+params = context.get('params')
 
 Review comment:
   @xianping 
   If the code you want to test is of operator_helpers.py then the test needs 
to be in:
   
https://github.com/apache/airflow/blob/master/tests/utils/test_operator_helpers.py


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] OmerJog commented on issue #4439: [AIRFLOW-3626] Fixed zipped Dag trigger

2019-04-10 Thread GitBox
OmerJog commented on issue #4439: [AIRFLOW-3626] Fixed zipped Dag trigger
URL: https://github.com/apache/airflow/pull/4439#issuecomment-481970823
 
 
   @shubhparekh can you rebase?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] XD-DENG commented on a change in pull request #5077: [AIRFLOW-4268] Add MsSqlToGoogleCloudStorageOperator

2019-04-10 Thread GitBox
XD-DENG commented on a change in pull request #5077: [AIRFLOW-4268] Add 
MsSqlToGoogleCloudStorageOperator
URL: https://github.com/apache/airflow/pull/5077#discussion_r274248794
 
 

 ##
 File path: airflow/contrib/operators/mssql_to_gcs.py
 ##
 @@ -0,0 +1,224 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import sys
+import json
+import decimal
+
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+from airflow.contrib.hooks.gcs_hook import GoogleCloudStorageHook
+from airflow.hooks.mssql_hook import MsSqlHook
+from tempfile import NamedTemporaryFile
+
+PY3 = sys.version_info[0] == 3
+
+
+class DecimalEncoder(json.JSONEncoder):
+def default(self, o):
+if isinstance(o, decimal.Decimal):
+return float(o)
+return super(DecimalEncoder, self).default(o)
+
+
+class MsSqlToGoogleCloudStorageOperator(BaseOperator):
+"""
+Copy data from Microsoft SQL Server to Google Cloud Storage
+in JSON format.
+"""
+template_fields = ('sql', 'bucket', 'filename', 'schema_filename')
+template_ext = ('.sql',)
+
+ui_color = '#e0a98c'
+
+@apply_defaults
+def __init__(self,
+ sql,
+ bucket,
+ filename,
+ schema_filename=None,
+ approx_max_file_size_bytes=19,
+ mssql_conn_id='mssql_default',
+ google_cloud_storage_conn_id='google_cloud_default',
+ delegate_to=None,
+ *args,
+ **kwargs):
+"""
+:param sql: The SQL to execute on the MSSQL table.
+:type sql: str
+:param bucket: The bucket to upload to.
+:type bucket: str
+:param filename: The filename to use as the object name when uploading
+to Google Cloud Storage. A {} should be specified in the filename
+to allow the operator to inject file numbers in cases where the
+file is split due to size.
+:type filename: str
+:param schema_filename: If set, the filename to use as the object name
+when uploading a .json file containing the BigQuery schema fields
+for the table that was dumped from MSSQL.
+:type schema_filename: str
+:param approx_max_file_size_bytes: This operator supports the ability
+to split large table dumps into multiple files. Google Cloud 
Storage
+allows for files to be a maximum of 4GB. This param allows
+developers to specify the file size of the splits.
+:type approx_max_file_size_bytes: long
+:param mssql_conn_id: Reference to a specific MSSQL hook.
+:type mssql_conn_id: str
+:param google_cloud_storage_conn_id: Reference to a specific Google
+cloud storage hook.
+:type google_cloud_storage_conn_id: str
+:param delegate_to: The account to impersonate, if any. For this to
+work, the service account making the request must have domain-wide
+delegation enabled.
 
 Review comment:
   type of `delegate_to` is missing here. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] XD-DENG commented on a change in pull request #5077: [AIRFLOW-4268] Add MsSqlToGoogleCloudStorageOperator

2019-04-10 Thread GitBox
XD-DENG commented on a change in pull request #5077: [AIRFLOW-4268] Add 
MsSqlToGoogleCloudStorageOperator
URL: https://github.com/apache/airflow/pull/5077#discussion_r274247711
 
 

 ##
 File path: tests/contrib/operators/test_mssql_to_gcs_operator.py
 ##
 @@ -0,0 +1,155 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import sys
+import unittest
+
+from airflow.contrib.operators.mssql_to_gcs import \
+MsSqlToGoogleCloudStorageOperator
+from tests.compat import mock
+
+PY3 = sys.version_info[0] == 3
 
 Review comment:
   In master branch we already dropped support to Python 2. So no need to have 
this (and the related lines)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] mans2singh commented on issue #4967: [AIRFLOW-4147] Operator to publish event to Redis

2019-04-10 Thread GitBox
mans2singh commented on issue #4967: [AIRFLOW-4147] Operator to publish event 
to Redis
URL: https://github.com/apache/airflow/pull/4967#issuecomment-481949904
 
 
   @Fokko @mik-laj @XD-DENG - Please let me know if you have any additional 
review recommendations for this PR.  Mans


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] mans2singh commented on issue #4887: [AIRFLOW-4055] Add AWS SQS Sensor

2019-04-10 Thread GitBox
mans2singh commented on issue #4887: [AIRFLOW-4055] Add AWS SQS Sensor
URL: https://github.com/apache/airflow/pull/4887#issuecomment-481949651
 
 
   @ashb @Fokko @mik-laj @XD-DENG - 
   
   Please let me know if you have any advice/recommendations for this request.  
   
   Thanks


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Commented] (AIRFLOW-4285) Current TI DEPS are confusing and not up to date

2019-04-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/AIRFLOW-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815044#comment-16815044
 ] 

ASF GitHub Bot commented on AIRFLOW-4285:
-

KevinYang21 commented on pull request #5079: [WIP][AIRFLOW-4285] Update task 
dependency context defination and usage
URL: https://github.com/apache/airflow/pull/5079
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [ ] Passes `flake8`
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Current TI DEPS are confusing and not up to date
> 
>
> Key: AIRFLOW-4285
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4285
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Kevin Yang
>Assignee: Kevin Yang
>Priority: Minor
>
> The name of the TI DEPS are not representative, not used efficiently and not 
> up to date.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[GitHub] [airflow] KevinYang21 opened a new pull request #5079: [WIP][AIRFLOW-4285] Update task dependency context defination and usage

2019-04-10 Thread GitBox
KevinYang21 opened a new pull request #5079: [WIP][AIRFLOW-4285] Update task 
dependency context defination and usage
URL: https://github.com/apache/airflow/pull/5079
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [ ] Passes `flake8`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Assigned] (AIRFLOW-4285) Current TI DEPS are confusing and not up to date

2019-04-10 Thread Kevin Yang (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Yang reassigned AIRFLOW-4285:
---

Assignee: Kevin Yang

> Current TI DEPS are confusing and not up to date
> 
>
> Key: AIRFLOW-4285
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4285
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Kevin Yang
>Assignee: Kevin Yang
>Priority: Minor
>
> The name of the TI DEPS are not representative, not used efficiently and not 
> up to date.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (AIRFLOW-4285) Current TI DEPS are confusing and not up to date

2019-04-10 Thread Kevin Yang (JIRA)
Kevin Yang created AIRFLOW-4285:
---

 Summary: Current TI DEPS are confusing and not up to date
 Key: AIRFLOW-4285
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4285
 Project: Apache Airflow
  Issue Type: Improvement
Reporter: Kevin Yang


The name of the TI DEPS are not representative, not used efficiently and not up 
to date.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[GitHub] [airflow] bryanyang0528 edited a comment on issue #5067: [AIRFLOW-4265] Lineage backend did not work normally

2019-04-10 Thread GitBox
bryanyang0528 edited a comment on issue #5067: [AIRFLOW-4265] Lineage backend 
did not work normally
URL: https://github.com/apache/airflow/pull/5067#issuecomment-481934288
 
 
   @NielsZeilemaker 
   Could I initiate the backend in the `_get_backend()` function and return a 
backend instance?
   
https://github.com/bryanyang0528/airflow/blob/d87fb873cdc27350324fd814bb97488dba1f126b/airflow/lineage/__init__.py#L45
   
   Or turn the interface/super class of the AtlasBackend in a staticmethod? 
Because `send_lineage` didn't use `self` as a parameter and refer to any 
resource in the class.
   
   @feng-tao Do you have any opinion or suggestion?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] bryanyang0528 removed a comment on issue #5067: [AIRFLOW-4265] Lineage backend did not work normally

2019-04-10 Thread GitBox
bryanyang0528 removed a comment on issue #5067: [AIRFLOW-4265] Lineage backend 
did not work normally
URL: https://github.com/apache/airflow/pull/5067#issuecomment-481575296
 
 
   @NielsZeilemaker  OK. I'll create another PR for this. Thx


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] bryanyang0528 edited a comment on issue #5067: [AIRFLOW-4265] Lineage backend did not work normally

2019-04-10 Thread GitBox
bryanyang0528 edited a comment on issue #5067: [AIRFLOW-4265] Lineage backend 
did not work normally
URL: https://github.com/apache/airflow/pull/5067#issuecomment-481934288
 
 
   @NielsZeilemaker 
   Could I initiate the backend in the `_get_backend()` function and return a 
backend instance?
   
https://github.com/bryanyang0528/airflow/blob/d87fb873cdc27350324fd814bb97488dba1f126b/airflow/lineage/__init__.py#L45
   
   @feng-tao Do you have any opinion or suggestion?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] jmcarp commented on issue #5030: [AIRFLOW-4227] Use python3-style type annotations.

2019-04-10 Thread GitBox
jmcarp commented on issue #5030: [AIRFLOW-4227] Use python3-style type 
annotations.
URL: https://github.com/apache/airflow/pull/5030#issuecomment-48189
 
 
   Added some typed namedtuples. Since we're supporting python3.5, we can't use 
variable annotations, so I used the backwards-compatible syntax.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] newtonle commented on issue #3956: [AIRFLOW-3123] Allow nested use of DAG as a context manager

2019-04-10 Thread GitBox
newtonle commented on issue #3956: [AIRFLOW-3123] Allow nested use of  DAG as a 
context manager
URL: https://github.com/apache/airflow/pull/3956#issuecomment-481837428
 
 
   > @newtonle curious to know why you'd need to nest this context manager?
   
   We've built some constructs that can build DAGs, often using the DAG context 
manager. The problem is that user can use these constructs within their own DAG 
definition files, often inside another context manager. The problem with having 
only the last context manager saved is that entering a second one will lose 
history of the first. The stack-based approach makes this more robust with 
minimal additional overhead.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] bcb edited a comment on issue #3956: [AIRFLOW-3123] Allow nested use of DAG as a context manager

2019-04-10 Thread GitBox
bcb edited a comment on issue #3956: [AIRFLOW-3123] Allow nested use of  DAG as 
a context manager
URL: https://github.com/apache/airflow/pull/3956#issuecomment-481830421
 
 
   @newtonle curious to know why you'd need to nest this context manager?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] bcb commented on issue #3956: [AIRFLOW-3123] Allow nested use of DAG as a context manager

2019-04-10 Thread GitBox
bcb commented on issue #3956: [AIRFLOW-3123] Allow nested use of  DAG as a 
context manager
URL: https://github.com/apache/airflow/pull/3956#issuecomment-481830421
 
 
   @newtonie curious to know why you'd need to nest this context manager?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] oxymor0n commented on issue #4390: [AIRFLOW-3584] Use ORM DAGs for index view.

2019-04-10 Thread GitBox
oxymor0n commented on issue #4390: [AIRFLOW-3584] Use ORM DAGs for index view.
URL: https://github.com/apache/airflow/pull/4390#issuecomment-481828025
 
 
   hey @jmcarp , just came by a error when I was trying to downgrade from 
1.10.3b1 to regular 1.10 because the down migration for 
`dd4ecb8fbee3_add_schedule_interval_to_dag` has a bug. 
   
   
https://github.com/apache/airflow/blob/c4a7d2bc38175a747e09c56b63f0b2de94d681bc/airflow/migrations/versions/dd4ecb8fbee3_add_schedule_interval_to_dag.py#L42
 should be
   
   `op.drop_column('dag', 'schedule_interval')`
   
   figured you could fix it much faster than I can


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] oxymor0n edited a comment on issue #4390: [AIRFLOW-3584] Use ORM DAGs for index view.

2019-04-10 Thread GitBox
oxymor0n edited a comment on issue #4390: [AIRFLOW-3584] Use ORM DAGs for index 
view.
URL: https://github.com/apache/airflow/pull/4390#issuecomment-481828025
 
 
   hey @jmcarp , just came by a error when I was trying to downgrade from 
1.10.3b1 to regular 1.10 because the down migration for 
`dd4ecb8fbee3_add_schedule_interval_to_dag` has a bug. 
   
   
https://github.com/apache/airflow/blob/c4a7d2bc38175a747e09c56b63f0b2de94d681bc/airflow/migrations/versions/dd4ecb8fbee3_add_schedule_interval_to_dag.py#L42
 should be
   `op.drop_column('dag', 'schedule_interval')`
   
   figured you could fix it much faster than I can


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] yishan-lin commented on issue #2708: [AIRFLOW-1746] Add a Nomad operator to trigger job from Airflow

2019-04-10 Thread GitBox
yishan-lin commented on issue #2708: [AIRFLOW-1746] Add a Nomad operator to 
trigger job from Airflow
URL: https://github.com/apache/airflow/pull/2708#issuecomment-481824561
 
 
   Hey folks!  Writing from the Nomad team.  What can we do to shepherd this 
through?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


svn commit: r33552 - /release/airflow/1.10.3/

2019-04-10 Thread ash
Author: ash
Date: Wed Apr 10 17:43:52 2019
New Revision: 33552

Log:
Airflow 1.10.3 release from RC2

Added:
release/airflow/1.10.3/
release/airflow/1.10.3/apache-airflow-1.10.3-bin.tar.gz
  - copied unchanged from r33494, 
dev/airflow/1.10.3rc2/apache-airflow-1.10.3rc2-bin.tar.gz
release/airflow/1.10.3/apache-airflow-1.10.3-bin.tar.gz.asc
  - copied unchanged from r33494, 
dev/airflow/1.10.3rc2/apache-airflow-1.10.3rc2-bin.tar.gz.asc
release/airflow/1.10.3/apache-airflow-1.10.3-bin.tar.gz.sha512
  - copied unchanged from r33494, 
dev/airflow/1.10.3rc2/apache-airflow-1.10.3rc2-bin.tar.gz.sha512
release/airflow/1.10.3/apache-airflow-1.10.3-source.tar.gz
  - copied unchanged from r33494, 
dev/airflow/1.10.3rc2/apache-airflow-1.10.3rc2-source.tar.gz
release/airflow/1.10.3/apache-airflow-1.10.3-source.tar.gz.asc
  - copied unchanged from r33494, 
dev/airflow/1.10.3rc2/apache-airflow-1.10.3rc2-source.tar.gz.asc
release/airflow/1.10.3/apache-airflow-1.10.3-source.tar.gz.sha512
  - copied unchanged from r33494, 
dev/airflow/1.10.3rc2/apache-airflow-1.10.3rc2-source.tar.gz.sha512
release/airflow/1.10.3/apache_airflow-1.10.3-py2.py3-none-any.whl
  - copied unchanged from r33494, 
dev/airflow/1.10.3rc2/apache_airflow-1.10.3rc2-py2.py3-none-any.whl
release/airflow/1.10.3/apache_airflow-1.10.3-py2.py3-none-any.whl.asc
  - copied unchanged from r33494, 
dev/airflow/1.10.3rc2/apache_airflow-1.10.3rc2-py2.py3-none-any.whl.asc
release/airflow/1.10.3/apache_airflow-1.10.3-py2.py3-none-any.whl.sha512
  - copied unchanged from r33494, 
dev/airflow/1.10.3rc2/apache_airflow-1.10.3rc2-py2.py3-none-any.whl.sha512



[jira] [Updated] (AIRFLOW-4278) Referential constraint: sla_miss.dag_id = dag.dag_id

2019-04-10 Thread Fokko Driesprong (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-4278:
--
Parent: AIRFLOW-3904  (was: AIRFLOW-4270)

> Referential constraint: sla_miss.dag_id = dag.dag_id
> 
>
> Key: AIRFLOW-4278
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4278
> Project: Apache Airflow
>  Issue Type: Sub-task
>Reporter: Fokko Driesprong
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (AIRFLOW-4273) Referential constraint: dag.dag_id = dag_stats.dag_id

2019-04-10 Thread Fokko Driesprong (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-4273:
--
Parent: AIRFLOW-3904  (was: AIRFLOW-4270)

> Referential constraint: dag.dag_id = dag_stats.dag_id
> -
>
> Key: AIRFLOW-4273
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4273
> Project: Apache Airflow
>  Issue Type: Sub-task
>Reporter: Fokko Driesprong
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (AIRFLOW-4280) Referential constraint: dag.dag_id = task_fail.dag_id

2019-04-10 Thread Fokko Driesprong (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-4280:
--
Parent: AIRFLOW-3904  (was: AIRFLOW-4270)

> Referential constraint: dag.dag_id = task_fail.dag_id
> -
>
> Key: AIRFLOW-4280
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4280
> Project: Apache Airflow
>  Issue Type: Sub-task
>Reporter: Fokko Driesprong
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (AIRFLOW-4283) Referential constraint: xcom.task_id = task_instance.task_id

2019-04-10 Thread Fokko Driesprong (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-4283:
--
Parent: AIRFLOW-3904  (was: AIRFLOW-4270)

> Referential constraint: xcom.task_id = task_instance.task_id
> 
>
> Key: AIRFLOW-4283
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4283
> Project: Apache Airflow
>  Issue Type: Sub-task
>Reporter: Fokko Driesprong
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (AIRFLOW-4284) Referential constraint: xcom.dag_id = dag.dag_id

2019-04-10 Thread Fokko Driesprong (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-4284:
--
Parent: AIRFLOW-3904  (was: AIRFLOW-4270)

> Referential constraint: xcom.dag_id = dag.dag_id
> 
>
> Key: AIRFLOW-4284
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4284
> Project: Apache Airflow
>  Issue Type: Sub-task
>Reporter: Fokko Driesprong
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (AIRFLOW-4272) Referential constraint: dag.dag_id = dag_run.dag_id

2019-04-10 Thread Fokko Driesprong (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-4272:
--
Parent: AIRFLOW-3904  (was: AIRFLOW-4270)

> Referential constraint: dag.dag_id = dag_run.dag_id
> ---
>
> Key: AIRFLOW-4272
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4272
> Project: Apache Airflow
>  Issue Type: Sub-task
>Reporter: Fokko Driesprong
>Priority: Major
>
> This is a tricky one



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (AIRFLOW-4279) Referential constraint: task_fail.task_id = task_instance.task_id

2019-04-10 Thread Fokko Driesprong (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-4279:
--
Parent: AIRFLOW-3904  (was: AIRFLOW-4270)

> Referential constraint: task_fail.task_id = task_instance.task_id
> -
>
> Key: AIRFLOW-4279
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4279
> Project: Apache Airflow
>  Issue Type: Sub-task
>Reporter: Fokko Driesprong
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (AIRFLOW-4282) Referential constraint: pool.pool = task_instance.pool

2019-04-10 Thread Fokko Driesprong (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-4282:
--
Parent: AIRFLOW-3904  (was: AIRFLOW-4270)

> Referential constraint: pool.pool = task_instance.pool
> --
>
> Key: AIRFLOW-4282
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4282
> Project: Apache Airflow
>  Issue Type: Sub-task
>Reporter: Fokko Driesprong
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (AIRFLOW-4275) Referential constraint: dag.dag_id = log.dag_id

2019-04-10 Thread Fokko Driesprong (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-4275:
--
Parent: AIRFLOW-3904  (was: AIRFLOW-4270)

> Referential constraint: dag.dag_id = log.dag_id
> ---
>
> Key: AIRFLOW-4275
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4275
> Project: Apache Airflow
>  Issue Type: Sub-task
>Reporter: Fokko Driesprong
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (AIRFLOW-4281) Referential constraint: task_instance.job_id = job.id

2019-04-10 Thread Fokko Driesprong (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-4281:
--
Parent: AIRFLOW-3904  (was: AIRFLOW-4270)

> Referential constraint: task_instance.job_id = job.id
> -
>
> Key: AIRFLOW-4281
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4281
> Project: Apache Airflow
>  Issue Type: Sub-task
>Reporter: Fokko Driesprong
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Closed] (AIRFLOW-4270) Create referential constraints in the database

2019-04-10 Thread Fokko Driesprong (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong closed AIRFLOW-4270.
-
Resolution: Duplicate

> Create referential constraints in the database
> --
>
> Key: AIRFLOW-4270
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4270
> Project: Apache Airflow
>  Issue Type: Task
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (AIRFLOW-4277) Referential constraint: sla_miss.task_id = task_instance.task_id

2019-04-10 Thread Fokko Driesprong (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-4277:
--
Parent: AIRFLOW-3904  (was: AIRFLOW-4270)

> Referential constraint: sla_miss.task_id = task_instance.task_id
> 
>
> Key: AIRFLOW-4277
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4277
> Project: Apache Airflow
>  Issue Type: Sub-task
>Reporter: Fokko Driesprong
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (AIRFLOW-4276) Referential constraint: log.task_id = task_instance.task_id

2019-04-10 Thread Fokko Driesprong (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-4276:
--
Parent: AIRFLOW-3904  (was: AIRFLOW-4270)

> Referential constraint: log.task_id = task_instance.task_id
> ---
>
> Key: AIRFLOW-4276
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4276
> Project: Apache Airflow
>  Issue Type: Sub-task
>Reporter: Fokko Driesprong
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (AIRFLOW-4274) Referential constraint: dag.dag_id = job.dag_id

2019-04-10 Thread Fokko Driesprong (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong updated AIRFLOW-4274:
--
Parent: AIRFLOW-3904  (was: AIRFLOW-4270)

> Referential constraint: dag.dag_id = job.dag_id
> ---
>
> Key: AIRFLOW-4274
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4274
> Project: Apache Airflow
>  Issue Type: Sub-task
>Reporter: Fokko Driesprong
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (AIRFLOW-4273) Referential constraint: dag.dag_id = dag_stats.dag_id

2019-04-10 Thread Fokko Driesprong (JIRA)
Fokko Driesprong created AIRFLOW-4273:
-

 Summary: Referential constraint: dag.dag_id = dag_stats.dag_id
 Key: AIRFLOW-4273
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4273
 Project: Apache Airflow
  Issue Type: Sub-task
Reporter: Fokko Driesprong






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Comment Edited] (AIRFLOW-4270) Create referential constraints in the database

2019-04-10 Thread Fokko Driesprong (JIRA)


[ 
https://issues.apache.org/jira/browse/AIRFLOW-4270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814666#comment-16814666
 ] 

Fokko Driesprong edited comment on AIRFLOW-4270 at 4/10/19 5:05 PM:


Dang, quick one [~ash]. Thanks!


was (Author: fokko):
Dang, quick one [~ash]

> Create referential constraints in the database
> --
>
> Key: AIRFLOW-4270
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4270
> Project: Apache Airflow
>  Issue Type: Task
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (AIRFLOW-4270) Create referential constraints in the database

2019-04-10 Thread Fokko Driesprong (JIRA)


[ 
https://issues.apache.org/jira/browse/AIRFLOW-4270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814666#comment-16814666
 ] 

Fokko Driesprong commented on AIRFLOW-4270:
---

Dang, quick one [~ash]

> Create referential constraints in the database
> --
>
> Key: AIRFLOW-4270
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4270
> Project: Apache Airflow
>  Issue Type: Task
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (AIRFLOW-4284) Referential constraint: xcom.dag_id = dag.dag_id

2019-04-10 Thread Fokko Driesprong (JIRA)
Fokko Driesprong created AIRFLOW-4284:
-

 Summary: Referential constraint: xcom.dag_id = dag.dag_id
 Key: AIRFLOW-4284
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4284
 Project: Apache Airflow
  Issue Type: Sub-task
Reporter: Fokko Driesprong






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (AIRFLOW-4283) Referential constraint: xcom.task_id = task_instance.task_id

2019-04-10 Thread Fokko Driesprong (JIRA)
Fokko Driesprong created AIRFLOW-4283:
-

 Summary: Referential constraint: xcom.task_id = 
task_instance.task_id
 Key: AIRFLOW-4283
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4283
 Project: Apache Airflow
  Issue Type: Sub-task
Reporter: Fokko Driesprong






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (AIRFLOW-4281) Referential constraint: task_instance.job_id = job.id

2019-04-10 Thread Fokko Driesprong (JIRA)
Fokko Driesprong created AIRFLOW-4281:
-

 Summary: Referential constraint: task_instance.job_id = job.id
 Key: AIRFLOW-4281
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4281
 Project: Apache Airflow
  Issue Type: Sub-task
Reporter: Fokko Driesprong






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (AIRFLOW-4282) Referential constraint: pool.pool = task_instance.pool

2019-04-10 Thread Fokko Driesprong (JIRA)
Fokko Driesprong created AIRFLOW-4282:
-

 Summary: Referential constraint: pool.pool = task_instance.pool
 Key: AIRFLOW-4282
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4282
 Project: Apache Airflow
  Issue Type: Sub-task
Reporter: Fokko Driesprong






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (AIRFLOW-4280) Referential constraint: dag.dag_id = task_fail.dag_id

2019-04-10 Thread Fokko Driesprong (JIRA)
Fokko Driesprong created AIRFLOW-4280:
-

 Summary: Referential constraint: dag.dag_id = task_fail.dag_id
 Key: AIRFLOW-4280
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4280
 Project: Apache Airflow
  Issue Type: Sub-task
Reporter: Fokko Driesprong






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[GitHub] [airflow] ashb merged pull request #5078: [AIRFLOW-XXX] Use Py3.7 on readthedocs to fix OOM

2019-04-10 Thread GitBox
ashb merged pull request #5078: [AIRFLOW-XXX] Use Py3.7 on readthedocs to fix 
OOM
URL: https://github.com/apache/airflow/pull/5078
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Created] (AIRFLOW-4278) Referential constraint: sla_miss.dag_id = dag.dag_id

2019-04-10 Thread Fokko Driesprong (JIRA)
Fokko Driesprong created AIRFLOW-4278:
-

 Summary: Referential constraint: sla_miss.dag_id = dag.dag_id
 Key: AIRFLOW-4278
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4278
 Project: Apache Airflow
  Issue Type: Sub-task
Reporter: Fokko Driesprong






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (AIRFLOW-4279) Referential constraint: task_fail.task_id = task_instance.task_id

2019-04-10 Thread Fokko Driesprong (JIRA)
Fokko Driesprong created AIRFLOW-4279:
-

 Summary: Referential constraint: task_fail.task_id = 
task_instance.task_id
 Key: AIRFLOW-4279
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4279
 Project: Apache Airflow
  Issue Type: Sub-task
Reporter: Fokko Driesprong






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[GitHub] [airflow] codecov-io commented on issue #5078: [AIRFLOW-XXX] Use Py3.7 on readthedocs to fix OOM

2019-04-10 Thread GitBox
codecov-io commented on issue #5078: [AIRFLOW-XXX] Use Py3.7 on readthedocs to 
fix OOM
URL: https://github.com/apache/airflow/pull/5078#issuecomment-481775709
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5078?src=pr=h1) 
Report
   > Merging 
[#5078](https://codecov.io/gh/apache/airflow/pull/5078?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/64331aa5b90c44a3a22631d0ed168c44842a1fc8?src=pr=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5078/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5078?src=pr=tree)
   
   ```diff
   @@   Coverage Diff   @@
   ##   master#5078   +/-   ##
   ===
 Coverage   77.01%   77.01%   
   ===
 Files 463  463   
 Lines   2975329753   
   ===
 Hits2291522915   
 Misses   6838 6838
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5078?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5078?src=pr=footer). 
Last update 
[64331aa...1efc648](https://codecov.io/gh/apache/airflow/pull/5078?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Commented] (AIRFLOW-4270) Create referential constraints in the database

2019-04-10 Thread Ash Berlin-Taylor (JIRA)


[ 
https://issues.apache.org/jira/browse/AIRFLOW-4270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814655#comment-16814655
 ] 

Ash Berlin-Taylor commented on AIRFLOW-4270:


Dupe of AIRFLOW-3904

> Create referential constraints in the database
> --
>
> Key: AIRFLOW-4270
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4270
> Project: Apache Airflow
>  Issue Type: Task
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (AIRFLOW-4275) Referential constraint: dag.dag_id = log.dag_id

2019-04-10 Thread Fokko Driesprong (JIRA)
Fokko Driesprong created AIRFLOW-4275:
-

 Summary: Referential constraint: dag.dag_id = log.dag_id
 Key: AIRFLOW-4275
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4275
 Project: Apache Airflow
  Issue Type: Sub-task
Reporter: Fokko Driesprong






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (AIRFLOW-4276) Referential constraint: log.task_id = task_instance.task_id

2019-04-10 Thread Fokko Driesprong (JIRA)
Fokko Driesprong created AIRFLOW-4276:
-

 Summary: Referential constraint: log.task_id = 
task_instance.task_id
 Key: AIRFLOW-4276
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4276
 Project: Apache Airflow
  Issue Type: Sub-task
Reporter: Fokko Driesprong






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (AIRFLOW-4277) Referential constraint: sla_miss.task_id = task_instance.task_id

2019-04-10 Thread Fokko Driesprong (JIRA)
Fokko Driesprong created AIRFLOW-4277:
-

 Summary: Referential constraint: sla_miss.task_id = 
task_instance.task_id
 Key: AIRFLOW-4277
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4277
 Project: Apache Airflow
  Issue Type: Sub-task
Reporter: Fokko Driesprong






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (AIRFLOW-4274) Referential constraint: dag.dag_id = job.dag_id

2019-04-10 Thread Fokko Driesprong (JIRA)
Fokko Driesprong created AIRFLOW-4274:
-

 Summary: Referential constraint: dag.dag_id = job.dag_id
 Key: AIRFLOW-4274
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4274
 Project: Apache Airflow
  Issue Type: Sub-task
Reporter: Fokko Driesprong






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (AIRFLOW-4272) Referential constraint: dag.dag_id = dag_run.dag_id

2019-04-10 Thread Fokko Driesprong (JIRA)
Fokko Driesprong created AIRFLOW-4272:
-

 Summary: Referential constraint: dag.dag_id = dag_run.dag_id
 Key: AIRFLOW-4272
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4272
 Project: Apache Airflow
  Issue Type: Sub-task
Reporter: Fokko Driesprong


This is a tricky one



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (AIRFLOW-4271) Referential constraint: dag.pickle_id = dag_pickle.id

2019-04-10 Thread Fokko Driesprong (JIRA)
Fokko Driesprong created AIRFLOW-4271:
-

 Summary: Referential constraint: dag.pickle_id = dag_pickle.id
 Key: AIRFLOW-4271
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4271
 Project: Apache Airflow
  Issue Type: Sub-task
Reporter: Fokko Driesprong






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (AIRFLOW-4270) Create referential constraints in the database

2019-04-10 Thread Fokko Driesprong (JIRA)
Fokko Driesprong created AIRFLOW-4270:
-

 Summary: Create referential constraints in the database
 Key: AIRFLOW-4270
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4270
 Project: Apache Airflow
  Issue Type: Task
Reporter: Fokko Driesprong
Assignee: Fokko Driesprong






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[GitHub] [airflow] codecov-io commented on issue #5076: [AIRFLOW-4269] Minor acceleration of jobs._process_task_instances()

2019-04-10 Thread GitBox
codecov-io commented on issue #5076: [AIRFLOW-4269] Minor acceleration of 
jobs._process_task_instances()
URL: https://github.com/apache/airflow/pull/5076#issuecomment-481761247
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5076?src=pr=h1) 
Report
   > Merging 
[#5076](https://codecov.io/gh/apache/airflow/pull/5076?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/64331aa5b90c44a3a22631d0ed168c44842a1fc8?src=pr=desc)
 will **not change** coverage.
   > The diff coverage is `0%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5076/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5076?src=pr=tree)
   
   ```diff
   @@   Coverage Diff   @@
   ##   master#5076   +/-   ##
   ===
 Coverage   77.01%   77.01%   
   ===
 Files 463  463   
 Lines   2975329753   
   ===
 Hits2291522915   
 Misses   6838 6838
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/5076?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/jobs.py](https://codecov.io/gh/apache/airflow/pull/5076/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5)
 | `78.8% <0%> (ø)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5076?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5076?src=pr=footer). 
Last update 
[64331aa...eb3bbc4](https://codecov.io/gh/apache/airflow/pull/5076?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] codecov-io edited a comment on issue #5077: [AIRFLOW-4268] Add MsSqlToGoogleCloudStorageOperator

2019-04-10 Thread GitBox
codecov-io edited a comment on issue #5077: [AIRFLOW-4268] Add 
MsSqlToGoogleCloudStorageOperator
URL: https://github.com/apache/airflow/pull/5077#issuecomment-481760912
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5077?src=pr=h1) 
Report
   > Merging 
[#5077](https://codecov.io/gh/apache/airflow/pull/5077?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/64331aa5b90c44a3a22631d0ed168c44842a1fc8?src=pr=desc)
 will **increase** coverage by `0.05%`.
   > The diff coverage is `96.51%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5077/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5077?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#5077  +/-   ##
   ==
   + Coverage   77.01%   77.07%   +0.05% 
   ==
 Files 463  464   +1 
 Lines   2975329839  +86 
   ==
   + Hits2291522997  +82 
   - Misses   6838 6842   +4
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/5077?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/contrib/operators/mssql\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/5077/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9tc3NxbF90b19nY3MucHk=)
 | `96.51% <96.51%> (ø)` | |
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/5077/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `92.42% <0%> (-0.18%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5077?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5077?src=pr=footer). 
Last update 
[64331aa...6d0b816](https://codecov.io/gh/apache/airflow/pull/5077?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] codecov-io commented on issue #5077: [AIRFLOW-4268] Add MsSqlToGoogleCloudStorageOperator

2019-04-10 Thread GitBox
codecov-io commented on issue #5077: [AIRFLOW-4268] Add 
MsSqlToGoogleCloudStorageOperator
URL: https://github.com/apache/airflow/pull/5077#issuecomment-481760912
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5077?src=pr=h1) 
Report
   > Merging 
[#5077](https://codecov.io/gh/apache/airflow/pull/5077?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/64331aa5b90c44a3a22631d0ed168c44842a1fc8?src=pr=desc)
 will **increase** coverage by `0.05%`.
   > The diff coverage is `96.51%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5077/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5077?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#5077  +/-   ##
   ==
   + Coverage   77.01%   77.07%   +0.05% 
   ==
 Files 463  464   +1 
 Lines   2975329839  +86 
   ==
   + Hits2291522997  +82 
   - Misses   6838 6842   +4
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/5077?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/contrib/operators/mssql\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/5077/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9tc3NxbF90b19nY3MucHk=)
 | `96.51% <96.51%> (ø)` | |
   | 
[airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/5077/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5)
 | `92.42% <0%> (-0.18%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5077?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5077?src=pr=footer). 
Last update 
[64331aa...6d0b816](https://codecov.io/gh/apache/airflow/pull/5077?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] ashb merged pull request #5075: [AIRFLOW-XXX] Change allowed version of Jinja2 to fix CVE-2019-10906

2019-04-10 Thread GitBox
ashb merged pull request #5075: [AIRFLOW-XXX] Change allowed version of Jinja2 
to fix CVE-2019-10906
URL: https://github.com/apache/airflow/pull/5075
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] ashb opened a new pull request #5078: [AIRFLOW-XXX] Use Py3.7 on readthedocs to fix OOM

2019-04-10 Thread GitBox
ashb opened a new pull request #5078: [AIRFLOW-XXX] Use Py3.7 on readthedocs to 
fix OOM
URL: https://github.com/apache/airflow/pull/5078
 
 
   ### Jira
   
   - [x] No Jira
   
   ### Description
   
   - [x] Previous builds were failing with "Command killed due to excessive
   memory consumption".
   
   This updated the RTD config, removes the un-needed extras (which we have
   mocked for a while now), switches to building on Py3.7. This seems to
   help the docs build again
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Commented] (AIRFLOW-4268) Add MsSqlToGoogleCloudStorageOperator

2019-04-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/AIRFLOW-4268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814546#comment-16814546
 ] 

ASF GitHub Bot commented on AIRFLOW-4268:
-

Tomme commented on pull request #5077: [AIRFLOW-4268] Add 
MsSqlToGoogleCloudStorageOperator
URL: https://github.com/apache/airflow/pull/5077
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following 
[AIRFLOW-4268](https://issues.apache.org/jira/browse/AIRFLOW-4268) issues and 
references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-4268
   
   ### Description
   
   - [x] Added MsSqlToGoogleCloudStorageOperator using MsSqlHook and 
GoogleCloudStorageHook.
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
 - tests/contrib/operators/test_mssql_to_gcs_operator.py
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [x] Passes `flake8`
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add MsSqlToGoogleCloudStorageOperator
> -
>
> Key: AIRFLOW-4268
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4268
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: contrib, operators
>Affects Versions: 1.10.2
>Reporter: Thomas Elvey
>Assignee: Thomas Elvey
>Priority: Minor
>
> Using both 
> [mysql_to_gcs.py|https://github.com/Tomme/airflow/blob/master/airflow/contrib/operators/mysql_to_gcs.py]
>  and 
> [postgres_to_gcs.py|https://github.com/Tomme/airflow/blob/master/airflow/contrib/operators/postgres_to_gcs_operator.py]
>  as a base I have created a new operator "MsSqlToGoogleCloudStorageOperator" 
> that may be of use and to further expand on Airflow's GCP offering.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[GitHub] [airflow] OmerJog commented on issue #5077: [AIRFLOW-4268] Add MsSqlToGoogleCloudStorageOperator

2019-04-10 Thread GitBox
OmerJog commented on issue #5077: [AIRFLOW-4268] Add 
MsSqlToGoogleCloudStorageOperator
URL: https://github.com/apache/airflow/pull/5077#issuecomment-481740191
 
 
   My 2 notes:
   1. I think it's better to support all the relevant argument of the hook. For 
example the operator introduced in this PR doesn't support the gzip flag of the 
 GoogleCloudStorageHook (reference https://github.com/apache/airflow/pull/3893 
) 
   
   2. PR https://github.com/apache/airflow/pull/5043 raised a question about 
the num_retries of the hook/operator. I didn't see the committers agree on 
course of action hopefully it will be done soon and this PR will be updated 
accordingly. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Updated] (AIRFLOW-4269) For-lop in jobs._process_task_instances can be faster by breaking from unnecessary steps

2019-04-10 Thread Xiaodong DENG (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaodong DENG updated AIRFLOW-4269:
---
Summary: For-lop in jobs._process_task_instances can be faster by breaking 
from unnecessary steps  (was: Minor refactoring on 
jobs._process_task_instances())

> For-lop in jobs._process_task_instances can be faster by breaking from 
> unnecessary steps
> 
>
> Key: AIRFLOW-4269
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4269
> Project: Apache Airflow
>  Issue Type: Improvement
>Affects Versions: 1.10.2
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Minor
>
> Issue-1:
> [https://github.com/apache/airflow/blob/64331aa5b90c44a3a22631d0ed168c44842a1fc8/airflow/jobs.py#L923]
>  is a "continue" while it should be a break.
> When *len(active_dag_runs) >= dag.max_active_runs* , the following part of 
> the for-loop will give the same result and result in a "continue" again and 
> again, doing nothing. 
> We should use a "break" here.
>  
> Issue-2:
> [https://github.com/apache/airflow/blob/64331aa5b90c44a3a22631d0ed168c44842a1fc8/airflow/jobs.py#L921]
> *"if len(active_dag_runs) >= dag.max_active_runs:"* should be "*if 
> len(active_dag_runs) > dag.max_active_runs:".*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (AIRFLOW-4269) For-lop in jobs._process_task_instances can be faster by breaking from unnecessary steps

2019-04-10 Thread Xiaodong DENG (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaodong DENG updated AIRFLOW-4269:
---
Description: 
[https://github.com/apache/airflow/blob/64331aa5b90c44a3a22631d0ed168c44842a1fc8/airflow/jobs.py#L923]
 is a "continue" while it should be a break.

When *len(active_dag_runs) >= dag.max_active_runs* , the following part of the 
for-loop will give the same result and result in a "continue" again and again, 
doing nothing. 

We should use a "break" here.

 

  was:
Issue-1:

[https://github.com/apache/airflow/blob/64331aa5b90c44a3a22631d0ed168c44842a1fc8/airflow/jobs.py#L923]
 is a "continue" while it should be a break.

When *len(active_dag_runs) >= dag.max_active_runs* , the following part of the 
for-loop will give the same result and result in a "continue" again and again, 
doing nothing. 

We should use a "break" here.

 

Issue-2:

[https://github.com/apache/airflow/blob/64331aa5b90c44a3a22631d0ed168c44842a1fc8/airflow/jobs.py#L921]

*"if len(active_dag_runs) >= dag.max_active_runs:"* should be "*if 
len(active_dag_runs) > dag.max_active_runs:".*


> For-lop in jobs._process_task_instances can be faster by breaking from 
> unnecessary steps
> 
>
> Key: AIRFLOW-4269
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4269
> Project: Apache Airflow
>  Issue Type: Improvement
>Affects Versions: 1.10.2
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Minor
>
> [https://github.com/apache/airflow/blob/64331aa5b90c44a3a22631d0ed168c44842a1fc8/airflow/jobs.py#L923]
>  is a "continue" while it should be a break.
> When *len(active_dag_runs) >= dag.max_active_runs* , the following part of 
> the for-loop will give the same result and result in a "continue" again and 
> again, doing nothing. 
> We should use a "break" here.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[GitHub] [airflow] Tomme opened a new pull request #5077: [AIRFLOW-4268] Add MsSqlToGoogleCloudStorageOperator

2019-04-10 Thread GitBox
Tomme opened a new pull request #5077: [AIRFLOW-4268] Add 
MsSqlToGoogleCloudStorageOperator
URL: https://github.com/apache/airflow/pull/5077
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following 
[AIRFLOW-4268](https://issues.apache.org/jira/browse/AIRFLOW-4268) issues and 
references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-4268
   
   ### Description
   
   - [x] Added MsSqlToGoogleCloudStorageOperator using MsSqlHook and 
GoogleCloudStorageHook.
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
 - tests/contrib/operators/test_mssql_to_gcs_operator.py
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [x] Passes `flake8`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Updated] (AIRFLOW-4269) Minor refactoring on jobs._process_task_instances()

2019-04-10 Thread Xiaodong DENG (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaodong DENG updated AIRFLOW-4269:
---
Description: 
Issue-1:

[https://github.com/apache/airflow/blob/64331aa5b90c44a3a22631d0ed168c44842a1fc8/airflow/jobs.py#L923]
 is a "continue" while it should be a break.

When *len(active_dag_runs) >= dag.max_active_runs* , the following part of the 
for-loop will give the same result and result in a "continue" again and again, 
doing nothing. 

We should use a "break" here.

 

Issue-2:

[https://github.com/apache/airflow/blob/64331aa5b90c44a3a22631d0ed168c44842a1fc8/airflow/jobs.py#L921]

*"if len(active_dag_runs) >= dag.max_active_runs:"* should be "*if 
len(active_dag_runs) > dag.max_active_runs:".*

  was:
[https://github.com/apache/airflow/blob/64331aa5b90c44a3a22631d0ed168c44842a1fc8/airflow/jobs.py#L923]
 is a "continue" while it should be a break.

When *len(active_dag_runs) >= dag.max_active_runs* , the following part of the 
for-loop will give the same result and result in a "continue" again and again, 
doing nothing. 

We should use a "break" here.


> Minor refactoring on jobs._process_task_instances()
> ---
>
> Key: AIRFLOW-4269
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4269
> Project: Apache Airflow
>  Issue Type: Improvement
>Affects Versions: 1.10.2
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Minor
>
> Issue-1:
> [https://github.com/apache/airflow/blob/64331aa5b90c44a3a22631d0ed168c44842a1fc8/airflow/jobs.py#L923]
>  is a "continue" while it should be a break.
> When *len(active_dag_runs) >= dag.max_active_runs* , the following part of 
> the for-loop will give the same result and result in a "continue" again and 
> again, doing nothing. 
> We should use a "break" here.
>  
> Issue-2:
> [https://github.com/apache/airflow/blob/64331aa5b90c44a3a22631d0ed168c44842a1fc8/airflow/jobs.py#L921]
> *"if len(active_dag_runs) >= dag.max_active_runs:"* should be "*if 
> len(active_dag_runs) > dag.max_active_runs:".*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (AIRFLOW-4269) Minor refactoring on jobs._process_task_instances()

2019-04-10 Thread Xiaodong DENG (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaodong DENG updated AIRFLOW-4269:
---
Summary: Minor refactoring on jobs._process_task_instances()  (was: For-lop 
in jobs._process_task_instances can be faster by breaking from unnecessary 
steps)

> Minor refactoring on jobs._process_task_instances()
> ---
>
> Key: AIRFLOW-4269
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4269
> Project: Apache Airflow
>  Issue Type: Improvement
>Affects Versions: 1.10.2
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Minor
>
> [https://github.com/apache/airflow/blob/64331aa5b90c44a3a22631d0ed168c44842a1fc8/airflow/jobs.py#L923]
>  is a "continue" while it should be a break.
> When *len(active_dag_runs) >= dag.max_active_runs* , the following part of 
> the for-loop will give the same result and result in a "continue" again and 
> again, doing nothing. 
> We should use a "break" here.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[GitHub] [airflow] XD-DENG commented on issue #5076: [AIRFLOW-4269] Minor acceleration of jobs._process_task_instances()

2019-04-10 Thread GitBox
XD-DENG commented on issue #5076: [AIRFLOW-4269] Minor acceleration of 
jobs._process_task_instances()
URL: https://github.com/apache/airflow/pull/5076#issuecomment-481733018
 
 
   Trying going through the scheduler-related codes for improving its 
performance recently, so raising "small" changes for the issues I found along 
the way.
   
   Please bear with the small size of each of my recent PRs ;-)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Commented] (AIRFLOW-4269) For-lop in jobs._process_task_instances can be faster by breaking from unnecessary steps

2019-04-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/AIRFLOW-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814533#comment-16814533
 ] 

ASF GitHub Bot commented on AIRFLOW-4269:
-

XD-DENG commented on pull request #5076: [AIRFLOW-4269] Minor acceleration of 
jobs._process_task_instances()
URL: https://github.com/apache/airflow/pull/5076
 
 
   ### Jira
   
 - https://issues.apache.org/jira/browse/AIRFLOW-4269
   
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   `continue` here should be a `break`.
   
   When `len(active_dag_runs) >= dag.max_active_runs`, the following part of 
the for-loop will give the same result and result in a `continue` again and 
again, while doing nothing. We should use a `break` here.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> For-lop in jobs._process_task_instances can be faster by breaking from 
> unnecessary steps
> 
>
> Key: AIRFLOW-4269
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4269
> Project: Apache Airflow
>  Issue Type: Improvement
>Affects Versions: 1.10.2
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Minor
>
> [https://github.com/apache/airflow/blob/64331aa5b90c44a3a22631d0ed168c44842a1fc8/airflow/jobs.py#L923]
>  is a "continue" while it should be a break.
> When *len(active_dag_runs) >= dag.max_active_runs* , the following part of 
> the for-loop will give the same result and result in a "continue" again and 
> again, doing nothing. 
> We should use a "break" here.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[GitHub] [airflow] XD-DENG opened a new pull request #5076: [AIRFLOW-4269] Minor acceleration of jobs._process_task_instances()

2019-04-10 Thread GitBox
XD-DENG opened a new pull request #5076: [AIRFLOW-4269] Minor acceleration of 
jobs._process_task_instances()
URL: https://github.com/apache/airflow/pull/5076
 
 
   ### Jira
   
 - https://issues.apache.org/jira/browse/AIRFLOW-4269
   
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   `continue` here should be a `break`.
   
   When `len(active_dag_runs) >= dag.max_active_runs`, the following part of 
the for-loop will give the same result and result in a `continue` again and 
again, while doing nothing. We should use a `break` here.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Updated] (AIRFLOW-4269) For-lop in jobs._process_task_instances can be faster by breaking from unnecessary steps

2019-04-10 Thread Xiaodong DENG (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaodong DENG updated AIRFLOW-4269:
---
Affects Version/s: 1.10.2

> For-lop in jobs._process_task_instances can be faster by breaking from 
> unnecessary steps
> 
>
> Key: AIRFLOW-4269
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4269
> Project: Apache Airflow
>  Issue Type: Improvement
>Affects Versions: 1.10.2
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Minor
>
> [https://github.com/apache/airflow/blob/64331aa5b90c44a3a22631d0ed168c44842a1fc8/airflow/jobs.py#L923]
>  is a "continue" while it should be a break.
> When *len(active_dag_runs) >= dag.max_active_runs* , the following part of 
> the for-loop will give the same result and result in a "continue" again and 
> again, doing nothing. 
> We should use a "break" here.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (AIRFLOW-4269) For-lop in jobs._process_task_instances can be faster by breaking from unnecessary steps

2019-04-10 Thread Xiaodong DENG (JIRA)
Xiaodong DENG created AIRFLOW-4269:
--

 Summary: For-lop in jobs._process_task_instances can be faster by 
breaking from unnecessary steps
 Key: AIRFLOW-4269
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4269
 Project: Apache Airflow
  Issue Type: Improvement
Reporter: Xiaodong DENG
Assignee: Xiaodong DENG


[https://github.com/apache/airflow/blob/64331aa5b90c44a3a22631d0ed168c44842a1fc8/airflow/jobs.py#L923]
 is a "continue" while it should be a break.

When *len(active_dag_runs) >= dag.max_active_runs* , the following part of the 
for-loop will give the same result and result in a "continue" again and again, 
doing nothing. 

We should use a "break" here.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (AIRFLOW-4268) Add MsSqlToGoogleCloudStorageOperator

2019-04-10 Thread Thomas Elvey (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Elvey updated AIRFLOW-4268:
--
Description: 
Using both 
[mysql_to_gcs.py|https://github.com/Tomme/airflow/blob/master/airflow/contrib/operators/mysql_to_gcs.py]
 and 
[postgres_to_gcs.py|https://github.com/Tomme/airflow/blob/master/airflow/contrib/operators/postgres_to_gcs_operator.py]
 as a base I have created a new operator "MsSqlToGoogleCloudStorageOperator" 
that may be of use and to further expand on Airflow's GCP offering.


  was:Using both 
[mysql_to_gcs.py|[https://github.com/apache/airflow/blob/master/airflow/contrib/operators/mysql_to_gcs.py|https://github.com/Tomme/airflow/blob/master/airflow/contrib/operators/mysql_to_gcs.py]]
 and 
[postgres_to_gcs.py|[https://github.com/apache/airflow/blob/master/airflow/contrib/operators/postgres_to_gcs_operator.py|https://github.com/Tomme/airflow/blob/master/airflow/contrib/operators/postgres_to_gcs_operator.py]]
 as a base I have created a new operator "MsSqlToGoogleCloudStorageOperator" 
that may be of use and to further expand on Airflow's GCP offering.


> Add MsSqlToGoogleCloudStorageOperator
> -
>
> Key: AIRFLOW-4268
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4268
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: contrib, operators
>Affects Versions: 1.10.2
>Reporter: Thomas Elvey
>Assignee: Thomas Elvey
>Priority: Minor
>
> Using both 
> [mysql_to_gcs.py|https://github.com/Tomme/airflow/blob/master/airflow/contrib/operators/mysql_to_gcs.py]
>  and 
> [postgres_to_gcs.py|https://github.com/Tomme/airflow/blob/master/airflow/contrib/operators/postgres_to_gcs_operator.py]
>  as a base I have created a new operator "MsSqlToGoogleCloudStorageOperator" 
> that may be of use and to further expand on Airflow's GCP offering.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[GitHub] [airflow] jrans edited a comment on issue #3585: [AIRFLOW-2731] Raise psutil restriction to <6.0.0

2019-04-10 Thread GitBox
jrans edited a comment on issue #3585: [AIRFLOW-2731] Raise psutil restriction 
to <6.0.0
URL: https://github.com/apache/airflow/pull/3585#issuecomment-481722913
 
 
   Yep sorry for that should have debugged one step further before making 
conclusion but regardless important to document somewhere as [not only 
one](https://stackoverflow.com/questions/54340226/airflow-errorattributeerror-module-airflow-utils-log-has-no-attribute-file)
 with issue. Just unfortunate that at time of my airflow upgrade `psutil` 
latest was in a bad state and only became a problem once recreating my 
environment because of the change of upper version limit.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] jrans edited a comment on issue #3585: [AIRFLOW-2731] Raise psutil restriction to <6.0.0

2019-04-10 Thread GitBox
jrans edited a comment on issue #3585: [AIRFLOW-2731] Raise psutil restriction 
to <6.0.0
URL: https://github.com/apache/airflow/pull/3585#issuecomment-481722913
 
 
   Yep sorry for that should have debugged one step further before making 
conclusion but regardless important to document somewhere as [not only 
one](https://stackoverflow.com/questions/54340226/airflow-errorattributeerror-module-airflow-utils-log-has-no-attribute-file)
 with issue. 
   Just unfortunate that at time of my airflow upgrade `psutil` latest was in a 
bad state and only became a problem once recreating my environment because of 
the change of upper version limit.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Created] (AIRFLOW-4268) Add MsSqlToGoogleCloudStorageOperator

2019-04-10 Thread Thomas Elvey (JIRA)
Thomas Elvey created AIRFLOW-4268:
-

 Summary: Add MsSqlToGoogleCloudStorageOperator
 Key: AIRFLOW-4268
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4268
 Project: Apache Airflow
  Issue Type: New Feature
  Components: contrib, operators
Affects Versions: 1.10.2
Reporter: Thomas Elvey
Assignee: Thomas Elvey


Using both 
[mysql_to_gcs.py|[https://github.com/apache/airflow/blob/master/airflow/contrib/operators/mysql_to_gcs.py|https://github.com/Tomme/airflow/blob/master/airflow/contrib/operators/mysql_to_gcs.py]]
 and 
[postgres_to_gcs.py|[https://github.com/apache/airflow/blob/master/airflow/contrib/operators/postgres_to_gcs_operator.py|https://github.com/Tomme/airflow/blob/master/airflow/contrib/operators/postgres_to_gcs_operator.py]]
 as a base I have created a new operator "MsSqlToGoogleCloudStorageOperator" 
that may be of use and to further expand on Airflow's GCP offering.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[GitHub] [airflow] jrans commented on issue #3585: [AIRFLOW-2731] Raise psutil restriction to <6.0.0

2019-04-10 Thread GitBox
jrans commented on issue #3585: [AIRFLOW-2731] Raise psutil restriction to 
<6.0.0
URL: https://github.com/apache/airflow/pull/3585#issuecomment-481722913
 
 
   Yep sorry for that should have debugged one step further before making 
conclusion but regardless important to document somewhere as [not only 
one](https://stackoverflow.com/questions/54340226/airflow-errorattributeerror-module-airflow-utils-log-has-no-attribute-file)
 with issue. Just unfortunate that at time of my airflow upgrade `psutil` 
latest was in a bad state and only became a problem once clearing environment 
because of the change of upper version limit.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] XD-DENG commented on issue #5075: [AIRFLOW-XXX] Change allowed version of Jinja2 to fix CVE-2019-10906

2019-04-10 Thread GitBox
XD-DENG commented on issue #5075: [AIRFLOW-XXX] Change allowed version of 
Jinja2 to fix CVE-2019-10906
URL: https://github.com/apache/airflow/pull/5075#issuecomment-481721518
 
 
   Thanks @kaxil @ashb .
   
   Thanks Ash for the clarification 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] ashb commented on issue #5075: [AIRFLOW-XXX] Change allowed version of Jinja2 to fix CVE-2019-10906

2019-04-10 Thread GitBox
ashb commented on issue #5075: [AIRFLOW-XXX] Change allowed version of Jinja2 
to fix CVE-2019-10906
URL: https://github.com/apache/airflow/pull/5075#issuecomment-481720698
 
 
   > In Pallets Jinja before 2.10.1, str.format_map allows a sandbox escape.
   
   We should update, but this doesn't pose any risk to us as we don't use the 
sandbox. From the [blog 
post](https://palletsprojects.com/blog/jinja-2-10-1-released/):
   
   > The sandbox is used to restrict what code can be evaluated when rendering 
untrusted, user-provided templates


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] XD-DENG opened a new pull request #5075: [AIRFLOW-XXX] Change allowed version of Jinja2 to fix CVE-2019-10906

2019-04-10 Thread GitBox
XD-DENG opened a new pull request #5075: [AIRFLOW-XXX] Change allowed version 
of Jinja2 to fix CVE-2019-10906
URL: https://github.com/apache/airflow/pull/5075
 
 
   To change allowed version of Jinja2 to fix CVE-2019-10906
   
   Ref: https://nvd.nist.gov/vuln/detail/CVE-2019-10906


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] feluelle commented on issue #5069: [AIRFLOW-4266] Add mypy to setup.py devel dependencies

2019-04-10 Thread GitBox
feluelle commented on issue #5069: [AIRFLOW-4266] Add mypy to setup.py devel 
dependencies
URL: https://github.com/apache/airflow/pull/5069#issuecomment-481703694
 
 
   That's true. But I thought that because of the different formattings, it 
could be useful to just reformat it completely.
   
   For example some are using: `foo=['bar']` and some are using 
   ```
   foo=[
 'bar',
   ]
   ```
   
   I think it is more readable but if you don't like it or it is not worth it 
because of potentially breaking PR's I revert it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] kaxil commented on issue #3585: [AIRFLOW-2731] Raise psutil restriction to <6.0.0

2019-04-10 Thread GitBox
kaxil commented on issue #3585: [AIRFLOW-2731] Raise psutil restriction to 
<6.0.0
URL: https://github.com/apache/airflow/pull/3585#issuecomment-481702999
 
 
   But that is the reason why "This PR has broken airflow for me, old 
limitation of <5 seems like it was for a reason.." is incorrect  . 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] jrans commented on issue #3585: [AIRFLOW-2731] Raise psutil restriction to <6.0.0

2019-04-10 Thread GitBox
jrans commented on issue #3585: [AIRFLOW-2731] Raise psutil restriction to 
<6.0.0
URL: https://github.com/apache/airflow/pull/3585#issuecomment-481700569
 
 
   Thanks for getting back to me. Yes definitely not an airflow issue but 
highlighting the change in range in the airflow upgrade `1.10.2` revealing this 
and hopefully will help someone else suffering.
   
   I can confirm `psutil` @  `5.6.1` and latest `5.6.2` not working for me but 
pinning to `5.6.0` produces no error.. will raise with them.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] Fokko edited a comment on issue #5069: [AIRFLOW-4266] Add mypy to setup.py devel dependencies

2019-04-10 Thread GitBox
Fokko edited a comment on issue #5069: [AIRFLOW-4266] Add mypy to setup.py 
devel dependencies
URL: https://github.com/apache/airflow/pull/5069#issuecomment-481693330
 
 
   Why all the reformatting? This will potentially break a lot of other PR's.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] Fokko commented on issue #5069: [AIRFLOW-4266] Add mypy to setup.py devel dependencies

2019-04-10 Thread GitBox
Fokko commented on issue #5069: [AIRFLOW-4266] Add mypy to setup.py devel 
dependencies
URL: https://github.com/apache/airflow/pull/5069#issuecomment-481693330
 
 
   Why all the reformatting?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] Fokko commented on issue #5070: [AIRFLOW-XXX] Add a git pre-commit hook template

2019-04-10 Thread GitBox
Fokko commented on issue #5070: [AIRFLOW-XXX] Add a git pre-commit hook template
URL: https://github.com/apache/airflow/pull/5070#issuecomment-481690135
 
 
   I'm on @BasPH. I never use pre-commit hooks because I haven't found them 
useful. I'm still open to it if someone else finds it useful.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] ashb commented on issue #3585: [AIRFLOW-2731] Raise psutil restriction to <6.0.0

2019-04-10 Thread GitBox
ashb commented on issue #3585: [AIRFLOW-2731] Raise psutil restriction to <6.0.0
URL: https://github.com/apache/airflow/pull/3585#issuecomment-481686966
 
 
   The nature of that error implies it is a bug/conflicting installation of 
psutil. It looks almost like you have some new and some old files there.
   
   The "Cannot resolve 
'airflow.utils.log.file_processor_handler.FileProcessorHandler" part of the 
error message is just Airflow saying while trying to FileProcessorHandler we 
got this other error.
   
   You should see the exact same error form running `pyton -c "import psutil"` 
which would entirely remove Airflow from the equation.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Updated] (AIRFLOW-4267) Fix Task Instance duration in Graph View in Fab-Based UI

2019-04-10 Thread Ash Berlin-Taylor (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor updated AIRFLOW-4267:
---
Fix Version/s: (was: 1.10.3)
   1.10.4

> Fix Task Instance duration in Graph View in Fab-Based UI
> 
>
> Key: AIRFLOW-4267
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4267
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: 1.10.2, 1.10.3
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Critical
> Fix For: 1.10.4
>
> Attachments: Screenshot 2019-04-09 at 21.58.54.png, Screenshot 
> 2019-04-09 at 22.01.42.png
>
>
> The duration field is missing in the Graph View for the Fab-based RBAC UI.
> This is because of the bug in the code. Instead of *task.duration* it should 
> be *ti.duration*.
> The duration is associated with a *Task Instance* and no a *Task*.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[GitHub] [airflow] ashb commented on issue #3719: [AIRFLOW-2874] Enables FAB's theme support

2019-04-10 Thread GitBox
ashb commented on issue #3719: [AIRFLOW-2874] Enables FAB's theme support
URL: https://github.com/apache/airflow/pull/3719#issuecomment-481651508
 
 
   @holandes22 Take out of bootstrap-theme.css and put in our main.css


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] holandes22 commented on issue #3719: [AIRFLOW-2874] Enables FAB's theme support

2019-04-10 Thread GitBox
holandes22 commented on issue #3719: [AIRFLOW-2874] Enables FAB's theme support
URL: https://github.com/apache/airflow/pull/3719#issuecomment-481648984
 
 
   @ashb any direction on which is the correct CSS file to place the fix? I'll 
gladly send a PR, but need some guidance on that


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] yohei1126 commented on issue #4448: [AIRFLOW-3601] add location support to BigQuery operators

2019-04-10 Thread GitBox
yohei1126 commented on issue #4448: [AIRFLOW-3601] add location support to 
BigQuery operators
URL: https://github.com/apache/airflow/pull/4448#issuecomment-481638151
 
 
   sorry continue this week. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Commented] (AIRFLOW-3757) [Kubernetes Executor] Http connection times out while submitting new Pod request via kube client

2019-04-10 Thread Anand (JIRA)


[ 
https://issues.apache.org/jira/browse/AIRFLOW-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814308#comment-16814308
 ] 

Anand commented on AIRFLOW-3757:


Not an airflow issue, but Issue with K8 Master Api server

> [Kubernetes Executor] Http connection times out while submitting new Pod 
> request via kube client
> 
>
> Key: AIRFLOW-3757
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3757
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Anand
>Priority: Major
> Attachments: Stack-trace.txt
>
>
> The issue is intermittent. I think catching the exception and retrying can 
> solve it to some extent. Time out stack trace is attached.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Closed] (AIRFLOW-3757) [Kubernetes Executor] Http connection times out while submitting new Pod request via kube client

2019-04-10 Thread Anand (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anand closed AIRFLOW-3757.
--
Resolution: Fixed

> [Kubernetes Executor] Http connection times out while submitting new Pod 
> request via kube client
> 
>
> Key: AIRFLOW-3757
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3757
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Anand
>Priority: Major
> Attachments: Stack-trace.txt
>
>
> The issue is intermittent. I think catching the exception and retrying can 
> solve it to some extent. Time out stack trace is attached.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[GitHub] [airflow] codecov-io commented on issue #5069: [AIRFLOW-4266] Add mypy to setup.py devel dependencies

2019-04-10 Thread GitBox
codecov-io commented on issue #5069: [AIRFLOW-4266] Add mypy to setup.py devel 
dependencies
URL: https://github.com/apache/airflow/pull/5069#issuecomment-481622058
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/5069?src=pr=h1) 
Report
   > Merging 
[#5069](https://codecov.io/gh/apache/airflow/pull/5069?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/4c482ac3fc9d909648649bb2fcd1eb2150658032?src=pr=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/5069/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/5069?src=pr=tree)
   
   ```diff
   @@   Coverage Diff   @@
   ##   master#5069   +/-   ##
   ===
 Coverage   77.01%   77.01%   
   ===
 Files 463  463   
 Lines   2975329753   
   ===
 Hits2291522915   
 Misses   6838 6838
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/5069?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/5069?src=pr=footer). 
Last update 
[4c482ac...ec78fee](https://codecov.io/gh/apache/airflow/pull/5069?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] kaxil merged pull request #5052: [AIRFLOW-XXX] Build a universal wheel with LICNESE files

2019-04-10 Thread GitBox
kaxil merged pull request #5052: [AIRFLOW-XXX] Build a universal wheel with 
LICNESE files
URL: https://github.com/apache/airflow/pull/5052
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] kaxil commented on issue #5052: [AIRFLOW-XXX] Build a universal wheel with LICNESE files

2019-04-10 Thread GitBox
kaxil commented on issue #5052: [AIRFLOW-XXX] Build a universal wheel with 
LICNESE files
URL: https://github.com/apache/airflow/pull/5052#issuecomment-481613852
 
 
   Pasting this link for future references: 
https://wheel.readthedocs.io/en/latest/user_guide.html


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] sumous commented on issue #4969: [AIRFLOW-4148]form_choices in DagRunModelView may be wrong

2019-04-10 Thread GitBox
sumous commented on issue #4969: [AIRFLOW-4148]form_choices in DagRunModelView 
may be wrong
URL: https://github.com/apache/airflow/pull/4969#issuecomment-481612145
 
 
   @ashb Did you reproduce this bug?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [airflow] kaxil commented on issue #4448: [AIRFLOW-3601] add location support to BigQuery operators

2019-04-10 Thread GitBox
kaxil commented on issue #4448: [AIRFLOW-3601] add location support to BigQuery 
operators
URL: https://github.com/apache/airflow/pull/4448#issuecomment-481611515
 
 
   @yohei1126 any updates on this?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Updated] (AIRFLOW-4176) [security] webui shows password - admin/log/?flt1_extra_contains=conn_password

2019-04-10 Thread Kaxil Naik (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik updated AIRFLOW-4176:

Fix Version/s: 1.10.4

> [security] webui shows password - admin/log/?flt1_extra_contains=conn_password
> --
>
> Key: AIRFLOW-4176
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4176
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: security, ui
>Affects Versions: 1.10.2
>Reporter: t oo
>Priority: Blocker
> Fix For: 1.10.4
>
> Attachments: airf.png
>
>
> First setup hivecli connection:
> source /home/ec2-user/venv/bin/activate; airflow connections -a --conn_id 
> query_hive --conn_type hive_cli --conn_host domainhere --conn_port 1 
> --conn_schema default --conn_extra "\{\"use_beeline\":\"true\", 
> \"ssl-options\":\"ssl=true;sslTrustStore=path-${RUNTIME_ENV}.jks;trustStorePassword=${QUERY_JKS_PASW}\"}"
>  --conn_login ${QUERY_HIVE_USER} --conn_password ${QUERY_HIVE_PASW}
>  
> On the webui navigate to domain/admin/log/?flt1_extra_contains=conn_password
> and you will be able to see cleartext user and password!
> see attachment



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Resolved] (AIRFLOW-2881) Add compile_assets to Release Process Documentation

2019-04-10 Thread Kaxil Naik (JIRA)


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik resolved AIRFLOW-2881.
-
   Resolution: Fixed
 Assignee: Kaxil Naik  (was: KAMALJIT CHAKRABORTY)
Fix Version/s: (was: 2.0.0)

This was added to docs while releasing 1.10.2

> Add compile_assets to Release Process Documentation 
> 
>
> Key: AIRFLOW-2881
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2881
> Project: Apache Airflow
>  Issue Type: Task
>  Components: Documentation
>Reporter: Jacob Hayes
>Assignee: Kaxil Naik
>Priority: Blocker
>
> AIRFLOW-2691 introduced a `compile_assets` command in setup.py that is 
> required during a release. This should be added to the [release 
> notes|https://cwiki.apache.org/confluence/display/AIRFLOW/Releasing+Airflow#ReleasingAirflow-PublishingtoPyPi]
>  after 1.10.x has been released.
> {code:java}
> python setup.py compile_assets sdist
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


  1   2   >