[GitHub] codecov-io commented on issue #3804: [AIRFLOW-2866] Fix missing CSRF token header when using RBAC UI
codecov-io commented on issue #3804: [AIRFLOW-2866] Fix missing CSRF token header when using RBAC UI URL: https://github.com/apache/incubator-airflow/pull/3804#issuecomment-417901865 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3804?src=pr=h1) Report > Merging [#3804](https://codecov.io/gh/apache/incubator-airflow/pull/3804?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/db9bb7f57dad69d8cee657f0a901ff64558b3ef9?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3804?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3804 +/- ## == - Coverage 77.43% 77.43% -0.01% == Files 203 203 Lines 1584615846 == - Hits1227112270 -1 - Misses 3575 3576 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3804?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3804/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `88.74% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3804?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3804?src=pr=footer). Last update [db9bb7f...03af8f8](https://codecov.io/gh/apache/incubator-airflow/pull/3804?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-2996) Airflow subdag is failed while the child task is still running or succeded
Vijay Arum created AIRFLOW-2996:
---
Summary: Airflow subdag is failed while the child task is still
running or succeded
Key: AIRFLOW-2996
URL: https://issues.apache.org/jira/browse/AIRFLOW-2996
Project: Apache Airflow
Issue Type: Bug
Components: subdag
Affects Versions: 1.9.0
Reporter: Vijay Arum
Airflow subdag is failed while the child task is still running or succeded.
subdag when retries on failure , tries to execute the child task which is
already in running , this causes a deadlock and subdag is failed eventually
after all attempts. while all child task succeds.
this issue occurring frequency is not that often .
i have a subdag with bunch of tasks. i have set retries to 2. once in a while
the subdag fails in {color:#14892c}first attempt bcoz of mysql
connection{color} "{color:#FF}sqlalchemy.exc.OperationalError:
(_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at
\'reading initial communication packet\', system error: 0 "Internal error/check
(Not system error)"') (Background on this error at:
[http://sqlalche.me/e/e3q8)]"{color}
subdag initiates retry 1, subdag tries to re execute running child task, which
violates task instance state violation. {color:#d04437}"Task is in the
'running' state which is not a valid state for execution. The task must be
cleared in order to be run." {color:#33}subdag retry evenually fails with
the message{color} "following taska are deadlocked "{color}
{color:#d04437}{color:#33} subdag initiates retry 2,{color} subdag tries to
re execute running child task, which violates task instance state violation.
"Task is in the 'running' state which is not a valid state for execution. The
task must be cleared in order to be run." {color:#33}subdag retry evenually
fails with the message{color} "following taska are deadlocked "{color}
{color:#33}finally subdag is marked as failed while all child tasks
suceeds{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] gsilk commented on issue #3804: [AIRFLOW-2866] Fix missing CSRF token header when using RBAC UI
gsilk commented on issue #3804: [AIRFLOW-2866] Fix missing CSRF token header when using RBAC UI URL: https://github.com/apache/incubator-airflow/pull/3804#issuecomment-417900651 Made changes, sorry for the delay! Let me know what you think This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #3733: [AIRFLOW-491] Add cache parameter in BigQuery query method - with 'api_resource_configs'
codecov-io edited a comment on issue #3733: [AIRFLOW-491] Add cache parameter in BigQuery query method - with 'api_resource_configs' URL: https://github.com/apache/incubator-airflow/pull/3733#issuecomment-413105867 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3733?src=pr=h1) Report > Merging [#3733](https://codecov.io/gh/apache/incubator-airflow/pull/3733?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/db9bb7f57dad69d8cee657f0a901ff64558b3ef9?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3733?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#3733 +/- ## === Coverage 77.43% 77.43% === Files 203 203 Lines 1584615846 === Hits1227112271 Misses 3575 3575 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3733?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3733?src=pr=footer). Last update [db9bb7f...e26f3bd](https://codecov.io/gh/apache/incubator-airflow/pull/3733?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #3733: [AIRFLOW-491] Add cache parameter in BigQuery query method - with 'api_resource_configs'
codecov-io edited a comment on issue #3733: [AIRFLOW-491] Add cache parameter in BigQuery query method - with 'api_resource_configs' URL: https://github.com/apache/incubator-airflow/pull/3733#issuecomment-413105867 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3733?src=pr=h1) Report > Merging [#3733](https://codecov.io/gh/apache/incubator-airflow/pull/3733?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/db9bb7f57dad69d8cee657f0a901ff64558b3ef9?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3733?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3733 +/- ## == - Coverage 77.43% 77.43% -0.01% == Files 203 203 Lines 1584615846 == - Hits1227112270 -1 - Misses 3575 3576 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3733?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3733/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `88.74% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3733?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3733?src=pr=footer). Last update [db9bb7f...e26f3bd](https://codecov.io/gh/apache/incubator-airflow/pull/3733?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #3733: [AIRFLOW-491] Add cache parameter in BigQuery query method - with 'api_resource_configs'
codecov-io edited a comment on issue #3733: [AIRFLOW-491] Add cache parameter in BigQuery query method - with 'api_resource_configs' URL: https://github.com/apache/incubator-airflow/pull/3733#issuecomment-413105867 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3733?src=pr=h1) Report > Merging [#3733](https://codecov.io/gh/apache/incubator-airflow/pull/3733?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/db9bb7f57dad69d8cee657f0a901ff64558b3ef9?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3733?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3733 +/- ## == - Coverage 77.43% 77.43% -0.01% == Files 203 203 Lines 1584615846 == - Hits1227112270 -1 - Misses 3575 3576 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3733?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3733/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `88.74% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3733?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3733?src=pr=footer). Last update [db9bb7f...e26f3bd](https://codecov.io/gh/apache/incubator-airflow/pull/3733?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #3733: [AIRFLOW-491] Add cache parameter in BigQuery query method - with 'api_resource_configs'
codecov-io edited a comment on issue #3733: [AIRFLOW-491] Add cache parameter in BigQuery query method - with 'api_resource_configs' URL: https://github.com/apache/incubator-airflow/pull/3733#issuecomment-413105867 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3733?src=pr=h1) Report > Merging [#3733](https://codecov.io/gh/apache/incubator-airflow/pull/3733?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/db9bb7f57dad69d8cee657f0a901ff64558b3ef9?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3733?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3733 +/- ## == - Coverage 77.43% 77.43% -0.01% == Files 203 203 Lines 1584615846 == - Hits1227112270 -1 - Misses 3575 3576 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3733?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3733/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `88.74% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3733?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3733?src=pr=footer). Last update [db9bb7f...e26f3bd](https://codecov.io/gh/apache/incubator-airflow/pull/3733?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (AIRFLOW-2824) Disable loading of default connections via airflow config
[
https://issues.apache.org/jira/browse/AIRFLOW-2824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Uellendall reassigned AIRFLOW-2824:
-
Assignee: Andy Cooper
> Disable loading of default connections via airflow config
> -
>
> Key: AIRFLOW-2824
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2824
> Project: Apache Airflow
> Issue Type: Wish
>Reporter: Felix Uellendall
>Assignee: Andy Cooper
>Priority: Major
>
> I would love to have a variable I can set in the airflow.cfg, like the DAG
> examples have, to not load the default connections.
> Either by using {{load_examples}} that is already
> [there|https://github.com/apache/incubator-airflow/blob/dfa7b26ddaca80ee8fd9915ee9f6eac50fac77f6/airflow/config_templates/default_airflow.cfg#L128]
> for loading dag examples or by a new one like {{load_default_connections}}
> to check if the user wants to have it or not.
> The implementation of the default connections starts
> [here|https://github.com/apache/incubator-airflow/blob/9e1d8ee837ea2c23e828d070b6a72a6331d98602/airflow/utils/db.py#L94]
> Let me know what you guys think of it, pls. :)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2780) Adds IMAP Hook to interact with a mail server
[ https://issues.apache.org/jira/browse/AIRFLOW-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-2780: -- Description: This Hook connects to a mail server via IMAP to be able to retrieve email attachments by using [Python's IMAP Library.|https://docs.python.org/3.6/library/imaplib.html] Features: - `has_mail_attachment`: Can be used in a `Sensor` to check if there is an attachment on the mail server with the given name. - `retrieve_mail_attachments`: Can be used in an `Operator` to do sth. with the attachments returned as list of tuple. - `download_mail_attachments`: Can be used in an `Operator` to download the attachment. was: This Hook connects to a mail server via IMAP to be able to retrieve email attachments by using [Python's IMAP Library.|https://docs.python.org/3.6/library/imaplib.html] Features: - `has_mail_attachments`: Can be used in a `Sensor` to check if there is an attachment on the mail server with the given name. - `retrieve_mail_attachments`: Can be used in an `Operator` to do sth. with the attachments returned as list of tuple. - `download_mail_attachments`: Can be used in an `Operator` to download the attachment. Tests: - `test_connect_and_disconnect`: this tests calls the ImapHook as ContextManager - `test_has_mail_attachments_found` - `test_has_mail_attachments_not_found` - `test_has_mail_attachments_with_regex_found` - `test_has_mail_attachments_with_regex_not_found` - `test_retrieve_mail_attachments_found` - `test_retrieve_mail_attachments_not_found` - `test_retrieve_mail_attachments_with_regex_found` - `test_retrieve_mail_attachments_with_regex_not_found` - `test_download_mail_attachments_found` - `test_download_mail_attachments_not_found` - `test_download_mail_attachments_with_regex_found` - `test_download_mail_attachments_with_regex_not_found` > Adds IMAP Hook to interact with a mail server > - > > Key: AIRFLOW-2780 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2780 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > > This Hook connects to a mail server via IMAP to be able to retrieve email > attachments by using [Python's IMAP > Library.|https://docs.python.org/3.6/library/imaplib.html] > Features: > - `has_mail_attachment`: Can be used in a `Sensor` to check if there is an > attachment on the mail server with the given name. > - `retrieve_mail_attachments`: Can be used in an `Operator` to do sth. with > the attachments returned as list of tuple. > - `download_mail_attachments`: Can be used in an `Operator` to download the > attachment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] xnuinside commented on a change in pull request #3733: [AIRFLOW-491] Add cache parameter in BigQuery query method - with 'api_resource_configs'
xnuinside commented on a change in pull request #3733: [AIRFLOW-491] Add cache
parameter in BigQuery query method - with 'api_resource_configs'
URL: https://github.com/apache/incubator-airflow/pull/3733#discussion_r214523981
##
File path: airflow/contrib/operators/bigquery_operator.py
##
@@ -118,7 +118,8 @@ def __init__(self,
query_params=None,
labels=None,
priority='INTERACTIVE',
- time_partitioning={},
+ time_partitioning=None,
+ api_resource_configs=None,
Review comment:
@kaxil done
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-2965) Add CLI command to find the next dag run.
[
https://issues.apache.org/jira/browse/AIRFLOW-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599661#comment-16599661
]
Xiaodong DENG commented on AIRFLOW-2965:
Hi [~sanand] and [~jackjack10], I would like to take this up. Will try to draft
a PR later.
> Add CLI command to find the next dag run.
> -
>
> Key: AIRFLOW-2965
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2965
> Project: Apache Airflow
> Issue Type: Task
>Reporter: jack
>Assignee: Xiaodong DENG
>Priority: Minor
> Fix For: 1.10.1
>
>
> I have a dag with the following properties:
> {code:java}
> dag = DAG(
> dag_id='mydag',
> default_args=args,
> schedule_interval='0 1 * * *',
> max_active_runs=1,
> catchup=False){code}
>
>
> This runs great.
> Last run is: 2018-08-26 01:00 (start date is 2018-08-27 01:00)
>
> Now it's 2018-08-27 17:55 I decided to change my dag to:
>
> {code:java}
> dag = DAG(
> dag_id='mydag',
> default_args=args,
> schedule_interval='0 23 * * *',
> max_active_runs=1,
> catchup=False){code}
>
> Now, I have no idea when will be the next dag run.
> Will it be today at 23:00? I can't be sure when the cycle is complete. I'm
> not even sure that this change will do what I wish.
> I'm sure you guys are expert and you can answer this question but most of us
> wouldn't know.
>
> The scheduler has the knowledge when the dag is available for running. All
> I'm asking is to take that knowledge and create a CLI command that I will
> give the dag_id and it will tell me the next date/hour which my dag will be
> runnable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-2965) Add CLI command to find the next dag run.
[
https://issues.apache.org/jira/browse/AIRFLOW-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaodong DENG reassigned AIRFLOW-2965:
--
Assignee: Xiaodong DENG
> Add CLI command to find the next dag run.
> -
>
> Key: AIRFLOW-2965
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2965
> Project: Apache Airflow
> Issue Type: Task
>Reporter: jack
>Assignee: Xiaodong DENG
>Priority: Minor
> Fix For: 1.10.1
>
>
> I have a dag with the following properties:
> {code:java}
> dag = DAG(
> dag_id='mydag',
> default_args=args,
> schedule_interval='0 1 * * *',
> max_active_runs=1,
> catchup=False){code}
>
>
> This runs great.
> Last run is: 2018-08-26 01:00 (start date is 2018-08-27 01:00)
>
> Now it's 2018-08-27 17:55 I decided to change my dag to:
>
> {code:java}
> dag = DAG(
> dag_id='mydag',
> default_args=args,
> schedule_interval='0 23 * * *',
> max_active_runs=1,
> catchup=False){code}
>
> Now, I have no idea when will be the next dag run.
> Will it be today at 23:00? I can't be sure when the cycle is complete. I'm
> not even sure that this change will do what I wish.
> I'm sure you guys are expert and you can answer this question but most of us
> wouldn't know.
>
> The scheduler has the knowledge when the dag is available for running. All
> I'm asking is to take that knowledge and create a CLI command that I will
> give the dag_id and it will tell me the next date/hour which my dag will be
> runnable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2981) TypeError in dataflow operators when using GCS jar or py_file
[
https://issues.apache.org/jira/browse/AIRFLOW-2981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik resolved AIRFLOW-2981.
-
Resolution: Fixed
Fix Version/s: 1.10.1
Resolved by https://github.com/apache/incubator-airflow/pull/3831
> TypeError in dataflow operators when using GCS jar or py_file
> --
>
> Key: AIRFLOW-2981
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2981
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib, Dataflow
>Affects Versions: 1.9.0, 1.10
>Reporter: Jeffrey Payne
>Assignee: Kaxil Naik
>Priority: Major
> Fix For: 1.10.1
>
>
> The {{GoogleCloudBucketHelper.google_cloud_to_local}} function attempts to
> compare a list to an int, resulting in the TypeError, with:
> {noformat}
> ...
> path_components = file_name[self.GCS_PREFIX_LENGTH:].split('/')
> if path_components < 2:
> ...
> {noformat}
> This should be {{if len(path_components) < 2:}}.
> Also, fix {{if file_size > 0:}} in same function...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] kaxil closed pull request #3831: [AIRFLOW-2981] Fix TypeError in dataflow operators
kaxil closed pull request #3831: [AIRFLOW-2981] Fix TypeError in dataflow
operators
URL: https://github.com/apache/incubator-airflow/pull/3831
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/contrib/operators/dataflow_operator.py
b/airflow/contrib/operators/dataflow_operator.py
index 3f6093b3ba..980b5792e7 100644
--- a/airflow/contrib/operators/dataflow_operator.py
+++ b/airflow/contrib/operators/dataflow_operator.py
@@ -16,7 +16,7 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
+import os
import re
import uuid
import copy
@@ -358,7 +358,7 @@ def google_cloud_to_local(self, file_name):
# Extracts bucket_id and object_id by first removing 'gs://' prefix and
# then split the remaining by path delimiter '/'.
path_components = file_name[self.GCS_PREFIX_LENGTH:].split('/')
-if path_components < 2:
+if len(path_components) < 2:
raise Exception(
'Invalid Google Cloud Storage (GCS) object path: {}.'
.format(file_name))
@@ -369,7 +369,7 @@ def google_cloud_to_local(self, file_name):
path_components[-1])
file_size = self._gcs_hook.download(bucket_id, object_id, local_file)
-if file_size > 0:
+if os.stat(file_size).st_size > 0:
return local_file
raise Exception(
'Failed to download Google Cloud Storage GCS object: {}'
diff --git a/tests/contrib/operators/test_dataflow_operator.py
b/tests/contrib/operators/test_dataflow_operator.py
index 4ea5f65698..a373126b24 100644
--- a/tests/contrib/operators/test_dataflow_operator.py
+++ b/tests/contrib/operators/test_dataflow_operator.py
@@ -20,9 +20,10 @@
import unittest
-from airflow.contrib.operators.dataflow_operator import
DataFlowPythonOperator, \
-DataFlowJavaOperator, DataflowTemplateOperator
-from airflow.contrib.operators.dataflow_operator import DataFlowPythonOperator
+from airflow.contrib.operators.dataflow_operator import \
+DataFlowPythonOperator, DataFlowJavaOperator, \
+DataflowTemplateOperator, GoogleCloudBucketHelper
+
from airflow.version import version
try:
@@ -186,3 +187,25 @@ def test_exec(self, dataflow_mock):
}
start_template_hook.assert_called_once_with(TASK_ID, expected_options,
PARAMETERS, TEMPLATE)
+
+
+class GoogleCloudBucketHelperTest(unittest.TestCase):
+
+@mock.patch(
+
'airflow.contrib.operators.dataflow_operator.GoogleCloudBucketHelper.__init__'
+)
+def test_invalid_object_path(self, mock_parent_init):
+
+# This is just the path of a bucket hence invalid filename
+file_name = 'gs://test-bucket'
+mock_parent_init.return_value = None
+
+gcs_bucket_helper = GoogleCloudBucketHelper()
+gcs_bucket_helper._gcs_hook = mock.Mock()
+
+with self.assertRaises(Exception) as context:
+gcs_bucket_helper.google_cloud_to_local(file_name)
+
+self.assertEquals(
+'Invalid Google Cloud Storage (GCS) object path:
{}.'.format(file_name),
+str(context.exception))
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-2981) TypeError in dataflow operators when using GCS jar or py_file
[
https://issues.apache.org/jira/browse/AIRFLOW-2981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599603#comment-16599603
]
ASF GitHub Bot commented on AIRFLOW-2981:
-
kaxil closed pull request #3831: [AIRFLOW-2981] Fix TypeError in dataflow
operators
URL: https://github.com/apache/incubator-airflow/pull/3831
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/contrib/operators/dataflow_operator.py
b/airflow/contrib/operators/dataflow_operator.py
index 3f6093b3ba..980b5792e7 100644
--- a/airflow/contrib/operators/dataflow_operator.py
+++ b/airflow/contrib/operators/dataflow_operator.py
@@ -16,7 +16,7 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
+import os
import re
import uuid
import copy
@@ -358,7 +358,7 @@ def google_cloud_to_local(self, file_name):
# Extracts bucket_id and object_id by first removing 'gs://' prefix and
# then split the remaining by path delimiter '/'.
path_components = file_name[self.GCS_PREFIX_LENGTH:].split('/')
-if path_components < 2:
+if len(path_components) < 2:
raise Exception(
'Invalid Google Cloud Storage (GCS) object path: {}.'
.format(file_name))
@@ -369,7 +369,7 @@ def google_cloud_to_local(self, file_name):
path_components[-1])
file_size = self._gcs_hook.download(bucket_id, object_id, local_file)
-if file_size > 0:
+if os.stat(file_size).st_size > 0:
return local_file
raise Exception(
'Failed to download Google Cloud Storage GCS object: {}'
diff --git a/tests/contrib/operators/test_dataflow_operator.py
b/tests/contrib/operators/test_dataflow_operator.py
index 4ea5f65698..a373126b24 100644
--- a/tests/contrib/operators/test_dataflow_operator.py
+++ b/tests/contrib/operators/test_dataflow_operator.py
@@ -20,9 +20,10 @@
import unittest
-from airflow.contrib.operators.dataflow_operator import
DataFlowPythonOperator, \
-DataFlowJavaOperator, DataflowTemplateOperator
-from airflow.contrib.operators.dataflow_operator import DataFlowPythonOperator
+from airflow.contrib.operators.dataflow_operator import \
+DataFlowPythonOperator, DataFlowJavaOperator, \
+DataflowTemplateOperator, GoogleCloudBucketHelper
+
from airflow.version import version
try:
@@ -186,3 +187,25 @@ def test_exec(self, dataflow_mock):
}
start_template_hook.assert_called_once_with(TASK_ID, expected_options,
PARAMETERS, TEMPLATE)
+
+
+class GoogleCloudBucketHelperTest(unittest.TestCase):
+
+@mock.patch(
+
'airflow.contrib.operators.dataflow_operator.GoogleCloudBucketHelper.__init__'
+)
+def test_invalid_object_path(self, mock_parent_init):
+
+# This is just the path of a bucket hence invalid filename
+file_name = 'gs://test-bucket'
+mock_parent_init.return_value = None
+
+gcs_bucket_helper = GoogleCloudBucketHelper()
+gcs_bucket_helper._gcs_hook = mock.Mock()
+
+with self.assertRaises(Exception) as context:
+gcs_bucket_helper.google_cloud_to_local(file_name)
+
+self.assertEquals(
+'Invalid Google Cloud Storage (GCS) object path:
{}.'.format(file_name),
+str(context.exception))
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> TypeError in dataflow operators when using GCS jar or py_file
> --
>
> Key: AIRFLOW-2981
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2981
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib, Dataflow
>Affects Versions: 1.9.0, 1.10
>Reporter: Jeffrey Payne
>Assignee: Kaxil Naik
>Priority: Major
>
> The {{GoogleCloudBucketHelper.google_cloud_to_local}} function attempts to
> compare a list to an int, resulting in the TypeError, with:
> {noformat}
> ...
> path_components = file_name[self.GCS_PREFIX_LENGTH:].split('/')
> if path_components < 2:
> ...
> {noformat}
> This should be {{if len(path_components) < 2:}}.
> Also, fix {{if file_size > 0:}} in same function...
--
This message was sent by Atlassian JIRA
[jira] [Commented] (AIRFLOW-2111) Add the ability to pause/unpause a task within a DAG
[ https://issues.apache.org/jira/browse/AIRFLOW-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599591#comment-16599591 ] Felix Uellendall commented on AIRFLOW-2111: --- I think this violates the design of how Airflow' scheduling is supposed to work, because a task is the smallest entity and not should have a paused state where you have to hold the process it is running on. This would be very complex and I dont think this fits in Airflow's Design Concept. *tl;dr* [~mreid] I think your task is too big when you have this kind of a issue. When you have smaller tasks that *only do one thing* it should be sufficient to pause the dag so after the current running task has succeeded it will be paused and no more tasks will be queued for running. > Add the ability to pause/unpause a task within a DAG > > > Key: AIRFLOW-2111 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2111 > Project: Apache Airflow > Issue Type: New Feature > Components: scheduler, ui >Reporter: Mark Reid >Priority: Minor > > It would be convenient to be able to pause/unpause a task within a DAG > (rather than the entire DAG). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-2934) Pools not respected for internal subdag tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-2934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599585#comment-16599585 ] Felix Uellendall edited comment on AIRFLOW-2934 at 9/1/18 9:26 AM: --- This is a [known bug|https://issues.apache.org/jira/browse/AIRFLOW-41] with SubDag's and LocalExecutor. In Airflow's 1.10 Release there is a quick fix where the SubDagOperator's executor param default value has been set to SequentialExecutor to workaround this issue. But of course if you dont want to upgrade you can set it by [yourself|https://github.com/apache/incubator-airflow/blob/master/airflow/operators/subdag_operator.py#L50] (and FYI https://issues.apache.org/jira/browse/AIRFLOW-74) *EDIT:* Oh and can you try what I described in [this comment|https://issues.apache.org/jira/browse/AIRFLOW-2371?focusedCommentId=16599590=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16599590] was (Author: feluelle): This is a [known bug|https://issues.apache.org/jira/browse/AIRFLOW-41] with SubDag's and LocalExecutor. In Airflow's 1.10 Release there is a quick fix where the SubDagOperator's executor param default value has been set to SequentialExecutor to workaround this issue. But of course if you dont want to upgrade you can set it by [yourself|https://github.com/apache/incubator-airflow/blob/master/airflow/operators/subdag_operator.py#L50] (and FYI https://issues.apache.org/jira/browse/AIRFLOW-74) > Pools not respected for internal subdag tasks > - > > Key: AIRFLOW-2934 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2934 > Project: Apache Airflow > Issue Type: Bug > Components: pools, subdag >Affects Versions: 1.9.0 > Environment: Linux victorvianna 4.14.65-1-MANJARO #1 SMP PREEMPT Sat > Aug 18 13:29:56 UTC 2018 x86_64 GNU/Linux > Python 3.6.6 >Reporter: Victor Vianna >Priority: Blocker > Labels: pool, subdag > Attachments: Screenshot from 2018-08-22 12-32-32.png, dag_pool.py > > > I'm trying to have some subdags execute one task at a time. The way I found > was to create a first pool for the SubdagOperators (pool1 in the attached > code file) and a second one for the internal tasks (pool2). However, it > appears that pools for subdag elements are not being respected. Running > airflow 1.9.0 with LocalExecutor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2371) SubdagOperator ignores pooling
[
https://issues.apache.org/jira/browse/AIRFLOW-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599590#comment-16599590
]
Felix Uellendall commented on AIRFLOW-2371:
---
{noformat}
Using SequentialExecutor is a workaround, however it slows down execution of
subdags.
{noformat}
I think initializing only one SequentialExecutor and passing it to all your
SubDagOperators is quite faster. Did you try that?
> SubdagOperator ignores pooling
> --
>
> Key: AIRFLOW-2371
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2371
> Project: Apache Airflow
> Issue Type: Bug
> Components: subdag
>Affects Versions: Airflow 1.9.0
> Environment: Airflow 1.9.0
>Reporter: Xiao Zhu
>Priority: Major
>
> Basically we need a long term solution for
> https://issues.apache.org/jira/browse/AIRFLOW-74 . Using SequentialExecutor
> is a workaround, however it slows down execution of subdags.
> Looking for a better solution.
> I also see that when running subdags, db connection pools are not shared
> either (since they are separate processes), which causes db disconnection
> issues like https://issues.apache.org/jira/browse/AIRFLOW-2340 on our db
> because there can be too many connections. I hope it can be fixed too when
> subdag operator works with pooling..
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-2934) Pools not respected for internal subdag tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-2934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599585#comment-16599585 ] Felix Uellendall edited comment on AIRFLOW-2934 at 9/1/18 9:13 AM: --- This is a [known bug|https://issues.apache.org/jira/browse/AIRFLOW-41] with SubDag's and LocalExecutor. In Airflow's 1.10 Release there is a quick fix where the SubDagOperator's executor param default value has been set to SequentialExecutor to workaround this issue. But of course if you dont want to upgrade you can set it by [yourself|https://github.com/apache/incubator-airflow/blob/master/airflow/operators/subdag_operator.py#L50] (and FYI https://issues.apache.org/jira/browse/AIRFLOW-74) was (Author: feluelle): This is a [known bug|https://issues.apache.org/jira/browse/AIRFLOW-41] with SubDag's and LocalExecutor. In Airflow's 1.10 Release there is a quick fix where the SubDagOperator's executor param default value has been set to SequentialExecutor to workaround this issue. But of course if you dont want to upgrade you can set it by [yourself|https://github.com/apache/incubator-airflow/blob/master/airflow/operators/subdag_operator.py#L50] > Pools not respected for internal subdag tasks > - > > Key: AIRFLOW-2934 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2934 > Project: Apache Airflow > Issue Type: Bug > Components: pools, subdag >Affects Versions: 1.9.0 > Environment: Linux victorvianna 4.14.65-1-MANJARO #1 SMP PREEMPT Sat > Aug 18 13:29:56 UTC 2018 x86_64 GNU/Linux > Python 3.6.6 >Reporter: Victor Vianna >Priority: Blocker > Labels: pool, subdag > Attachments: Screenshot from 2018-08-22 12-32-32.png, dag_pool.py > > > I'm trying to have some subdags execute one task at a time. The way I found > was to create a first pool for the SubdagOperators (pool1 in the attached > code file) and a second one for the internal tasks (pool2). However, it > appears that pools for subdag elements are not being respected. Running > airflow 1.9.0 with LocalExecutor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2934) Pools not respected for internal subdag tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-2934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599585#comment-16599585 ] Felix Uellendall commented on AIRFLOW-2934: --- This is a [known bug|https://issues.apache.org/jira/browse/AIRFLOW-41] with SubDag's and LocalExecutor. In Airflow's 1.10 Release there is a quick fix where the SubDagOperator's executor param default value has been set to SequentialExecutor to workaround this issue. But of course if you dont want to upgrade you can set it by [yourself|https://github.com/apache/incubator-airflow/blob/master/airflow/operators/subdag_operator.py#L50] > Pools not respected for internal subdag tasks > - > > Key: AIRFLOW-2934 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2934 > Project: Apache Airflow > Issue Type: Bug > Components: pools, subdag >Affects Versions: 1.9.0 > Environment: Linux victorvianna 4.14.65-1-MANJARO #1 SMP PREEMPT Sat > Aug 18 13:29:56 UTC 2018 x86_64 GNU/Linux > Python 3.6.6 >Reporter: Victor Vianna >Priority: Blocker > Labels: pool, subdag > Attachments: Screenshot from 2018-08-22 12-32-32.png, dag_pool.py > > > I'm trying to have some subdags execute one task at a time. The way I found > was to create a first pool for the SubdagOperators (pool1 in the attached > code file) and a second one for the internal tasks (pool2). However, it > appears that pools for subdag elements are not being respected. Running > airflow 1.9.0 with LocalExecutor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ashb edited a comment on issue #3796: [AIRFLOW-2824] - Add config to disable default conn creation
ashb edited a comment on issue #3796: [AIRFLOW-2824] - Add config to disable default conn creation URL: https://github.com/apache/incubator-airflow/pull/3796#issuecomment-417844535 The classes are always available (even when the rbac is not enabled) so we should be able to create migrations for them somehow. Edit: thinking about it a bit more I'm not sure it makes sense to create Alembic migrations for the Flask RBAC models: if we import the model classes in the migration then the migrations can change over time (we got stung with this already once on our _own_ migrations), or we have to duplicate the model classes in the migration file, which means we have to manually keep them in sync/notice changes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on issue #3796: [AIRFLOW-2824] - Add config to disable default conn creation
ashb commented on issue #3796: [AIRFLOW-2824] - Add config to disable default conn creation URL: https://github.com/apache/incubator-airflow/pull/3796#issuecomment-417844535 The classes are always available (even when the rbac is not enabled) so we should be able to create migrations for them somehow. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #3823: [AIRFLOW-2985] An operator for S3 object copying
codecov-io edited a comment on issue #3823: [AIRFLOW-2985] An operator for S3 object copying URL: https://github.com/apache/incubator-airflow/pull/3823#issuecomment-417296502 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3823?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@f279151`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3823?src=pr=tree) ```diff @@Coverage Diff@@ ## master#3823 +/- ## = Coverage ? 77.43% = Files ? 203 Lines ?15846 Branches ?0 = Hits ?12271 Misses? 3575 Partials ?0 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3823?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3823?src=pr=footer). Last update [f279151...0f32481](https://codecov.io/gh/apache/incubator-airflow/pull/3823?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #3823: [AIRFLOW-2985] An operator for S3 object copying
codecov-io edited a comment on issue #3823: [AIRFLOW-2985] An operator for S3 object copying URL: https://github.com/apache/incubator-airflow/pull/3823#issuecomment-417296502 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3823?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@f279151`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3823?src=pr=tree) ```diff @@Coverage Diff@@ ## master#3823 +/- ## = Coverage ? 77.43% = Files ? 203 Lines ?15846 Branches ?0 = Hits ?12271 Misses? 3575 Partials ?0 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3823?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3823?src=pr=footer). Last update [f279151...0f32481](https://codecov.io/gh/apache/incubator-airflow/pull/3823?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
