[GitHub] [airflow] zhongjiajie commented on issue #7729: [AIRFLOW-7066] Use sphinx syntax in concepts.rst
zhongjiajie commented on issue #7729: [AIRFLOW-7066] Use sphinx syntax in concepts.rst URL: https://github.com/apache/airflow/pull/7729#issuecomment-599305130 > > BTW, I have some question could not understand, in my PR #7709, raise pylint error three time, but `airflow/www/views.py` is in `pylint_todo.txt`, which mean will not test by pylint, also test folder will not test by pylint. should where the pylint error? (I only change two files) > > I test in breeze using `pylint airflow/www/views.py` and `pylint test/www/views.py` only `airflow/www/views.py` have some error hint(but it in pylint_todo.txt) > > @exploy > > It is the test/ that is failing. We have two separate pre-commits - pylint and pylint-test . The test one is failing in the other build and should be fixed there. Ok, will take a look to the test, BTW, thanks for merging This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] villasv edited a comment on issue #7629: [AIRFLOW-6987] Avoid creating default connections
villasv edited a comment on issue #7629: [AIRFLOW-6987] Avoid creating default connections URL: https://github.com/apache/airflow/pull/7629#issuecomment-599293728 Looking forward to using this! :-) Looks like the tests failed but it was a timeout. Could someone trigger a new attempt? Also @NBardelot you need to include the link to the JIRA issue in the PR description. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] villasv commented on issue #7629: [AIRFLOW-6987] Avoid creating default connections
villasv commented on issue #7629: [AIRFLOW-6987] Avoid creating default connections URL: https://github.com/apache/airflow/pull/7629#issuecomment-599293728 Looking forward to using this! :-) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4175) S3Hook load_file should support ACL policy parameter
[
https://issues.apache.org/jira/browse/AIRFLOW-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059826#comment-17059826
]
ASF GitHub Bot commented on AIRFLOW-4175:
-
OmairK commented on pull request #7733: [AIRFLOW-4175] S3Hook load_file should
support ACL policy paramete
URL: https://github.com/apache/airflow/pull/7733
- Added acl_policy parameter to all the S3Hook.load_*() and
S3Hook.copy_object() function
- Added unittest to test the response permissions when the
policy is passed
- Updated the docstring of the function
Co-authored-by: retornam
---
Issue link: WILL BE INSERTED BY
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- =
JIRA ID*
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
* For document-only changes commit message can start with
`[AIRFLOW-]`.
---
In case of fundamental code change, Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
for more information.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> S3Hook load_file should support ACL policy parameter
>
>
> Key: AIRFLOW-4175
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4175
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
>Affects Versions: 1.10.2
>Reporter: Keith O'Brien
>Priority: Major
> Labels: gsoc, gsoc2020, mentor
>
> We have a use case where we are uploading files to an S3 bucket in a
> different AWS account to the one Airflow is running in. AWS S3 supports this
> situation using the pre canned ACL policy, specifically
> {{bucket-owner-full-control. }}
> However, the current implementations of the {{S3Hook.load_*}}() and
> {{S3Hook.copy_object}}() methods do not allow us to supply any ACL policy for
> the file being uploaded/copied to S3.
> It would be good to add another optional parameter to the {{S3Hook}} methods
> called {{acl_policy}} which would then be passed into the boto3 client method
> calls like so
>
> {code}
> # load_file
> ...
> if encrypt:
>extra_args['ServerSideEncryption'] = "AES256"
> if acl_policy:
>extra_args['ACL'] = acl_policy
> client.upload_file(filename, bucket_name, key, ExtraArgs=extra_args){code}
>
> {code}
> # load_bytes
> ...
> if encrypt:
>extra_args['ServerSideEncryption'] = "AES256"
> if acl_policy:
>extra_args['ACL'] = acl_policy
> client.upload_file(filename, bucket_name, key, ExtraArgs=extra_args){code}
> {code}
> # copy_object
> self.get_conn().copy_object(Bucket=dest_bucket_name,
>Key=dest_bucket_key,
>CopySource=CopySource,
>ACL=acl_policy)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] OmairK opened a new pull request #7733: [AIRFLOW-4175] S3Hook load_file should support ACL policy paramete
OmairK opened a new pull request #7733: [AIRFLOW-4175] S3Hook load_file should support ACL policy paramete URL: https://github.com/apache/airflow/pull/7733 - Added acl_policy parameter to all the S3Hook.load_*() and S3Hook.copy_object() function - Added unittest to test the response permissions when the policy is passed - Updated the docstring of the function Co-authored-by: retornam --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7689: [AIRFLOW-7038] Cassandra sensors tests
nuclearpinguin commented on a change in pull request #7689: [AIRFLOW-7038]
Cassandra sensors tests
URL: https://github.com/apache/airflow/pull/7689#discussion_r392713867
##
File path: tests/providers/apache/cassandra/sensors/test_record.py
##
@@ -0,0 +1,71 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+
+import pytest
+
+from airflow import DAG
+from airflow.providers.apache.cassandra.hooks.cassandra import CassandraHook
+from airflow.providers.apache.cassandra.sensors.record import
CassandraRecordSensor
+from airflow.utils import timezone
+
+DEFAULT_DATE = timezone.datetime(2017, 1, 1)
+
+
+@pytest.mark.integration("cassandra")
+class TestCassandraRecordSensor(unittest.TestCase):
+def setUp(self) -> None:
+args = {
+'owner': 'airflow',
+'start_date': DEFAULT_DATE
+}
+
+self.dag = DAG('test_dag_id', default_args=args)
+
+def test_poke(self):
+hook = CassandraHook(cassandra_conn_id="cassandra_default")
+session = hook.get_conn()
+cqls = [
+"DROP TABLE IF EXISTS s.t",
+"CREATE TABLE s.t (pk1 text, pk2 text, c text, PRIMARY KEY (pk1,
pk2))",
+"INSERT INTO s.t (pk1, pk2, c) VALUES ('foo', 'bar', 'baz')",
+]
+for cql in cqls:
+session.execute(cql)
Review comment:
I would mock `CassandraHook` and:
- check if it was initialized with required parameter
- the `record_exists` of initialized `hook` was called with expected
parameters
- check if the expected value was returned (mocking it)
- make sure that all methods I mock are already covered by tests. In this
case, I would check tests for `CassandraHook`
At least that is the approach we use for GCP operators / hooks. Once you
cover the hook with tests you can mock the hook in operator tests. In my
opinion, it's a nice way to separate and highlight what exactly is tested.
Of course, mocking only asserts that arguments are passed as expected and
some methods are called. To check integrity we should use system test (that
runs example DAG). It's something still discussed but is easy to do with
`SystemTest`:
https://github.com/apache/airflow/blob/master/TESTING.rst#id24
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-4175) S3Hook load_file should support ACL policy parameter
[
https://issues.apache.org/jira/browse/AIRFLOW-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059818#comment-17059818
]
ASF GitHub Bot commented on AIRFLOW-4175:
-
OmairK commented on pull request #7732: [AIRFLOW-4175] S3Hook load_file should
support ACL policy paramete
URL: https://github.com/apache/airflow/pull/7732
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> S3Hook load_file should support ACL policy parameter
>
>
> Key: AIRFLOW-4175
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4175
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
>Affects Versions: 1.10.2
>Reporter: Keith O'Brien
>Priority: Major
> Labels: gsoc, gsoc2020, mentor
>
> We have a use case where we are uploading files to an S3 bucket in a
> different AWS account to the one Airflow is running in. AWS S3 supports this
> situation using the pre canned ACL policy, specifically
> {{bucket-owner-full-control. }}
> However, the current implementations of the {{S3Hook.load_*}}() and
> {{S3Hook.copy_object}}() methods do not allow us to supply any ACL policy for
> the file being uploaded/copied to S3.
> It would be good to add another optional parameter to the {{S3Hook}} methods
> called {{acl_policy}} which would then be passed into the boto3 client method
> calls like so
>
> {code}
> # load_file
> ...
> if encrypt:
>extra_args['ServerSideEncryption'] = "AES256"
> if acl_policy:
>extra_args['ACL'] = acl_policy
> client.upload_file(filename, bucket_name, key, ExtraArgs=extra_args){code}
>
> {code}
> # load_bytes
> ...
> if encrypt:
>extra_args['ServerSideEncryption'] = "AES256"
> if acl_policy:
>extra_args['ACL'] = acl_policy
> client.upload_file(filename, bucket_name, key, ExtraArgs=extra_args){code}
> {code}
> # copy_object
> self.get_conn().copy_object(Bucket=dest_bucket_name,
>Key=dest_bucket_key,
>CopySource=CopySource,
>ACL=acl_policy)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4175) S3Hook load_file should support ACL policy parameter
[
https://issues.apache.org/jira/browse/AIRFLOW-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059817#comment-17059817
]
ASF GitHub Bot commented on AIRFLOW-4175:
-
OmairK commented on pull request #7732: [AIRFLOW-4175] S3Hook load_file should
support ACL policy paramete
URL: https://github.com/apache/airflow/pull/7732
- Added acl_policy parameter to all the S3Hook.load_*() and
S3Hook.copy_object() function
- Added unittest to test the response permissions when the
policy is passed
- Updated the docstring of the function
---
Issue link: WILL BE INSERTED BY
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- =
JIRA ID*
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
* For document-only changes commit message can start with
`[AIRFLOW-]`.
---
In case of fundamental code change, Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
for more information.
Co-authored-by: @retornam
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> S3Hook load_file should support ACL policy parameter
>
>
> Key: AIRFLOW-4175
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4175
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
>Affects Versions: 1.10.2
>Reporter: Keith O'Brien
>Priority: Major
> Labels: gsoc, gsoc2020, mentor
>
> We have a use case where we are uploading files to an S3 bucket in a
> different AWS account to the one Airflow is running in. AWS S3 supports this
> situation using the pre canned ACL policy, specifically
> {{bucket-owner-full-control. }}
> However, the current implementations of the {{S3Hook.load_*}}() and
> {{S3Hook.copy_object}}() methods do not allow us to supply any ACL policy for
> the file being uploaded/copied to S3.
> It would be good to add another optional parameter to the {{S3Hook}} methods
> called {{acl_policy}} which would then be passed into the boto3 client method
> calls like so
>
> {code}
> # load_file
> ...
> if encrypt:
>extra_args['ServerSideEncryption'] = "AES256"
> if acl_policy:
>extra_args['ACL'] = acl_policy
> client.upload_file(filename, bucket_name, key, ExtraArgs=extra_args){code}
>
> {code}
> # load_bytes
> ...
> if encrypt:
>extra_args['ServerSideEncryption'] = "AES256"
> if acl_policy:
>extra_args['ACL'] = acl_policy
> client.upload_file(filename, bucket_name, key, ExtraArgs=extra_args){code}
> {code}
> # copy_object
> self.get_conn().copy_object(Bucket=dest_bucket_name,
>Key=dest_bucket_key,
>CopySource=CopySource,
>ACL=acl_policy)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] OmairK closed pull request #7732: [AIRFLOW-4175] S3Hook load_file should support ACL policy paramete
OmairK closed pull request #7732: [AIRFLOW-4175] S3Hook load_file should support ACL policy paramete URL: https://github.com/apache/airflow/pull/7732 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] OmairK opened a new pull request #7732: [AIRFLOW-4175] S3Hook load_file should support ACL policy paramete
OmairK opened a new pull request #7732: [AIRFLOW-4175] S3Hook load_file should support ACL policy paramete URL: https://github.com/apache/airflow/pull/7732 - Added acl_policy parameter to all the S3Hook.load_*() and S3Hook.copy_object() function - Added unittest to test the response permissions when the policy is passed - Updated the docstring of the function --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. Co-authored-by: @retornam This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7731: [AIRFLOW-7068] Create EC2 Hook, Operator and Sensor
nuclearpinguin commented on a change in pull request #7731: [AIRFLOW-7068]
Create EC2 Hook, Operator and Sensor
URL: https://github.com/apache/airflow/pull/7731#discussion_r392713098
##
File path: airflow/providers/amazon/aws/operators/ec2.py
##
@@ -0,0 +1,102 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+import time
+from typing import Optional
+
+from airflow.exceptions import AirflowException
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.ec2 import EC2Hook
+from airflow.utils.decorators import apply_defaults
+
+
+class EC2Operator(BaseOperator):
+"""
+Manage AWS EC2 instance using boto3.
+Change instance state by applying given operation.
+
+:param operation: action to be taken on AWS EC2 instance
+valid values: "start", "stop"
+:type operation: str
+:param instance_id: id of the AWS EC2 instance
+:type instance_id: str
+:param aws_conn_id: aws connection to use
+:type aws_conn_id: str
+:param region_name: (optional) aws region name associated with the client
+:type region_name: Optional[str]
+:param check_interval: time in seconds that the job should wait in
+between each instance state checks until operation is completed
+:type check_interval: float
+"""
+
+template_fields = ["operation", "region_name"]
+ui_color = "#eeaa11"
+ui_fgcolor = "#ff"
+valid_operations = ["start", "stop"]
+operation_target_state_map = {
+"start": "running",
+"stop": "stopped",
+}
+
+@apply_defaults
+def __init__(self,
+ operation: str,
+ instance_id: str,
+ aws_conn_id: str = "aws_default",
+ region_name: Optional[str] = None,
+ check_interval: float = 15,
+ *args,
+ **kwargs):
+if operation not in self.valid_operations:
+raise AirflowException(f"Invalid operation: {operation}")
+super().__init__(*args, **kwargs)
+self.operation = operation
+self.instance_id = instance_id
+self.aws_conn_id = aws_conn_id
+self.region_name = region_name
+self.check_interval = check_interval
+self.target_state = self.operation_target_state_map[self.operation]
+self.hook = self.get_hook()
+
+def execute(self, context):
+self.log.info("Executing: %s %s", self.operation, self.instance_id)
+
+instance = self.hook.get_conn().Instance(id=self.instance_id)
+
+if self.operation == "start":
Review comment:
I agree with @mik-laj, we should two operators even if the difference is
small :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] OmairK commented on issue #7726: [AIRFLOW-4175] S3Hook load_file should support ACL policy parameter
OmairK commented on issue #7726: [AIRFLOW-4175] S3Hook load_file should support ACL policy parameter URL: https://github.com/apache/airflow/pull/7726#issuecomment-599265008 > And BTW - both @OmairK and @retornam can be co-authors of the change https://help.github.com/en/github/committing-changes-to-your-project/creating-a-commit-with-multiple-authors - we've done that multiple times and this is true open-source spirit ... - so why don't you add @retornam co-author line to the change :)? Sure I will do that right now. > @OmairK I can close mine, when you submit a new PR. Thanks @retornam :smile: This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] retornam commented on issue #7726: [AIRFLOW-4175] S3Hook load_file should support ACL policy parameter
retornam commented on issue #7726: [AIRFLOW-4175] S3Hook load_file should support ACL policy parameter URL: https://github.com/apache/airflow/pull/7726#issuecomment-599264509 @OmairK I can close mine, when you submit a new PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7726: [AIRFLOW-4175] S3Hook load_file should support ACL policy parameter
potiuk commented on issue #7726: [AIRFLOW-4175] S3Hook load_file should support ACL policy parameter URL: https://github.com/apache/airflow/pull/7726#issuecomment-599264373 Somehow I can't re-open it :(. Likely you need to reopen it yourself @OmairK or create a new one with new branch name This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7726: [AIRFLOW-4175] S3Hook load_file should support ACL policy parameter
potiuk commented on issue #7726: [AIRFLOW-4175] S3Hook load_file should support ACL policy parameter URL: https://github.com/apache/airflow/pull/7726#issuecomment-599264312 And BTW - both @OmairK and @retornam can be co-authors of the change https://help.github.com/en/github/committing-changes-to-your-project/creating-a-commit-with-multiple-authors - we've done that multiple times and this is true open-source spirit ... - so why don't you add @retornam co-author line to the change :)? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-6530) Allow for custom Statsd client
[
https://issues.apache.org/jira/browse/AIRFLOW-6530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik resolved AIRFLOW-6530.
-
Resolution: Fixed
> Allow for custom Statsd client
> --
>
> Key: AIRFLOW-6530
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6530
> Project: Apache Airflow
> Issue Type: New Feature
> Components: scheduler, webserver
>Affects Versions: 1.10.7
>Reporter: Usman Arshad
>Assignee: Usman Arshad
>Priority: Major
> Labels: features
> Fix For: 1.10.10
>
>
> We are currently using Airflow at Skyscanner and we have a custom
> implementation of Statsd which offers features which wires in nicely into our
> metrics platform/tooling.
> I'm quite sure that other companies who are using Airflow would also find
> great benefit in being able to utilise their own custom Statsd client,
> therefore i am proposing this addition.
>
> The proposed solution looks something along the lines of changing this:
> {code:java}
> statsd = StatsClient(
> host=conf.get('scheduler', 'statsd_host'),
> port=conf.getint('scheduler', 'statsd_port'),
> prefix=conf.get('scheduler', 'statsd_prefix'))
> {code}
> Into
> {code:java}
> statsd = conf.get('STATSD_CLIENT') or StatsClient(
> host=conf.get('scheduler', 'statsd_host'),
> port=conf.getint('scheduler', 'statsd_port'),
> prefix=conf.get('scheduler', 'statsd_prefix'))
> {code}
> Note: Psuedocode, not actual code
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6530) Allow for custom Statsd client
[
https://issues.apache.org/jira/browse/AIRFLOW-6530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059804#comment-17059804
]
ASF GitHub Bot commented on AIRFLOW-6530:
-
kaxil commented on pull request #7227: [AIRFLOW-6530] Allow Custom Statsd Client
URL: https://github.com/apache/airflow/pull/7227
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Allow for custom Statsd client
> --
>
> Key: AIRFLOW-6530
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6530
> Project: Apache Airflow
> Issue Type: New Feature
> Components: scheduler, webserver
>Affects Versions: 1.10.7
>Reporter: Usman Arshad
>Assignee: Usman Arshad
>Priority: Major
> Labels: features
> Fix For: 1.10.10
>
>
> We are currently using Airflow at Skyscanner and we have a custom
> implementation of Statsd which offers features which wires in nicely into our
> metrics platform/tooling.
> I'm quite sure that other companies who are using Airflow would also find
> great benefit in being able to utilise their own custom Statsd client,
> therefore i am proposing this addition.
>
> The proposed solution looks something along the lines of changing this:
> {code:java}
> statsd = StatsClient(
> host=conf.get('scheduler', 'statsd_host'),
> port=conf.getint('scheduler', 'statsd_port'),
> prefix=conf.get('scheduler', 'statsd_prefix'))
> {code}
> Into
> {code:java}
> statsd = conf.get('STATSD_CLIENT') or StatsClient(
> host=conf.get('scheduler', 'statsd_host'),
> port=conf.getint('scheduler', 'statsd_port'),
> prefix=conf.get('scheduler', 'statsd_prefix'))
> {code}
> Note: Psuedocode, not actual code
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] kaxil merged pull request #7227: [AIRFLOW-6530] Allow Custom Statsd Client
kaxil merged pull request #7227: [AIRFLOW-6530] Allow Custom Statsd Client URL: https://github.com/apache/airflow/pull/7227 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] boring-cyborg[bot] commented on issue #7227: [AIRFLOW-6530] Allow Custom Statsd Client
boring-cyborg[bot] commented on issue #7227: [AIRFLOW-6530] Allow Custom Statsd Client URL: https://github.com/apache/airflow/pull/7227#issuecomment-599260333 Awesome work, congrats on your first merged pull request! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7227: [AIRFLOW-6530] Allow Custom Statsd Client
kaxil commented on a change in pull request #7227: [AIRFLOW-6530] Allow Custom
Statsd Client
URL: https://github.com/apache/airflow/pull/7227#discussion_r392709260
##
File path: airflow/stats.py
##
@@ -200,10 +200,35 @@ def __init__(self, *args, **kwargs):
self.__class__.instance = DummyStatsLogger()
except (socket.gaierror, ImportError) as e:
log.warning("Could not configure StatsClient: %s, using
DummyStatsLogger instead.", e)
+self.__class__.instance = DummyStatsLogger()
def get_statsd_logger(self):
-from statsd import StatsClient
-statsd = StatsClient(
+if conf.getboolean('scheduler', 'statsd_on'):
+
+from statsd import StatsClient
+
+if conf.has_option('scheduler', 'statsd_custom_client_path'):
+
+custom_statsd_module_path = conf.get('scheduler',
'statsd_custom_client_path')
+
+try:
+stats_class = import_string(custom_statsd_module_path)
+
Review comment:
```suggestion
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7227: [AIRFLOW-6530] Allow Custom Statsd Client
kaxil commented on a change in pull request #7227: [AIRFLOW-6530] Allow Custom
Statsd Client
URL: https://github.com/apache/airflow/pull/7227#discussion_r392709232
##
File path: airflow/stats.py
##
@@ -200,10 +200,35 @@ def __init__(self, *args, **kwargs):
self.__class__.instance = DummyStatsLogger()
except (socket.gaierror, ImportError) as e:
log.warning("Could not configure StatsClient: %s, using
DummyStatsLogger instead.", e)
+self.__class__.instance = DummyStatsLogger()
def get_statsd_logger(self):
-from statsd import StatsClient
-statsd = StatsClient(
+if conf.getboolean('scheduler', 'statsd_on'):
+
Review comment:
```suggestion
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7227: [AIRFLOW-6530] Allow Custom Statsd Client
kaxil commented on a change in pull request #7227: [AIRFLOW-6530] Allow Custom
Statsd Client
URL: https://github.com/apache/airflow/pull/7227#discussion_r392709252
##
File path: airflow/stats.py
##
@@ -200,10 +200,35 @@ def __init__(self, *args, **kwargs):
self.__class__.instance = DummyStatsLogger()
except (socket.gaierror, ImportError) as e:
log.warning("Could not configure StatsClient: %s, using
DummyStatsLogger instead.", e)
+self.__class__.instance = DummyStatsLogger()
def get_statsd_logger(self):
-from statsd import StatsClient
-statsd = StatsClient(
+if conf.getboolean('scheduler', 'statsd_on'):
+
+from statsd import StatsClient
+
+if conf.has_option('scheduler', 'statsd_custom_client_path'):
+
Review comment:
```suggestion
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] OmairK edited a comment on issue #7726: [AIRFLOW-4175] S3Hook load_file should support ACL policy parameter
OmairK edited a comment on issue #7726: [AIRFLOW-4175] S3Hook load_file should support ACL policy parameter URL: https://github.com/apache/airflow/pull/7726#issuecomment-599255128 > @OmairK -> I think you will have to rebase and push the branch again so that we can re-open it. @potiuk I have rebased and forced pushed the branch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] OmairK commented on issue #7726: [AIRFLOW-4175] S3Hook load_file should support ACL policy parameter
OmairK commented on issue #7726: [AIRFLOW-4175] S3Hook load_file should support ACL policy parameter URL: https://github.com/apache/airflow/pull/7726#issuecomment-599255128 > @OmairK -> I think you will have to rebase and push the branch again so that we can re-open it. @potiuk sure on it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mustafagok commented on a change in pull request #7731: [AIRFLOW-7068] Create EC2 Hook, Operator and Sensor
mustafagok commented on a change in pull request #7731: [AIRFLOW-7068] Create
EC2 Hook, Operator and Sensor
URL: https://github.com/apache/airflow/pull/7731#discussion_r392705410
##
File path: airflow/providers/amazon/aws/hooks/ec2.py
##
@@ -0,0 +1,48 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from typing import Optional
+
+from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook
+
+
+class EC2Hook(AwsBaseHook):
+"""
+Interact with AWS EC2 Service.
+"""
+
+def __init__(self,
+ region_name: Optional[str] = None,
+ *args,
+ **kwargs):
+self.region_name = region_name
+self.conn = None
+super().__init__(*args, **kwargs)
+
+def get_conn(self):
+"""
+Return self.conn, if it is None initialize ec2 resource object.
+
+:return: ec2 resource
+:rtype: boto3.resource
+"""
+# set self.conn in first call
+if not self.conn:
+self.conn = self.get_resource_type("ec2", self.region_name)
+return self.conn
Review comment:
You are right, I think I should add a method named get_instance(instance_id)
or get_instance_state(instance_id) to thi EC2Hook class. Then I can call this
from operator/sensor. What do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mustafagok commented on a change in pull request #7731: [AIRFLOW-7068] Create EC2 Hook, Operator and Sensor
mustafagok commented on a change in pull request #7731: [AIRFLOW-7068] Create
EC2 Hook, Operator and Sensor
URL: https://github.com/apache/airflow/pull/7731#discussion_r392705410
##
File path: airflow/providers/amazon/aws/hooks/ec2.py
##
@@ -0,0 +1,48 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from typing import Optional
+
+from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook
+
+
+class EC2Hook(AwsBaseHook):
+"""
+Interact with AWS EC2 Service.
+"""
+
+def __init__(self,
+ region_name: Optional[str] = None,
+ *args,
+ **kwargs):
+self.region_name = region_name
+self.conn = None
+super().__init__(*args, **kwargs)
+
+def get_conn(self):
+"""
+Return self.conn, if it is None initialize ec2 resource object.
+
+:return: ec2 resource
+:rtype: boto3.resource
+"""
+# set self.conn in first call
+if not self.conn:
+self.conn = self.get_resource_type("ec2", self.region_name)
+return self.conn
Review comment:
You are right, I think I should add a method named get_instance(instance_id)
or get_instance_state(instance_id) to this EC2Hook class. Then I can call this
from operator/sensor. What do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mustafagok commented on a change in pull request #7731: [AIRFLOW-7068] Create EC2 Hook, Operator and Sensor
mustafagok commented on a change in pull request #7731: [AIRFLOW-7068] Create
EC2 Hook, Operator and Sensor
URL: https://github.com/apache/airflow/pull/7731#discussion_r392705107
##
File path: airflow/providers/amazon/aws/operators/ec2.py
##
@@ -0,0 +1,102 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+import time
+from typing import Optional
+
+from airflow.exceptions import AirflowException
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.ec2 import EC2Hook
+from airflow.utils.decorators import apply_defaults
+
+
+class EC2Operator(BaseOperator):
+"""
+Manage AWS EC2 instance using boto3.
+Change instance state by applying given operation.
+
+:param operation: action to be taken on AWS EC2 instance
+valid values: "start", "stop"
+:type operation: str
+:param instance_id: id of the AWS EC2 instance
+:type instance_id: str
+:param aws_conn_id: aws connection to use
+:type aws_conn_id: str
+:param region_name: (optional) aws region name associated with the client
+:type region_name: Optional[str]
+:param check_interval: time in seconds that the job should wait in
+between each instance state checks until operation is completed
+:type check_interval: float
+"""
+
+template_fields = ["operation", "region_name"]
+ui_color = "#eeaa11"
+ui_fgcolor = "#ff"
+valid_operations = ["start", "stop"]
+operation_target_state_map = {
+"start": "running",
+"stop": "stopped",
+}
+
+@apply_defaults
+def __init__(self,
+ operation: str,
+ instance_id: str,
+ aws_conn_id: str = "aws_default",
+ region_name: Optional[str] = None,
+ check_interval: float = 15,
+ *args,
+ **kwargs):
+if operation not in self.valid_operations:
+raise AirflowException(f"Invalid operation: {operation}")
+super().__init__(*args, **kwargs)
+self.operation = operation
+self.instance_id = instance_id
+self.aws_conn_id = aws_conn_id
+self.region_name = region_name
+self.check_interval = check_interval
+self.target_state = self.operation_target_state_map[self.operation]
+self.hook = self.get_hook()
+
+def execute(self, context):
+self.log.info("Executing: %s %s", self.operation, self.instance_id)
+
+instance = self.hook.get_conn().Instance(id=self.instance_id)
+
+if self.operation == "start":
Review comment:
I can write EC2StartInstanceOperator and EC2StopInstanceOperator, but these
operators will be identical except 2 lines. What do you suggest?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] blcksrx commented on a change in pull request #7689: [AIRFLOW-7038] Cassandra sensors tests
blcksrx commented on a change in pull request #7689: [AIRFLOW-7038] Cassandra
sensors tests
URL: https://github.com/apache/airflow/pull/7689#discussion_r392705089
##
File path: tests/providers/apache/cassandra/sensors/test_record.py
##
@@ -0,0 +1,71 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+
+import pytest
+
+from airflow import DAG
+from airflow.providers.apache.cassandra.hooks.cassandra import CassandraHook
+from airflow.providers.apache.cassandra.sensors.record import
CassandraRecordSensor
+from airflow.utils import timezone
+
+DEFAULT_DATE = timezone.datetime(2017, 1, 1)
+
+
+@pytest.mark.integration("cassandra")
+class TestCassandraRecordSensor(unittest.TestCase):
+def setUp(self) -> None:
+args = {
+'owner': 'airflow',
+'start_date': DEFAULT_DATE
+}
+
+self.dag = DAG('test_dag_id', default_args=args)
+
+def test_poke(self):
+hook = CassandraHook(cassandra_conn_id="cassandra_default")
+session = hook.get_conn()
+cqls = [
+"DROP TABLE IF EXISTS s.t",
+"CREATE TABLE s.t (pk1 text, pk2 text, c text, PRIMARY KEY (pk1,
pk2))",
+"INSERT INTO s.t (pk1, pk2, c) VALUES ('foo', 'bar', 'baz')",
+]
+for cql in cqls:
+session.execute(cql)
Review comment:
As far as the sensor uses the hook methods, what's your idea to mock the
hook methods? For example, mocking `record_exists` in here:
```
def poke(self, context: Dict[str, str]) -> bool:
self.log.info('Sensor check existence of record: %s', self.keys)
hook = CassandraHook(self.cassandra_conn_id)
return hook.record_exists(self.table, self.keys)
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7726: [AIRFLOW-4175] S3Hook load_file should support ACL policy parameter
potiuk commented on issue #7726: [AIRFLOW-4175] S3Hook load_file should support ACL policy parameter URL: https://github.com/apache/airflow/pull/7726#issuecomment-599254110 @OmairK -> I think you will have to rebase and push the branch again so that we can re-open it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7726: [AIRFLOW-4175] S3Hook load_file should support ACL policy parameter
potiuk commented on issue #7726: [AIRFLOW-4175] S3Hook load_file should support ACL policy parameter URL: https://github.com/apache/airflow/pull/7726#issuecomment-599254033 I think we can combine the two.. There are more tests in this one - so I think we should use it :). @retornam - how about you review this one instead of merging yours :)? Let's join forces. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7731: [AIRFLOW-7068] Create EC2 Hook, Operator and Sensor
mik-laj commented on a change in pull request #7731: [AIRFLOW-7068] Create EC2
Hook, Operator and Sensor
URL: https://github.com/apache/airflow/pull/7731#discussion_r392704096
##
File path: airflow/providers/amazon/aws/hooks/ec2.py
##
@@ -0,0 +1,48 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from typing import Optional
+
+from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook
+
+
+class EC2Hook(AwsBaseHook):
+"""
+Interact with AWS EC2 Service.
+"""
+
+def __init__(self,
+ region_name: Optional[str] = None,
+ *args,
+ **kwargs):
+self.region_name = region_name
+self.conn = None
+super().__init__(*args, **kwargs)
+
+def get_conn(self):
+"""
+Return self.conn, if it is None initialize ec2 resource object.
+
+:return: ec2 resource
+:rtype: boto3.resource
+"""
+# set self.conn in first call
+if not self.conn:
+self.conn = self.get_resource_type("ec2", self.region_name)
+return self.conn
Review comment:
This operator has too few methods. In an ideal world, operators should never
call the get_conn method directly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7731: [AIRFLOW-7068] Create EC2 Hook, Operator and Sensor
mik-laj commented on a change in pull request #7731: [AIRFLOW-7068] Create EC2
Hook, Operator and Sensor
URL: https://github.com/apache/airflow/pull/7731#discussion_r392703975
##
File path: airflow/providers/amazon/aws/operators/ec2.py
##
@@ -0,0 +1,102 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+import time
+from typing import Optional
+
+from airflow.exceptions import AirflowException
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.ec2 import EC2Hook
+from airflow.utils.decorators import apply_defaults
+
+
+class EC2Operator(BaseOperator):
+"""
+Manage AWS EC2 instance using boto3.
+Change instance state by applying given operation.
+
+:param operation: action to be taken on AWS EC2 instance
+valid values: "start", "stop"
+:type operation: str
+:param instance_id: id of the AWS EC2 instance
+:type instance_id: str
+:param aws_conn_id: aws connection to use
+:type aws_conn_id: str
+:param region_name: (optional) aws region name associated with the client
+:type region_name: Optional[str]
+:param check_interval: time in seconds that the job should wait in
+between each instance state checks until operation is completed
+:type check_interval: float
+"""
+
+template_fields = ["operation", "region_name"]
+ui_color = "#eeaa11"
+ui_fgcolor = "#ff"
+valid_operations = ["start", "stop"]
+operation_target_state_map = {
+"start": "running",
+"stop": "stopped",
+}
+
+@apply_defaults
+def __init__(self,
+ operation: str,
+ instance_id: str,
+ aws_conn_id: str = "aws_default",
+ region_name: Optional[str] = None,
+ check_interval: float = 15,
+ *args,
+ **kwargs):
+if operation not in self.valid_operations:
+raise AirflowException(f"Invalid operation: {operation}")
+super().__init__(*args, **kwargs)
+self.operation = operation
+self.instance_id = instance_id
+self.aws_conn_id = aws_conn_id
+self.region_name = region_name
+self.check_interval = check_interval
+self.target_state = self.operation_target_state_map[self.operation]
+self.hook = self.get_hook()
+
+def execute(self, context):
+self.log.info("Executing: %s %s", self.operation, self.instance_id)
+
+instance = self.hook.get_conn().Instance(id=self.instance_id)
+
+if self.operation == "start":
Review comment:
It is recommended to create one operator for each operation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] OmairK commented on issue #7726: [AIRFLOW-4175] S3Hook load_file should support ACL policy parameter
OmairK commented on issue #7726: [AIRFLOW-4175] S3Hook load_file should support ACL policy parameter URL: https://github.com/apache/airflow/pull/7726#issuecomment-599247922 > I am sorry, but this is a duplicate. #7635 already adds ACL Header. @feluelle I have already completed the task and made the changes @potiuk suggested, and since this issue was not explicitly assigned to anybody I have been working on this issue for the past 2 days. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] retornam commented on issue #7635: [AIRFLOW-5062] Allow ACL Header in S3Hook
retornam commented on issue #7635: [AIRFLOW-5062] Allow ACL Header in S3Hook URL: https://github.com/apache/airflow/pull/7635#issuecomment-599243113 Yes. I’ll update the PR later today. > On Mar 15, 2020, at 09:44, Felix Uellendall wrote: > > > @retornam are you still working on this? Because there is also another PR #7726 which adds the same feature. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub, or unsubscribe. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7730: [AIRFLOW-7067] Pinned version of Apache Airflow
potiuk commented on issue #7730: [AIRFLOW-7067] Pinned version of Apache Airflow URL: https://github.com/apache/airflow/pull/7730#issuecomment-599242772 And all green! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7730: [AIRFLOW-7067] Pinned version of Apache Airflow
codecov-io edited a comment on issue #7730: [AIRFLOW-7067] Pinned version of Apache Airflow URL: https://github.com/apache/airflow/pull/7730#issuecomment-599199404 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=h1) Report > Merging [#7730](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/271ee6477b80633049457557020a4f525f9135cd=desc) will **not change** coverage by `%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#7730 +/- ## === Coverage 86.93% 86.93% === Files 915 915 Lines 4415844158 === Hits3838738387 Misses 5771 5771 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/example\_dags/subdags/subdag.py](https://codecov.io/gh/apache/airflow/pull/7730/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3Mvc3ViZGFncy9zdWJkYWcucHk=) | `100.00% <0.00%> (ø)` | | | [airflow/example\_dags/example\_subdag\_operator.py](https://codecov.io/gh/apache/airflow/pull/7730/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9zdWJkYWdfb3BlcmF0b3IucHk=) | `100.00% <0.00%> (ø)` | | | [...w/example\_dags/example\_latest\_only\_with\_trigger.py](https://codecov.io/gh/apache/airflow/pull/7730/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9sYXRlc3Rfb25seV93aXRoX3RyaWdnZXIucHk=) | `100.00% <0.00%> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=footer). Last update [271ee64...bfd5d21](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7730: [AIRFLOW-7067] Pinned version of Apache Airflow
codecov-io edited a comment on issue #7730: [AIRFLOW-7067] Pinned version of Apache Airflow URL: https://github.com/apache/airflow/pull/7730#issuecomment-599199404 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=h1) Report > Merging [#7730](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/271ee6477b80633049457557020a4f525f9135cd=desc) will **not change** coverage by `%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#7730 +/- ## === Coverage 86.93% 86.93% === Files 915 915 Lines 4415844158 === Hits3838738387 Misses 5771 5771 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/example\_dags/subdags/subdag.py](https://codecov.io/gh/apache/airflow/pull/7730/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3Mvc3ViZGFncy9zdWJkYWcucHk=) | `100.00% <0.00%> (ø)` | | | [airflow/example\_dags/example\_subdag\_operator.py](https://codecov.io/gh/apache/airflow/pull/7730/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9zdWJkYWdfb3BlcmF0b3IucHk=) | `100.00% <0.00%> (ø)` | | | [...w/example\_dags/example\_latest\_only\_with\_trigger.py](https://codecov.io/gh/apache/airflow/pull/7730/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9sYXRlc3Rfb25seV93aXRoX3RyaWdnZXIucHk=) | `100.00% <0.00%> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=footer). Last update [271ee64...bfd5d21](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7730: [AIRFLOW-7067] Pinned version of Apache Airflow
codecov-io edited a comment on issue #7730: [AIRFLOW-7067] Pinned version of Apache Airflow URL: https://github.com/apache/airflow/pull/7730#issuecomment-599199404 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=h1) Report > Merging [#7730](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/271ee6477b80633049457557020a4f525f9135cd=desc) will **not change** coverage by `%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#7730 +/- ## === Coverage 86.93% 86.93% === Files 915 915 Lines 4415844158 === Hits3838738387 Misses 5771 5771 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/example\_dags/subdags/subdag.py](https://codecov.io/gh/apache/airflow/pull/7730/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3Mvc3ViZGFncy9zdWJkYWcucHk=) | `100.00% <0.00%> (ø)` | | | [airflow/example\_dags/example\_subdag\_operator.py](https://codecov.io/gh/apache/airflow/pull/7730/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9zdWJkYWdfb3BlcmF0b3IucHk=) | `100.00% <0.00%> (ø)` | | | [...w/example\_dags/example\_latest\_only\_with\_trigger.py](https://codecov.io/gh/apache/airflow/pull/7730/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9sYXRlc3Rfb25seV93aXRoX3RyaWdnZXIucHk=) | `100.00% <0.00%> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=footer). Last update [271ee64...bfd5d21](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7730: [AIRFLOW-7067] Pinned version of Apache Airflow
codecov-io edited a comment on issue #7730: [AIRFLOW-7067] Pinned version of Apache Airflow URL: https://github.com/apache/airflow/pull/7730#issuecomment-599199404 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=h1) Report > Merging [#7730](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/271ee6477b80633049457557020a4f525f9135cd=desc) will **not change** coverage by `%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#7730 +/- ## === Coverage 86.93% 86.93% === Files 915 915 Lines 4415844158 === Hits3838738387 Misses 5771 5771 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/example\_dags/subdags/subdag.py](https://codecov.io/gh/apache/airflow/pull/7730/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3Mvc3ViZGFncy9zdWJkYWcucHk=) | `100.00% <0.00%> (ø)` | | | [airflow/example\_dags/example\_subdag\_operator.py](https://codecov.io/gh/apache/airflow/pull/7730/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9zdWJkYWdfb3BlcmF0b3IucHk=) | `100.00% <0.00%> (ø)` | | | [...w/example\_dags/example\_latest\_only\_with\_trigger.py](https://codecov.io/gh/apache/airflow/pull/7730/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9sYXRlc3Rfb25seV93aXRoX3RyaWdnZXIucHk=) | `100.00% <0.00%> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=footer). Last update [271ee64...bfd5d21](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7730: [AIRFLOW-7067] Pinned version of Apache Airflow
potiuk commented on issue #7730: [AIRFLOW-7067] Pinned version of Apache Airflow URL: https://github.com/apache/airflow/pull/7730#issuecomment-599237136 All should be update and much nicer now @ashb ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7724: [AIRFLOW-1536] Inherit umask from parent process in daemon mode
potiuk commented on a change in pull request #7724: [AIRFLOW-1536] Inherit umask from parent process in daemon mode URL: https://github.com/apache/airflow/pull/7724#discussion_r392692428 ## File path: airflow/cli/commands/celery_command.py ## @@ -135,6 +135,7 @@ def worker(args): ctx = daemon.DaemonContext( files_preserve=[handle], +umask=cli_utils.get_umask(), Review comment: Just to give some example - there are problems with Dockerfile caching when git by default uses umask of the system rather than have predefined value. Different systems at different times had different default umask (for example debian/mint had it different and different versions) and that means that your daemon might behave differently for those different systems, This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7724: [AIRFLOW-1536] Inherit umask from parent process in daemon mode
potiuk commented on a change in pull request #7724: [AIRFLOW-1536] Inherit umask from parent process in daemon mode URL: https://github.com/apache/airflow/pull/7724#discussion_r392692428 ## File path: airflow/cli/commands/celery_command.py ## @@ -135,6 +135,7 @@ def worker(args): ctx = daemon.DaemonContext( files_preserve=[handle], +umask=cli_utils.get_umask(), Review comment: Just to give some example - there are problems with Dockerfile caching when git by default uses umask of the systems (for example debian/mint had it different and different versions) rather than have predefined value. Different systems at different times had different default umask and that means that your daemon might behave differently for those different systems, This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7724: [AIRFLOW-1536] Inherit umask from parent process in daemon mode
potiuk commented on a change in pull request #7724: [AIRFLOW-1536] Inherit umask from parent process in daemon mode URL: https://github.com/apache/airflow/pull/7724#discussion_r392692256 ## File path: airflow/cli/commands/celery_command.py ## @@ -135,6 +135,7 @@ def worker(args): ctx = daemon.DaemonContext( files_preserve=[handle], +umask=cli_utils.get_umask(), Review comment: The problem is that you do not know what the umask is in the parent process - it depends on the system umask setting. It's a nuance but I think it's a good practice to have it fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] stale[bot] commented on issue #7294: [AIRFLOW-6681][WIP] Load example DAGs for all providers
stale[bot] commented on issue #7294: [AIRFLOW-6681][WIP] Load example DAGs for all providers URL: https://github.com/apache/airflow/pull/7294#issuecomment-599236232 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7730: [AIRFLOW-7067] Pinned version of Apache Airflow
potiuk commented on a change in pull request #7730: [AIRFLOW-7067] Pinned version of Apache Airflow URL: https://github.com/apache/airflow/pull/7730#discussion_r392691267 ## File path: INSTALL ## @@ -1,22 +1,5 @@ # INSTALL / BUILD instructions for Apache Airflow -# [required] fetch the tarball and untar the source -# change into the directory that was untarred. - -# [optional] run Apache RAT (release audit tool) to validate license headers -# RAT docs here: https://creadur.apache.org/rat/. Requires Java and Apache Rat -java -jar apache-rat.jar -E ./.rat-excludes -d . - -# [optional] Airflow pulls in quite a lot of dependencies in order -# to connect to other services. You might want to test or run Airflow -# from a virtual env to make sure those dependencies are separated -# from your system wide versions -python -m my_env -source my_env/bin/activate - -# [required] building and installing -# by pip (preferred) -pip install . - -# or directly -python setup.py install +We used to have generic instructions for local installation for development here +However it's much easier to install and develop Airflow via Breeze development Review comment: Yep. I updated the description now and made even more valid with additional information and generated extras list This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on issue #7635: [AIRFLOW-5062] Allow ACL Header in S3Hook
feluelle commented on issue #7635: [AIRFLOW-5062] Allow ACL Header in S3Hook URL: https://github.com/apache/airflow/pull/7635#issuecomment-599234916 @retornam are you still working on this? Because there is also another PR #7726 which adds the same feature. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7731: [AIRFLOW-7068] Create EC2 Hook, Operator and Sensor
codecov-io commented on issue #7731: [AIRFLOW-7068] Create EC2 Hook, Operator and Sensor URL: https://github.com/apache/airflow/pull/7731#issuecomment-599234519 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7731?src=pr=h1) Report > Merging [#7731](https://codecov.io/gh/apache/airflow/pull/7731?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/0740dafa0a971b9dc05c6a1b3f034b869c827aa8=desc) will **decrease** coverage by `0.25%`. > The diff coverage is `95.94%`. [](https://codecov.io/gh/apache/airflow/pull/7731?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7731 +/- ## == - Coverage 86.93% 86.67% -0.26% == Files 915 918 +3 Lines 4415844232 +74 == - Hits3838738339 -48 - Misses 5771 5893 +122 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7731?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/providers/amazon/aws/operators/ec2.py](https://codecov.io/gh/apache/airflow/pull/7731/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9vcGVyYXRvcnMvZWMyLnB5) | `91.89% <91.89%> (ø)` | | | [airflow/providers/amazon/aws/hooks/ec2.py](https://codecov.io/gh/apache/airflow/pull/7731/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9ob29rcy9lYzIucHk=) | `100.00% <100.00%> (ø)` | | | [airflow/providers/amazon/aws/sensors/ec2.py](https://codecov.io/gh/apache/airflow/pull/7731/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9zZW5zb3JzL2VjMi5weQ==) | `100.00% <100.00%> (ø)` | | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7731/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7731/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7731/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: | | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7731/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.69% <0.00%> (-25.26%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7731/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0.00%> (-23.53%)` | :arrow_down: | | ... and [1 more](https://codecov.io/gh/apache/airflow/pull/7731/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7731?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7731?src=pr=footer). Last update [0740daf...5772f7a](https://codecov.io/gh/apache/airflow/pull/7731?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (AIRFLOW-4175) S3Hook load_file should support ACL policy parameter
[
https://issues.apache.org/jira/browse/AIRFLOW-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Uellendall closed AIRFLOW-4175.
-
Resolution: Fixed
> S3Hook load_file should support ACL policy parameter
>
>
> Key: AIRFLOW-4175
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4175
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
>Affects Versions: 1.10.2
>Reporter: Keith O'Brien
>Priority: Major
> Labels: gsoc, gsoc2020, mentor
>
> We have a use case where we are uploading files to an S3 bucket in a
> different AWS account to the one Airflow is running in. AWS S3 supports this
> situation using the pre canned ACL policy, specifically
> {{bucket-owner-full-control. }}
> However, the current implementations of the {{S3Hook.load_*}}() and
> {{S3Hook.copy_object}}() methods do not allow us to supply any ACL policy for
> the file being uploaded/copied to S3.
> It would be good to add another optional parameter to the {{S3Hook}} methods
> called {{acl_policy}} which would then be passed into the boto3 client method
> calls like so
>
> {code}
> # load_file
> ...
> if encrypt:
>extra_args['ServerSideEncryption'] = "AES256"
> if acl_policy:
>extra_args['ACL'] = acl_policy
> client.upload_file(filename, bucket_name, key, ExtraArgs=extra_args){code}
>
> {code}
> # load_bytes
> ...
> if encrypt:
>extra_args['ServerSideEncryption'] = "AES256"
> if acl_policy:
>extra_args['ACL'] = acl_policy
> client.upload_file(filename, bucket_name, key, ExtraArgs=extra_args){code}
> {code}
> # copy_object
> self.get_conn().copy_object(Bucket=dest_bucket_name,
>Key=dest_bucket_key,
>CopySource=CopySource,
>ACL=acl_policy)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4175) S3Hook load_file should support ACL policy parameter
[
https://issues.apache.org/jira/browse/AIRFLOW-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059727#comment-17059727
]
ASF GitHub Bot commented on AIRFLOW-4175:
-
feluelle commented on pull request #7726: [AIRFLOW-4175] S3Hook load_file
should support ACL policy parameter
URL: https://github.com/apache/airflow/pull/7726
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> S3Hook load_file should support ACL policy parameter
>
>
> Key: AIRFLOW-4175
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4175
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
>Affects Versions: 1.10.2
>Reporter: Keith O'Brien
>Priority: Major
> Labels: gsoc, gsoc2020, mentor
>
> We have a use case where we are uploading files to an S3 bucket in a
> different AWS account to the one Airflow is running in. AWS S3 supports this
> situation using the pre canned ACL policy, specifically
> {{bucket-owner-full-control. }}
> However, the current implementations of the {{S3Hook.load_*}}() and
> {{S3Hook.copy_object}}() methods do not allow us to supply any ACL policy for
> the file being uploaded/copied to S3.
> It would be good to add another optional parameter to the {{S3Hook}} methods
> called {{acl_policy}} which would then be passed into the boto3 client method
> calls like so
>
> {code}
> # load_file
> ...
> if encrypt:
>extra_args['ServerSideEncryption'] = "AES256"
> if acl_policy:
>extra_args['ACL'] = acl_policy
> client.upload_file(filename, bucket_name, key, ExtraArgs=extra_args){code}
>
> {code}
> # load_bytes
> ...
> if encrypt:
>extra_args['ServerSideEncryption'] = "AES256"
> if acl_policy:
>extra_args['ACL'] = acl_policy
> client.upload_file(filename, bucket_name, key, ExtraArgs=extra_args){code}
> {code}
> # copy_object
> self.get_conn().copy_object(Bucket=dest_bucket_name,
>Key=dest_bucket_key,
>CopySource=CopySource,
>ACL=acl_policy)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] feluelle commented on issue #7726: [AIRFLOW-4175] S3Hook load_file should support ACL policy parameter
feluelle commented on issue #7726: [AIRFLOW-4175] S3Hook load_file should support ACL policy parameter URL: https://github.com/apache/airflow/pull/7726#issuecomment-599233883 I am sorry, but this is a duplicate. #7635 already adds ACL Header. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle closed pull request #7726: [AIRFLOW-4175] S3Hook load_file should support ACL policy parameter
feluelle closed pull request #7726: [AIRFLOW-4175] S3Hook load_file should support ACL policy parameter URL: https://github.com/apache/airflow/pull/7726 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] CaptainCuddleCube commented on issue #7671: [AIRFLOW-6574] Adding private_environment to docker operator.
CaptainCuddleCube commented on issue #7671: [AIRFLOW-6574] Adding private_environment to docker operator. URL: https://github.com/apache/airflow/pull/7671#issuecomment-599233637 @feluelle Just following up on the state of this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] aggarwaldeepak commented on a change in pull request #7724: [AIRFLOW-1536] Inherit umask from parent process in daemon mode
aggarwaldeepak commented on a change in pull request #7724: [AIRFLOW-1536] Inherit umask from parent process in daemon mode URL: https://github.com/apache/airflow/pull/7724#discussion_r392688027 ## File path: airflow/cli/commands/celery_command.py ## @@ -135,6 +135,7 @@ def worker(args): ctx = daemon.DaemonContext( files_preserve=[handle], +umask=cli_utils.get_umask(), Review comment: I have referred few articles to get into the depth but could find why the value should not depend on parent process or what repercussions can it have if I inherit it from parent process . However all articles suggest it to set to a predefined value. I will make the value configurable with a reasonable default. - https://smnd.sk/anino/programming/c/c-faq_comp.unix.programmer/faq_2.html - http://www.enderunix.org/docs/eng/daemon.php This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mustafagok opened a new pull request #7731: [AIRFLOW-7068] Create EC2 Hook, Operator and Sensor
mustafagok opened a new pull request #7731: [AIRFLOW-7068] Create EC2 Hook, Operator and Sensor URL: https://github.com/apache/airflow/pull/7731 Create EC2 Hook, Operator and Sensor - New hook to interact with AWS EC2 Service. - New operator to manage AWS EC2 instance and to change instance state by applying given operation using boto3. - New sensor to check the state of the AWS EC2 instance until state of the instance become equal to the target state. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-7068) Create EC2 Hook, Operator and Sensor
[ https://issues.apache.org/jira/browse/AIRFLOW-7068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059709#comment-17059709 ] ASF GitHub Bot commented on AIRFLOW-7068: - mustafagok commented on pull request #7731: [AIRFLOW-7068] Create EC2 Hook, Operator and Sensor URL: https://github.com/apache/airflow/pull/7731 Create EC2 Hook, Operator and Sensor - New hook to interact with AWS EC2 Service. - New operator to manage AWS EC2 instance and to change instance state by applying given operation using boto3. - New sensor to check the state of the AWS EC2 instance until state of the instance become equal to the target state. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Create EC2 Hook, Operator and Sensor > > > Key: AIRFLOW-7068 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7068 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 1.10.9 >Reporter: Mustafa Gök >Assignee: Mustafa Gök >Priority: Minor > > * New hook to interact with AWS EC2 Service. > * New operator to manage AWS EC2 instance and to change instance state by > applying given operation using boto3. > * New sensor to check the state of the AWS EC2 instance until state of the > instance become equal to the target state. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk commented on issue #7730: [AIRFLOW-7067] Pinned version of Apache Airflow
potiuk commented on issue #7730: [AIRFLOW-7067] Pinned version of Apache Airflow URL: https://github.com/apache/airflow/pull/7730#issuecomment-599221736 BTW. I am going to simplify it even more - no need to store fixed requirements in setup.py This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7730: [AIRFLOW-7067] Pinned version of Apache Airflow
potiuk commented on a change in pull request #7730: [AIRFLOW-7067] Pinned version of Apache Airflow URL: https://github.com/apache/airflow/pull/7730#discussion_r392682021 ## File path: INSTALL ## @@ -1,22 +1,5 @@ # INSTALL / BUILD instructions for Apache Airflow -# [required] fetch the tarball and untar the source -# change into the directory that was untarred. - -# [optional] run Apache RAT (release audit tool) to validate license headers -# RAT docs here: https://creadur.apache.org/rat/. Requires Java and Apache Rat -java -jar apache-rat.jar -E ./.rat-excludes -d . - -# [optional] Airflow pulls in quite a lot of dependencies in order -# to connect to other services. You might want to test or run Airflow -# from a virtual env to make sure those dependencies are separated -# from your system wide versions -python -m my_env -source my_env/bin/activate - -# [required] building and installing -# by pip (preferred) -pip install . - -# or directly -python setup.py install +We used to have generic instructions for local installation for development here +However it's much easier to install and develop Airflow via Breeze development Review comment: It was recommending to install -e . :). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7730: [AIRFLOW-7067] Pinned version of Apache Airflow
ashb commented on a change in pull request #7730: [AIRFLOW-7067] Pinned version of Apache Airflow URL: https://github.com/apache/airflow/pull/7730#discussion_r392679015 ## File path: INSTALL ## @@ -1,22 +1,5 @@ # INSTALL / BUILD instructions for Apache Airflow -# [required] fetch the tarball and untar the source -# change into the directory that was untarred. - -# [optional] run Apache RAT (release audit tool) to validate license headers -# RAT docs here: https://creadur.apache.org/rat/. Requires Java and Apache Rat -java -jar apache-rat.jar -E ./.rat-excludes -d . - -# [optional] Airflow pulls in quite a lot of dependencies in order -# to connect to other services. You might want to test or run Airflow -# from a virtual env to make sure those dependencies are separated -# from your system wide versions -python -m my_env -source my_env/bin/activate - -# [required] building and installing -# by pip (preferred) -pip install . - -# or directly -python setup.py install +We used to have generic instructions for local installation for development here Review comment: I think this was first added to follow Apache guide-lines, let me see if I can find that again This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7730: [AIRFLOW-7067] Pinned version of Apache Airflow
ashb commented on a change in pull request #7730: [AIRFLOW-7067] Pinned version of Apache Airflow URL: https://github.com/apache/airflow/pull/7730#discussion_r392679061 ## File path: INSTALL ## @@ -1,22 +1,5 @@ # INSTALL / BUILD instructions for Apache Airflow -# [required] fetch the tarball and untar the source -# change into the directory that was untarred. - -# [optional] run Apache RAT (release audit tool) to validate license headers -# RAT docs here: https://creadur.apache.org/rat/. Requires Java and Apache Rat -java -jar apache-rat.jar -E ./.rat-excludes -d . - -# [optional] Airflow pulls in quite a lot of dependencies in order -# to connect to other services. You might want to test or run Airflow -# from a virtual env to make sure those dependencies are separated -# from your system wide versions -python -m my_env -source my_env/bin/activate - -# [required] building and installing -# by pip (preferred) -pip install . - -# or directly -python setup.py install +We used to have generic instructions for local installation for development here +However it's much easier to install and develop Airflow via Breeze development Review comment: Install is for end users, so breeze isn't appropriate. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] OmairK commented on issue #7726: [AIRFLOW-4175] S3Hook load_file should support ACL policy parameter
OmairK commented on issue #7726: [AIRFLOW-4175] S3Hook load_file should support ACL policy parameter URL: https://github.com/apache/airflow/pull/7726#issuecomment-599215817 > It looks good. But one thing - can you also extend the tests to verify that the default "private" acl has also the right permissions ? You are talking about the copy_object() hook ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work started] (AIRFLOW-7068) Create EC2 Hook, Operator and Sensor
[ https://issues.apache.org/jira/browse/AIRFLOW-7068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-7068 started by Mustafa Gök. > Create EC2 Hook, Operator and Sensor > > > Key: AIRFLOW-7068 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7068 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 1.10.9 >Reporter: Mustafa Gök >Assignee: Mustafa Gök >Priority: Minor > > * New hook to interact with AWS EC2 Service. > * New operator to manage AWS EC2 instance and to change instance state by > applying given operation using boto3. > * New sensor to check the state of the AWS EC2 instance until state of the > instance become equal to the target state. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-7068) Create EC2 Hook, Operator and Sensor
Mustafa Gök created AIRFLOW-7068: Summary: Create EC2 Hook, Operator and Sensor Key: AIRFLOW-7068 URL: https://issues.apache.org/jira/browse/AIRFLOW-7068 Project: Apache Airflow Issue Type: Improvement Components: aws Affects Versions: 1.10.9 Reporter: Mustafa Gök Assignee: Mustafa Gök * New hook to interact with AWS EC2 Service. * New operator to manage AWS EC2 instance and to change instance state by applying given operation using boto3. * New sensor to check the state of the AWS EC2 instance until state of the instance become equal to the target state. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6518) Task did not retry when there was temporary metastore db connectivity loss
[
https://issues.apache.org/jira/browse/AIRFLOW-6518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059645#comment-17059645
]
t oo commented on AIRFLOW-6518:
---
depends on what level of fault-tolerance we want to achieve from airflow.
personally I think a 1 second network 'blip' is uncommon but not an unheard of
/ rare scenario, perhaps once every 6 months? yes, currently there are bigger
fish to fry
> Task did not retry when there was temporary metastore db connectivity loss
> --
>
> Key: AIRFLOW-6518
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6518
> Project: Apache Airflow
> Issue Type: Bug
> Components: database, scheduler
>Affects Versions: 1.10.6
>Reporter: t oo
>Priority: Major
>
> My DAG has got retries configured at the task level. I started a dagrun, then
> while a task was running the metastore db crashed, the task failed, but the
> dagrun did not attempt to retry the task (even though task retries are
> configured!), db recovered 3 seconds after the task failed, instead the
> dagrun went to FAILED state.
> *Last part of log of TaskInstance:*
> [2020-01-08 17:34:46,301] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk Traceback (most recent call last):
> [2020-01-08 17:34:46,301] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk File "/home/ec2-user/venv/bin/airflow", line 37, in
> [2020-01-08 17:34:46,302] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk args.func(args)
> [2020-01-08 17:34:46,302] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk File
> "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/utils/cli.py",
> line 74, in wrapper
> [2020-01-08 17:34:46,302] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk return f(*args, **kwargs)
> [2020-01-08 17:34:46,302] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk File
> "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/bin/cli.py",
> line 551, in run
> [2020-01-08 17:34:46,302] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk _run(args, dag, ti)
> [2020-01-08 17:34:46,302] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk File
> "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/bin/cli.py",
> line 469, in _run
> [2020-01-08 17:34:46,302] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk pool=args.pool,
> [2020-01-08 17:34:46,302] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk File
> "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/utils/db.py",
> line 74, in wrapper
> [2020-01-08 17:34:46,302] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk return func(*args, **kwargs)
> [2020-01-08 17:34:46,302] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk File
> "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/models/taskinstance.py",
> line 962, in _run_raw_task
> [2020-01-08 17:34:46,302] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk self.refresh_from_db()
> [2020-01-08 17:34:46,303] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk File
> "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/utils/db.py",
> line 74, in wrapper
> [2020-01-08 17:34:46,303] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk return func(*args, **kwargs)
> [2020-01-08 17:34:46,303] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk File
> "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/models/taskinstance.py",
> line 461, in refresh_from_db
> [2020-01-08 17:34:46,303] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk ti = qry.first()
> [2020-01-08 17:34:46,303] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk File
> "/home/ec2-user/venv/local/lib64/python2.7/site-packages/sqlalchemy/orm/query.py",
> line 3265, in first
> [2020-01-08 17:34:46,303] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk ret = list(self[0:1])
> [2020-01-08 17:34:46,303] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk File
> "/home/ec2-user/venv/local/lib64/python2.7/site-packages/sqlalchemy/orm/query.py",
> line 3043, in __getitem__
> [2020-01-08 17:34:46,303] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk return list(res)
> [2020-01-08 17:34:46,303] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk File
> "/home/ec2-user/venv/local/lib64/python2.7/site-packages/sqlalchemy/orm/query.py",
> line 3367, in __iter__
> [2020-01-08 17:34:46,303] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk return self._execute_and_instances(context)
> [2020-01-08 17:34:46,304] {base_task_runner.py:115} INFO - Job 34662: Subtask
>
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #7728: [AIRFLOW-5610] Add ability to specify multiple objects to copy in GCSToGCSOperator
ephraimbuddy commented on a change in pull request #7728: [AIRFLOW-5610] Add ability to specify multiple objects to copy in GCSToGCSOperator URL: https://github.com/apache/airflow/pull/7728#discussion_r392667594 ## File path: airflow/providers/google/cloud/operators/gcs_to_gcs.py ## @@ -40,32 +37,27 @@ class GCSToGCSOperator(BaseOperator): :param source_bucket: The source Google Cloud Storage bucket where the object is. (templated) :type source_bucket: str -:param source_object: The source name of the object to copy in the Google cloud +:param source_objects: A list of prefix of the objects to copy in the Google cloud storage bucket. (templated) -You can use only one wildcard for objects (filenames) within your -bucket. The wildcard can appear inside the object name or at the -end of the object name. Appending a wildcard to the bucket name is -unsupported. -:type source_object: str +:type source_objects: List[str] :param destination_bucket: The destination Google Cloud Storage bucket where the object should be. If the destination_bucket is None, it defaults to source_bucket. (templated) :type destination_bucket: str :param destination_object: The destination name of the object in the destination Google Cloud Storage bucket. (templated) -If a wildcard is supplied in the source_object argument, this is the -prefix that will be prepended to the final destination objects' paths. -Note that the source path's part before the wildcard will be removed; -if it needs to be retained it should be appended to destination_object. -For example, with prefix ``foo/*`` and destination_object ``blah/``, the -file ``foo/baz`` will be copied to ``blah/baz``; to retain the prefix write -the destination_object as e.g. ``blah/foo``, in which case the copied file -will be named ``blah/foo/baz``. +If destination object is not specified, then it defaults to each of the source objects. +For example, if source_objects = ['foo/sales','bah/inventory'], then destination will be +'foo/sales' and 'bah/inventory' if destination_object is not specified. :type destination_object: str :param move_object: When move object is True, the object is moved instead of copied to the new location. This is the equivalent of a mv command as opposed to a cp command. :type move_object: bool +:type delimiter: str Review comment: Ok. Thanks very much for the review. I'm thinking of removing it entirely and add back `google_cloud_storage_conn_id` which I removed. This is for backward compatibility. The arguments are already 10 so leaving the both arguments will make tests not to pass. If `delimiter` is removed, then search will now be based on source object wildcard. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #7728: [AIRFLOW-5610] Add ability to specify multiple objects to copy in GCSToGCSOperator
ephraimbuddy commented on a change in pull request #7728: [AIRFLOW-5610] Add
ability to specify multiple objects to copy in GCSToGCSOperator
URL: https://github.com/apache/airflow/pull/7728#discussion_r392667627
##
File path: airflow/providers/google/cloud/operators/gcs_to_gcs.py
##
@@ -169,30 +158,25 @@ def execute(self, context):
'destination_bucket is None. Defaulting it to source_bucket
(%s)',
self.source_bucket)
self.destination_bucket = self.source_bucket
-
-if WILDCARD in self.source_object:
-total_wildcards = self.source_object.count(WILDCARD)
-if total_wildcards > 1:
-error_msg = "Only one wildcard '*' is allowed in source_object
parameter. " \
-"Found {} in {}.".format(total_wildcards,
self.source_object)
-
-raise AirflowException(error_msg)
-
-prefix, delimiter = self.source_object.split(WILDCARD, 1)
-objects = hook.list(self.source_bucket, prefix=prefix,
delimiter=delimiter)
-
+if not all(isinstance(item, str) for item in self.source_objects):
+raise AirflowException('At least, one of the `objects` in the
`source_objects` is not a string')
Review comment:
Ok. Noted
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #7728: [AIRFLOW-5610] Add ability to specify multiple objects to copy in GCSToGCSOperator
ephraimbuddy commented on a change in pull request #7728: [AIRFLOW-5610] Add ability to specify multiple objects to copy in GCSToGCSOperator URL: https://github.com/apache/airflow/pull/7728#discussion_r392667615 ## File path: airflow/providers/google/cloud/operators/gcs_to_gcs.py ## @@ -40,32 +37,27 @@ class GCSToGCSOperator(BaseOperator): :param source_bucket: The source Google Cloud Storage bucket where the object is. (templated) :type source_bucket: str -:param source_object: The source name of the object to copy in the Google cloud +:param source_objects: A list of prefix of the objects to copy in the Google cloud Review comment: Ok. Noted. Thanks a lot This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-7066) Use sphinx including syntax in concepts.rst
[ https://issues.apache.org/jira/browse/AIRFLOW-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059629#comment-17059629 ] ASF GitHub Bot commented on AIRFLOW-7066: - potiuk commented on pull request #7729: [AIRFLOW-7066] Use sphinx syntax in concepts.rst URL: https://github.com/apache/airflow/pull/7729 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Use sphinx including syntax in concepts.rst > --- > > Key: AIRFLOW-7066 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7066 > Project: Apache Airflow > Issue Type: Improvement > Components: documentation >Affects Versions: 1.10.9 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Major > > Use sphinx syntax if possible instead of python code -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk merged pull request #7729: [AIRFLOW-7066] Use sphinx syntax in concepts.rst
potiuk merged pull request #7729: [AIRFLOW-7066] Use sphinx syntax in concepts.rst URL: https://github.com/apache/airflow/pull/7729 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7729: [AIRFLOW-7066] Use sphinx syntax in concepts.rst
potiuk commented on issue #7729: [AIRFLOW-7066] Use sphinx syntax in concepts.rst URL: https://github.com/apache/airflow/pull/7729#issuecomment-599201994 Thanks @zhongjiajie ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #7728: [AIRFLOW-5610] Add ability to specify multiple objects to copy in GCSToGCSOperator
ephraimbuddy commented on a change in pull request #7728: [AIRFLOW-5610] Add ability to specify multiple objects to copy in GCSToGCSOperator URL: https://github.com/apache/airflow/pull/7728#discussion_r392667594 ## File path: airflow/providers/google/cloud/operators/gcs_to_gcs.py ## @@ -40,32 +37,27 @@ class GCSToGCSOperator(BaseOperator): :param source_bucket: The source Google Cloud Storage bucket where the object is. (templated) :type source_bucket: str -:param source_object: The source name of the object to copy in the Google cloud +:param source_objects: A list of prefix of the objects to copy in the Google cloud storage bucket. (templated) -You can use only one wildcard for objects (filenames) within your -bucket. The wildcard can appear inside the object name or at the -end of the object name. Appending a wildcard to the bucket name is -unsupported. -:type source_object: str +:type source_objects: List[str] :param destination_bucket: The destination Google Cloud Storage bucket where the object should be. If the destination_bucket is None, it defaults to source_bucket. (templated) :type destination_bucket: str :param destination_object: The destination name of the object in the destination Google Cloud Storage bucket. (templated) -If a wildcard is supplied in the source_object argument, this is the -prefix that will be prepended to the final destination objects' paths. -Note that the source path's part before the wildcard will be removed; -if it needs to be retained it should be appended to destination_object. -For example, with prefix ``foo/*`` and destination_object ``blah/``, the -file ``foo/baz`` will be copied to ``blah/baz``; to retain the prefix write -the destination_object as e.g. ``blah/foo``, in which case the copied file -will be named ``blah/foo/baz``. +If destination object is not specified, then it defaults to each of the source objects. +For example, if source_objects = ['foo/sales','bah/inventory'], then destination will be +'foo/sales' and 'bah/inventory' if destination_object is not specified. :type destination_object: str :param move_object: When move object is True, the object is moved instead of copied to the new location. This is the equivalent of a mv command as opposed to a cp command. :type move_object: bool +:type delimiter: str Review comment: Ok. Thanks very much for the review. I'm thinking of removing it entirely and add back `google_cloud_storage_conn_id` which I removed. This is for backward compatibility. The arguments are already 10 so leaving the both arguments will make tests not to pass. If `delimiter` is removed, then search will now be based on source object wildcard. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7729: [AIRFLOW-7066] Use sphinx syntax in concepts.rst
potiuk commented on issue #7729: [AIRFLOW-7066] Use sphinx syntax in concepts.rst URL: https://github.com/apache/airflow/pull/7729#issuecomment-599201940 > BTW, I have some question could not understand, in my PR #7709, raise pylint error three time, but `airflow/www/views.py` is in `pylint_todo.txt`, which mean will not test by pylint, also test folder will not test by pylint. should where the pylint error? (I only change two files) > > I test in breeze using `pylint airflow/www/views.py` and `pylint test/www/views.py` only `airflow/www/views.py` have some error hint(but it in pylint_todo.txt) > @exploy It is the test/ that is failing. We have two separate pre-commits - pylint and pylint-test . The test one is failing in the other build and should be fixed there. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7730: [AIRFLOW-7067] Pinned version of Apache Airflow
codecov-io edited a comment on issue #7730: [AIRFLOW-7067] Pinned version of Apache Airflow URL: https://github.com/apache/airflow/pull/7730#issuecomment-599199404 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=h1) Report > Merging [#7730](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/271ee6477b80633049457557020a4f525f9135cd=desc) will **not change** coverage by `%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#7730 +/- ## === Coverage 86.93% 86.93% === Files 915 915 Lines 4415844158 === Hits3838738387 Misses 5771 5771 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=footer). Last update [271ee64...6d36e13](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7730: [AIRFLOW-7067] Pinned version of Apache Airflow
codecov-io edited a comment on issue #7730: [AIRFLOW-7067] Pinned version of Apache Airflow URL: https://github.com/apache/airflow/pull/7730#issuecomment-599199404 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=h1) Report > Merging [#7730](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/271ee6477b80633049457557020a4f525f9135cd=desc) will **not change** coverage by `%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#7730 +/- ## === Coverage 86.93% 86.93% === Files 915 915 Lines 4415844158 === Hits3838738387 Misses 5771 5771 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=footer). Last update [271ee64...6d36e13](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7730: [AIRFLOW-7067] Pinned version of Apache Airflow
codecov-io edited a comment on issue #7730: [AIRFLOW-7067] Pinned version of Apache Airflow URL: https://github.com/apache/airflow/pull/7730#issuecomment-599199404 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=h1) Report > Merging [#7730](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/271ee6477b80633049457557020a4f525f9135cd=desc) will **not change** coverage by `%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#7730 +/- ## === Coverage 86.93% 86.93% === Files 915 915 Lines 4415844158 === Hits3838738387 Misses 5771 5771 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=footer). Last update [271ee64...6d36e13](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7730: [AIRFLOW-7067] Pinned version of Apache Airflow
codecov-io edited a comment on issue #7730: [AIRFLOW-7067] Pinned version of Apache Airflow URL: https://github.com/apache/airflow/pull/7730#issuecomment-599199404 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=h1) Report > Merging [#7730](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/271ee6477b80633049457557020a4f525f9135cd=desc) will **not change** coverage by `%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#7730 +/- ## === Coverage 86.93% 86.93% === Files 915 915 Lines 4415844158 === Hits3838738387 Misses 5771 5771 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=footer). Last update [271ee64...5b8c573](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7730: [AIRFLOW-7067] Pinned version of Apache Airflow
codecov-io edited a comment on issue #7730: [AIRFLOW-7067] Pinned version of Apache Airflow URL: https://github.com/apache/airflow/pull/7730#issuecomment-599199404 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=h1) Report > Merging [#7730](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/271ee6477b80633049457557020a4f525f9135cd?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#7730 +/- ## === Coverage 86.93% 86.93% === Files 915 915 Lines 4415844158 === Hits3838738387 Misses 5771 5771 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=footer). Last update [271ee64...5b8c573](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7730: [AIRFLOW-7067] Pinned version of Apache Airflow
codecov-io commented on issue #7730: [AIRFLOW-7067] Pinned version of Apache Airflow URL: https://github.com/apache/airflow/pull/7730#issuecomment-599199404 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=h1) Report > Merging [#7730](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/271ee6477b80633049457557020a4f525f9135cd?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#7730 +/- ## === Coverage 86.93% 86.93% === Files 915 915 Lines 4415844158 === Hits3838738387 Misses 5771 5771 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=footer). Last update [271ee64...5b8c573](https://codecov.io/gh/apache/airflow/pull/7730?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6533) Add cli commands to manage config
[ https://issues.apache.org/jira/browse/AIRFLOW-6533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059612#comment-17059612 ] ASF GitHub Bot commented on AIRFLOW-6533: - stale[bot] commented on pull request #7130: [AIRFLOW-6533] Add cli commands to manage config URL: https://github.com/apache/airflow/pull/7130 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add cli commands to manage config > - > > Key: AIRFLOW-6533 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6533 > Project: Apache Airflow > Issue Type: New Feature > Components: cli >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] stale[bot] closed pull request #7130: [AIRFLOW-6533] Add cli commands to manage config
stale[bot] closed pull request #7130: [AIRFLOW-6533] Add cli commands to manage config URL: https://github.com/apache/airflow/pull/7130 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-7067) Add apache-airflow-pinned version
[ https://issues.apache.org/jira/browse/AIRFLOW-7067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059610#comment-17059610 ] ASF GitHub Bot commented on AIRFLOW-7067: - potiuk commented on pull request #7730: [AIRFLOW-7067] Pinned version of Apache Airflow URL: https://github.com/apache/airflow/pull/7730 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add apache-airflow-pinned version > - > > Key: AIRFLOW-7067 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7067 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 2.0.0, 1.10.9 >Reporter: Jarek Potiuk >Priority: Major > > For official docker image we need to have fixed set of requirements so that > we know rebuilding the image can be reproducible. > We need a -pinned version of apache-airflow for that. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk opened a new pull request #7730: [AIRFLOW-7067] Pinned version of Apache Airflow
potiuk opened a new pull request #7730: [AIRFLOW-7067] Pinned version of Apache Airflow URL: https://github.com/apache/airflow/pull/7730 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-7067) Add apache-airflow-pinned version
Jarek Potiuk created AIRFLOW-7067: - Summary: Add apache-airflow-pinned version Key: AIRFLOW-7067 URL: https://issues.apache.org/jira/browse/AIRFLOW-7067 Project: Apache Airflow Issue Type: Improvement Components: ci Affects Versions: 1.10.9, 2.0.0 Reporter: Jarek Potiuk For official docker image we need to have fixed set of requirements so that we know rebuilding the image can be reproducible. We need a -pinned version of apache-airflow for that. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #7729: [AIRFLOW-7066] Use sphinx syntax in concepts.rst
codecov-io edited a comment on issue #7729: [AIRFLOW-7066] Use sphinx syntax in concepts.rst URL: https://github.com/apache/airflow/pull/7729#issuecomment-599191432 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7729?src=pr=h1) Report > Merging [#7729](https://codecov.io/gh/apache/airflow/pull/7729?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/271ee6477b80633049457557020a4f525f9135cd=desc) will **not change** coverage by `%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7729?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#7729 +/- ## === Coverage 86.93% 86.93% === Files 915 915 Lines 4415844158 === Hits3838738387 Misses 5771 5771 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7729?src=pr=tree) | Coverage Δ | | |---|---|---| | [...w/example\_dags/example\_latest\_only\_with\_trigger.py](https://codecov.io/gh/apache/airflow/pull/7729/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9sYXRlc3Rfb25seV93aXRoX3RyaWdnZXIucHk=) | `100.00% <ø> (ø)` | | | [airflow/example\_dags/example\_subdag\_operator.py](https://codecov.io/gh/apache/airflow/pull/7729/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9zdWJkYWdfb3BlcmF0b3IucHk=) | `100.00% <ø> (ø)` | | | [airflow/example\_dags/subdags/subdag.py](https://codecov.io/gh/apache/airflow/pull/7729/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3Mvc3ViZGFncy9zdWJkYWcucHk=) | `100.00% <ø> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7729?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7729?src=pr=footer). Last update [271ee64...4eaaba8](https://codecov.io/gh/apache/airflow/pull/7729?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7729: [AIRFLOW-7066] Use sphinx syntax in concepts.rst
codecov-io commented on issue #7729: [AIRFLOW-7066] Use sphinx syntax in concepts.rst URL: https://github.com/apache/airflow/pull/7729#issuecomment-599191432 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7729?src=pr=h1) Report > Merging [#7729](https://codecov.io/gh/apache/airflow/pull/7729?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/271ee6477b80633049457557020a4f525f9135cd=desc) will **not change** coverage by `%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7729?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#7729 +/- ## === Coverage 86.93% 86.93% === Files 915 915 Lines 4415844158 === Hits3838738387 Misses 5771 5771 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7729?src=pr=tree) | Coverage Δ | | |---|---|---| | [...w/example\_dags/example\_latest\_only\_with\_trigger.py](https://codecov.io/gh/apache/airflow/pull/7729/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9sYXRlc3Rfb25seV93aXRoX3RyaWdnZXIucHk=) | `100.00% <ø> (ø)` | | | [airflow/example\_dags/example\_subdag\_operator.py](https://codecov.io/gh/apache/airflow/pull/7729/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9zdWJkYWdfb3BlcmF0b3IucHk=) | `100.00% <ø> (ø)` | | | [airflow/example\_dags/subdags/subdag.py](https://codecov.io/gh/apache/airflow/pull/7729/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3Mvc3ViZGFncy9zdWJkYWcucHk=) | `100.00% <ø> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7729?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7729?src=pr=footer). Last update [271ee64...4eaaba8](https://codecov.io/gh/apache/airflow/pull/7729?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7683: [AIRFLOW-7033] Change dag and task state meanwhile
codecov-io edited a comment on issue #7683: [AIRFLOW-7033] Change dag and task state meanwhile URL: https://github.com/apache/airflow/pull/7683#issuecomment-597665730 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=h1) Report > Merging [#7683](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/271ee6477b80633049457557020a4f525f9135cd=desc) will **decrease** coverage by `0.00%`. > The diff coverage is `91.30%`. [](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7683 +/- ## == - Coverage 86.93% 86.92% -0.01% == Files 915 915 Lines 4415844168 +10 == + Hits3838738395 +8 - Misses 5771 5773 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/api/common/experimental/mark\_tasks.py](https://codecov.io/gh/apache/airflow/pull/7683/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9tYXJrX3Rhc2tzLnB5) | `94.51% <90.47%> (-1.01%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/7683/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.30% <100.00%> (+0.03%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=footer). Last update [271ee64...035c685](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7683: [AIRFLOW-7033] Change dag and task state meanwhile
codecov-io edited a comment on issue #7683: [AIRFLOW-7033] Change dag and task state meanwhile URL: https://github.com/apache/airflow/pull/7683#issuecomment-597665730 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=h1) Report > Merging [#7683](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/271ee6477b80633049457557020a4f525f9135cd=desc) will **decrease** coverage by `0.00%`. > The diff coverage is `91.30%`. [](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7683 +/- ## == - Coverage 86.93% 86.92% -0.01% == Files 915 915 Lines 4415844168 +10 == + Hits3838738395 +8 - Misses 5771 5773 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/api/common/experimental/mark\_tasks.py](https://codecov.io/gh/apache/airflow/pull/7683/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9tYXJrX3Rhc2tzLnB5) | `94.51% <90.47%> (-1.01%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/7683/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.30% <100.00%> (+0.03%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=footer). Last update [271ee64...035c685](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7683: [AIRFLOW-7033] Change dag and task state meanwhile
codecov-io edited a comment on issue #7683: [AIRFLOW-7033] Change dag and task state meanwhile URL: https://github.com/apache/airflow/pull/7683#issuecomment-597665730 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=h1) Report > Merging [#7683](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/271ee6477b80633049457557020a4f525f9135cd=desc) will **decrease** coverage by `0.00%`. > The diff coverage is `91.30%`. [](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7683 +/- ## == - Coverage 86.93% 86.92% -0.01% == Files 915 915 Lines 4415844168 +10 == + Hits3838738395 +8 - Misses 5771 5773 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/api/common/experimental/mark\_tasks.py](https://codecov.io/gh/apache/airflow/pull/7683/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9tYXJrX3Rhc2tzLnB5) | `94.51% <90.47%> (-1.01%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/7683/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.30% <100.00%> (+0.03%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=footer). Last update [271ee64...035c685](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7683: [AIRFLOW-7033] Change dag and task state meanwhile
codecov-io edited a comment on issue #7683: [AIRFLOW-7033] Change dag and task state meanwhile URL: https://github.com/apache/airflow/pull/7683#issuecomment-597665730 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=h1) Report > Merging [#7683](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/271ee6477b80633049457557020a4f525f9135cd=desc) will **decrease** coverage by `0.00%`. > The diff coverage is `91.30%`. [](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7683 +/- ## == - Coverage 86.93% 86.92% -0.01% == Files 915 915 Lines 4415844168 +10 == + Hits3838738395 +8 - Misses 5771 5773 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/api/common/experimental/mark\_tasks.py](https://codecov.io/gh/apache/airflow/pull/7683/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9tYXJrX3Rhc2tzLnB5) | `94.51% <90.47%> (-1.01%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/7683/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.30% <100.00%> (+0.03%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=footer). Last update [271ee64...035c685](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7683: [AIRFLOW-7033] Change dag and task state meanwhile
codecov-io edited a comment on issue #7683: [AIRFLOW-7033] Change dag and task state meanwhile URL: https://github.com/apache/airflow/pull/7683#issuecomment-597665730 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=h1) Report > Merging [#7683](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/271ee6477b80633049457557020a4f525f9135cd=desc) will **decrease** coverage by `0.00%`. > The diff coverage is `91.30%`. [](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7683 +/- ## == - Coverage 86.93% 86.92% -0.01% == Files 915 915 Lines 4415844168 +10 == + Hits3838738395 +8 - Misses 5771 5773 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/api/common/experimental/mark\_tasks.py](https://codecov.io/gh/apache/airflow/pull/7683/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9tYXJrX3Rhc2tzLnB5) | `94.51% <90.47%> (-1.01%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/7683/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.30% <100.00%> (+0.03%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=footer). Last update [271ee64...035c685](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7593: [AIRFLOW-6959] Use NULL as dag.description default value
codecov-io edited a comment on issue #7593: [AIRFLOW-6959] Use NULL as dag.description default value URL: https://github.com/apache/airflow/pull/7593#issuecomment-592953279 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7593?src=pr=h1) Report > Merging [#7593](https://codecov.io/gh/apache/airflow/pull/7593?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/271ee6477b80633049457557020a4f525f9135cd=desc) will **not change** coverage by `%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7593?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#7593 +/- ## === Coverage 86.93% 86.93% === Files 915 915 Lines 4415844158 === Hits3838738387 Misses 5771 5771 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7593?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7593/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `91.21% <ø> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7593?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7593?src=pr=footer). Last update [271ee64...17b608f](https://codecov.io/gh/apache/airflow/pull/7593?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7683: [AIRFLOW-7033] Change dag and task state meanwhile
codecov-io edited a comment on issue #7683: [AIRFLOW-7033] Change dag and task state meanwhile URL: https://github.com/apache/airflow/pull/7683#issuecomment-597665730 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=h1) Report > Merging [#7683](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/271ee6477b80633049457557020a4f525f9135cd=desc) will **decrease** coverage by `0.06%`. > The diff coverage is `91.30%`. [](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7683 +/- ## == - Coverage 86.93% 86.87% -0.07% == Files 915 915 Lines 4415844168 +10 == - Hits3838738369 -18 - Misses 5771 5799 +28 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/api/common/experimental/mark\_tasks.py](https://codecov.io/gh/apache/airflow/pull/7683/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9tYXJrX3Rhc2tzLnB5) | `94.51% <90.47%> (-1.01%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/7683/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.30% <100.00%> (+0.03%)` | :arrow_up: | | [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7683/diff?src=pr=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `30.43% <0.00%> (-45.66%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7683/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `90.98% <0.00%> (-1.13%)` | :arrow_down: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7683/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `90.73% <0.00%> (-0.15%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=footer). Last update [271ee64...035c685](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] zhongjiajie edited a comment on issue #7729: [AIRFLOW-7066] Use sphinx syntax in concepts.rst
zhongjiajie edited a comment on issue #7729: [AIRFLOW-7066] Use sphinx syntax in concepts.rst URL: https://github.com/apache/airflow/pull/7729#issuecomment-599189560 BTW, I have some question could not understand, in my PR https://github.com/apache/airflow/pull/7709, raise pylint error three time, but `airflow/www/views.py` is in `pylint_todo.txt`, which mean will not test by pylint, also test folder will not test by pylint. should where the pylint error? (I only change two files) I test in breeze using `pylint airflow/www/views.py` and `pylint test/www/views.py` only `airflow/www/views.py` have some error hint(but it in pylint_todo.txt) @potiuk This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7683: [AIRFLOW-7033] Change dag and task state meanwhile
codecov-io edited a comment on issue #7683: [AIRFLOW-7033] Change dag and task state meanwhile URL: https://github.com/apache/airflow/pull/7683#issuecomment-597665730 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=h1) Report > Merging [#7683](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/271ee6477b80633049457557020a4f525f9135cd=desc) will **decrease** coverage by `0.00%`. > The diff coverage is `91.30%`. [](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7683 +/- ## == - Coverage 86.93% 86.92% -0.01% == Files 915 915 Lines 4415844168 +10 == + Hits3838738395 +8 - Misses 5771 5773 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/api/common/experimental/mark\_tasks.py](https://codecov.io/gh/apache/airflow/pull/7683/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9tYXJrX3Rhc2tzLnB5) | `94.51% <90.47%> (-1.01%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/7683/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.30% <100.00%> (+0.03%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=footer). Last update [271ee64...035c685](https://codecov.io/gh/apache/airflow/pull/7683?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] zhongjiajie commented on issue #7729: [AIRFLOW-7066] Use sphinx syntax in concepts.rst
zhongjiajie commented on issue #7729: [AIRFLOW-7066] Use sphinx syntax in concepts.rst URL: https://github.com/apache/airflow/pull/7729#issuecomment-599189560 BTW, I have some question could not understand, in my PR https://github.com/apache/airflow/pull/7709, raise pylint error three time, but `airflow/www/views.py` is in `pylint_todo.txt`, which mean will not test by pylint, also test folder will not test by pylint. should where the pylint error? (I only change two files) I test in breeze using `pylint airflow/www/views.py` and `pylint test/www/views.py` only `airflow/www/views.py` have some error hint(but it in pylint_todo.txt) @exploy This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7593: [AIRFLOW-6959] Use NULL as dag.description default value
codecov-io edited a comment on issue #7593: [AIRFLOW-6959] Use NULL as dag.description default value URL: https://github.com/apache/airflow/pull/7593#issuecomment-592953279 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7593?src=pr=h1) Report > Merging [#7593](https://codecov.io/gh/apache/airflow/pull/7593?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/271ee6477b80633049457557020a4f525f9135cd=desc) will **not change** coverage by `%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7593?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#7593 +/- ## === Coverage 86.93% 86.93% === Files 915 915 Lines 4415844158 === Hits3838738387 Misses 5771 5771 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7593?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7593/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `91.21% <ø> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7593?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7593?src=pr=footer). Last update [271ee64...17b608f](https://codecov.io/gh/apache/airflow/pull/7593?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7593: [AIRFLOW-6959] Use NULL as dag.description default value
codecov-io edited a comment on issue #7593: [AIRFLOW-6959] Use NULL as dag.description default value URL: https://github.com/apache/airflow/pull/7593#issuecomment-592953279 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7593?src=pr=h1) Report > Merging [#7593](https://codecov.io/gh/apache/airflow/pull/7593?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/271ee6477b80633049457557020a4f525f9135cd?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7593?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#7593 +/- ## === Coverage 86.93% 86.93% === Files 915 915 Lines 4415844158 === Hits3838738387 Misses 5771 5771 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7593?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7593/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `91.21% <ø> (ø)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7593?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7593?src=pr=footer). Last update [271ee64...17b608f](https://codecov.io/gh/apache/airflow/pull/7593?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7729: [AIRFLOW-7066] Use sphinx syntax in concepts.rst
potiuk commented on issue #7729: [AIRFLOW-7066] Use sphinx syntax in concepts.rst URL: https://github.com/apache/airflow/pull/7729#issuecomment-599188850 Restarted This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7593: [AIRFLOW-6959] Use NULL as dag.description default value
codecov-io edited a comment on issue #7593: [AIRFLOW-6959] Use NULL as dag.description default value URL: https://github.com/apache/airflow/pull/7593#issuecomment-592953279 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7593?src=pr=h1) Report > Merging [#7593](https://codecov.io/gh/apache/airflow/pull/7593?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/271ee6477b80633049457557020a4f525f9135cd=desc) will **not change** coverage by `%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7593?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#7593 +/- ## === Coverage 86.93% 86.93% === Files 915 915 Lines 4415844158 === Hits3838738387 Misses 5771 5771 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7593?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7593/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `91.21% <ø> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7593?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7593?src=pr=footer). Last update [271ee64...17b608f](https://codecov.io/gh/apache/airflow/pull/7593?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] zhongjiajie commented on issue #7729: [AIRFLOW-7066] Use sphinx syntax in concepts.rst
zhongjiajie commented on issue #7729: [AIRFLOW-7066] Use sphinx syntax in concepts.rst URL: https://github.com/apache/airflow/pull/7729#issuecomment-599188735 Seem CI failed with unrelated error, https://travis-ci.org/github/apache/airflow/jobs/662646733?utm_medium=notification_source=github_status This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-6588) json_format and write_stdout are boolean
[ https://issues.apache.org/jira/browse/AIRFLOW-6588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-6588: -- Affects Version/s: (was: 1.10.10) 1.10.9 > json_format and write_stdout are boolean > > > Key: AIRFLOW-6588 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6588 > Project: Apache Airflow > Issue Type: Bug > Components: logging >Affects Versions: 1.10.9 >Reporter: Ping Zhang >Assignee: Ping Zhang >Priority: Minor > Fix For: 1.10.10 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
