[GitHub] [airflow] curiousjazz77 opened a new issue #8449: Create batch version of the AWSAthenaOperator

[GitBox]

curiousjazz77 opened a new issue #8449: Create batch version of the 
AWSAthenaOperator
URL: https://github.com/apache/airflow/issues/8449
 
 
   **Description**
   
   Create batch version of the AWSAthenaOperator that can accept multiple 
queries and execute them
   
   **Use case / motivation**
   
   Currently, the AWSAthenaOperator is built to handle one query and poll for 
its success. In the event that you have multiple queries to execute via Athena, 
you must move logic into the dag to run the AWSAthenaOperator in a for loop 
over a number of queries. This is not best practice in the event that you have 
a task that generates batch queries to be submitted to Athena.
   
   This issue proposes that an AWSAthenaBatchOperator be created that can 
execute a batch of queries. This would allow Airflow users to contain logic to 
the tasks instead of the dags. 
   
   A first take on creating a new operator like this:
   ```
   class AWSAthenaBatchOperator(BaseOperator):
   """
   An operator that submit a batch of presto queries to athena for the same 
database.
   If ``do_xcom_push`` is True, the QueryExecutionID assigned to the
   query will be pushed to an XCom when it successfuly completes.
   :param query: Presto to be run on athena. (templated)
   :type queries: str demlinited by ";\n"
   :param database: Database to select. (templated)
   :type database: str
   :param output_location: s3 path to write the query results into. 
(templated)
   :type output_location: str
   :param aws_conn_id: aws connection to use
   :type aws_conn_id: str
   :param sleep_time: Time to wait between two consecutive call to check 
query status on athena
   :type sleep_time: int
   :param max_tries: Number of times to poll for query state before 
function exits
   :type max_triex: int
   """
   
   ui_color = '#44b5e2'
   template_fields = ('query', 'database', 'output_location')
   template_ext = ('.sql', )
   
   @apply_defaults
   def __init__(  # pylint: disable=too-many-arguments
   self,
   queries,
   database,
   output_location,
   aws_conn_id="aws_default",
   workgroup="primary",
   query_execution_context=None,
   result_configuration=None,
   sleep_time=30,
   max_tries=None,
   *args,
   **kwargs
   ):
   super().__init__(*args, **kwargs)
   self.queries = queries
   self.database = database
   self.output_location = output_location
   self.aws_conn_id = aws_conn_id
   self.workgroup = workgroup
   self.query_execution_context = query_execution_context or {}
   self.result_configuration = result_configuration or {}
   self.sleep_time = sleep_time
   self.max_tries = max_tries
   self.query_execution_id = None
   self.hook = None
   self.query_execution_ids = []
   
   def get_hook(self):
   """Create and return an AWSAthenaHook."""
   return AWSAthenaHook(self.aws_conn_id, self.sleep_time)
   
   def execute(self, context):
   """
   Run Presto Query on Athena
   """
   self.hook = self.get_hook()
   
   self.query_execution_context['Database'] = self.database
   self.result_configuration['OutputLocation'] = self.output_location
   
   batch = self.queries.split(";\n")
   
   for query in batch:
   self.client_request_token = str(uuid4())  # new each time 
for idempotency
   self.query_execution_id = self.hook.run_query(self.query, 
self.query_execution_context,
   
self.result_configuration, self.client_request_token,
self.workgroup)
   query_status = 
self.hook.poll_query_status(self.query_execution_id, self.max_tries)
   
   if query_status in AWSAthenaHook.FAILURE_STATES:
   error_message = 
self.hook.get_state_change_reason(self.query_execution_id)
   raise Exception(
   'Final state of Athena job is {}, query_execution_id 
is {}. Error: {}'
   .format(query_status, self.query_execution_id, 
error_message))
   elif not query_status or query_status in 
AWSAthenaHook.INTERMEDIATE_STATES:
   raise Exception(
   'Final state of Athena job is {}. '
   'Max tries of poll status exceeded, 
query_execution_id is {}.'
   .format(query_status, self.query_execution_id))
   self.query_execution_ids.append(self.query_execution_id)
   
   return query_execution_ids 
   ``` 


This 

[GitHub] [airflow] curiousjazz77 opened a new issue #8448: Create S3ListPrefixesOperator that returns subfolders from an S3 bucket.

[GitBox]

curiousjazz77 opened a new issue #8448: Create S3ListPrefixesOperator that 
returns subfolders from an S3 bucket.
URL: https://github.com/apache/airflow/issues/8448
 
 
   **Description**
   
   Create operator that lists s3 prefixes in order to return a list of 
subfolders from an S3 bucket. 
   
   **Use case / motivation**
   We would like to use an operator to return subfolders in an s3 bucket. This 
would help us determine partitioning keys for an ETL. Getting these subfolders 
would allow us to parse them and then pass them to a LivyOperator or the 
AWSAthenaOperator to then ETL data into a table partitioned by the returned 
partition key (based on the subfolders found in said s3 bucket). 
   
   Currently, the 
[S3ListOperator](https://github.com/apache/airflow/blob/master/airflow/providers/amazon/aws/operators/s3_list.py)
 uses the S3Hook to list keys in a bucket:
   
   ```
   def execute(self, context):
   hook = S3Hook(aws_conn_id=self.aws_conn_id, verify=self.verify)
   
   self.log.info(
   'Getting the list of files from bucket: %s in prefix: %s 
(Delimiter {%s)',
   self.bucket, self.prefix, self.delimiter
   )
   
   return hook.list_keys(
   bucket_name=self.bucket,
   prefix=self.prefix,
   delimiter=self.delimiter)
   ```
   
   It only returns files and not subfolders because the 
[S3Hook's](https://github.com/apache/airflow/blob/8517cb189872aebdd70e9d5d584478cd7fb65d0f/airflow/providers/amazon/aws/hooks/s3.py#L241)
 list_key function parses each result page that contains contents. 
   
   ```
   for page in response:
   if 'Contents' in page:
   has_results = True
   for k in page['Contents']:
   keys.append(k['Key'])
   ```
   
   If we want to get subfolders in a bucket, we instead want to use the 
[list_prefixes](https://github.com/apache/airflow/blob/8517cb189872aebdd70e9d5d584478cd7fb65d0f/airflow/providers/amazon/aws/hooks/s3.py#L199)
 method of the S3Hook because it will look for subfolders found under 'Common 
Prefixes' for a given page in the response. 
   
   Currently, there is not an operator using this method. This issue proposes 
that an S3ListPrefixesOperator be created that uses the `list_prefixes` method 
of the S3Hook. 
   ```
   for page in response:
   if 'CommonPrefixes' in page:
   has_results = True
   for common_prefix in page['CommonPrefixes']:
   prefixes.append(common_prefix['Prefix'])
   
   if has_results:
   return prefixes
   return None
   ```
   
   We have tested the creation of a new operator like this internally and it 
has worked in our preliminary trials. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] boring-cyborg[bot] commented on issue #8448: Create S3ListPrefixesOperator that returns subfolders from an S3 bucket.

[GitBox]

boring-cyborg[bot] commented on issue #8448: Create S3ListPrefixesOperator that 
returns subfolders from an S3 bucket.
URL: https://github.com/apache/airflow/issues/8448#issuecomment-616003440
 
 
   Thanks for opening your first issue here! Be sure to follow the issue 
template!
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6645) Pass the TaskHandlers configuration using the constructor

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086698#comment-17086698
 ] 

ASF GitHub Bot commented on AIRFLOW-6645:
-

stale[bot] commented on pull request #7267: [AIRFLOW-6645] Pass the 
TaskHandlers configuration using the constructor
URL: https://github.com/apache/airflow/pull/7267
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Pass the TaskHandlers configuration using the constructor
> -
>
> Key: AIRFLOW-6645
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6645
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: logging
>Affects Versions: 1.10.7
>Reporter: Kamil Bregula
>Priority: Major
>
> It is difficult to understand the configurations of these classes if they 
> contain hidden configuration options. All configuration parameters should be 
> passed by the constructor.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] stale[bot] closed pull request #7267: [AIRFLOW-6645] Pass the TaskHandlers configuration using the constructor

[GitBox]

stale[bot] closed pull request #7267: [AIRFLOW-6645] Pass the TaskHandlers 
configuration using the constructor
URL: https://github.com/apache/airflow/pull/7267
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ivorynoise commented on issue #7724: [AIRFLOW-1536] Inherit umask from parent process in daemon mode

[GitBox]

ivorynoise commented on issue #7724: [AIRFLOW-1536] Inherit umask from parent 
process in daemon mode
URL: https://github.com/apache/airflow/pull/7724#issuecomment-615955122
 
 
   @mik-laj @potiuk request you to review the changes and merge with the 
master. Thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ivorynoise opened a new pull request #7724: [AIRFLOW-1536] Inherit umask from parent process in daemon mode

[GitBox]

ivorynoise opened a new pull request #7724: [AIRFLOW-1536] Inherit umask from 
parent process in daemon mode
URL: https://github.com/apache/airflow/pull/7724
 
 
   When celery workers are run in daemon mode, umask was set to default(0)
   which create dirs/files with 0777/0666 permissions.
   
   ---
   Issue link: 
[AIRFLOW-1536](https://issues.apache.org/jira/browse/AIRFLOW-1536)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-1536) DaemonContext uses default umask 0

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086664#comment-17086664
 ] 

ASF GitHub Bot commented on AIRFLOW-1536:
-

ivorynoise commented on pull request #7724: [AIRFLOW-1536] Inherit umask from 
parent process in daemon mode
URL: https://github.com/apache/airflow/pull/7724
 
 
   When celery workers are run in daemon mode, umask was set to default(0)
   which create dirs/files with 0777/0666 permissions.
   
   ---
   Issue link: 
[AIRFLOW-1536](https://issues.apache.org/jira/browse/AIRFLOW-1536)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> DaemonContext uses default umask 0
> --
>
> Key: AIRFLOW-1536
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1536
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: cli, security
>Reporter: Timothy O'Keefe
>Assignee: Deepak Aggarwal
>Priority: Major
>
> All DaemonContext instances used for worker, scheduler, webserver, flower, 
> etc. do not supply a umask argument. See here for example:
> https://github.com/apache/incubator-airflow/blob/b0669b532a7be9aa34a4390951deaa25897c62e6/airflow/bin/cli.py#L869
> As a result, the DaemonContext will use the default umask=0 which leaves user 
> data exposed. A BashOperator for example that writes any files would have 
> permissions rw-rw-rw- as would any airflow logs.
> I believe the umask should either be configurable, or inherited from the 
> parent shell, or both.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] mik-laj opened a new issue #8447: Command descriptions too extensive

[GitBox]

mik-laj opened a new issue #8447: Command descriptions too extensive
URL: https://github.com/apache/airflow/issues/8447
 
 
   Hello,
   
   The description of some of the commands on the list are too long. This makes 
them difficult to read.
   https://user-images.githubusercontent.com/12058428/79671178-bd719000-81c8-11ea-97b5-fc4b4aae0abf.png";>
   
   I think that some text should be moved to descriptions of individual 
commands e.g. `airflow days backfill --help``.
   We currently do not have descriptions for individual commands. After 
executing the `` --help '' command, we only have a list of arguments.
   https://user-images.githubusercontent.com/12058428/79671219-248f4480-81c9-11ea-8b6c-030495e3be86.png";>
   
   After making changes it may look like the screen below.
   https://user-images.githubusercontent.com/12058428/79671262-8059cd80-81c9-11ea-8539-dee59122c511.png";>
   
   https://user-images.githubusercontent.com/12058428/79671338-e5152800-81c9-11ea-9f3a-1f28628d6af5.png";>
   
   Descriptions can be set by the parameters `description` or` epilogue`.
   
https://github.com/apache/airflow/blob/8517cb189872aebdd70e9d5d584478cd7fb65d0f/airflow/cli/cli_parser.py#L1280-L1282
   
   ```diff
   sub_proc = subparsers.add_parser(
   -sub.name, help=sub.help
   +sub.name, help=sub.help, description="DESCRIPTION", epilog="EPILOG"
   )
   ```
   
   I would be happy if all the commands had a description. It may duplicate the 
description from the list.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #8439: Clean up the test_variable_commnand.py file.

[GitBox]

mik-laj commented on issue #8439: Clean up the test_variable_commnand.py file.
URL: https://github.com/apache/airflow/issues/8439#issuecomment-615947215
 
 
   Yes.  I want to divide one large test method into several smaller ones.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] Kopurlso commented on issue #8439: Clean up the test_variable_commnand.py file.

[GitBox]

Kopurlso commented on issue #8439: Clean up the test_variable_commnand.py file.
URL: https://github.com/apache/airflow/issues/8439#issuecomment-615946596
 
 
   Hey @mik-laj I'm a bit of a beginner, but I'd love to try my hand at this 
issue, if I'm understanding the ask correctly, you want to refactor out each 
variable_command method into its own testing method for easier testing.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ephraimbuddy commented on issue #8272: Cloud Life Sciences operator and hook

[GitBox]

ephraimbuddy commented on issue #8272: Cloud Life Sciences operator and hook
URL: https://github.com/apache/airflow/issues/8272#issuecomment-615940531
 
 
   Please assign this to me, I'm now working on it. Thanks


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil merged pull request #8446: Fix formatting in Lineage docs

[GitBox]

kaxil merged pull request #8446: Fix formatting in Lineage docs
URL: https://github.com/apache/airflow/pull/8446
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dimberman merged pull request #8444: Make KubernetesPodOperator clear in docs

[GitBox]

dimberman merged pull request #8444: Make KubernetesPodOperator clear in docs
URL: https://github.com/apache/airflow/pull/8444
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dimberman merged pull request #8445: Improve language in Pod Mutation Hook docs

[GitBox]

dimberman merged pull request #8445: Improve language in Pod Mutation Hook docs
URL: https://github.com/apache/airflow/pull/8445
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on issue #8446: Fix formatting in Lineage docs

[GitBox]

kaxil commented on issue #8446: Fix formatting in Lineage docs
URL: https://github.com/apache/airflow/pull/8446#issuecomment-615936348
 
 
   We need https://github.com/apache/airflow/issues/8091 to eliminate this kind 
of issues 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk merged pull request #8437: Not pulling base python images does not work for bugfixes

[GitBox]

potiuk merged pull request #8437: Not pulling base python images does not work 
for bugfixes
URL: https://github.com/apache/airflow/pull/8437
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil opened a new pull request #8446: Fix formatting in Lineage docs

[GitBox]

kaxil opened a new pull request #8446: Fix formatting in Lineage docs
URL: https://github.com/apache/airflow/pull/8446
 
 
   **Before**:
   
![image](https://user-images.githubusercontent.com/8811558/79669917-12a4a600-81b7-11ea-9eb1-d93ba91ab9fb.png)
   
   
   **After**:
   
![image](https://user-images.githubusercontent.com/8811558/79669969-7202b600-81b7-11ea-8364-a8db3d020bad.png)
   
   
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] wkhudgins92 commented on issue #8442: [AIRFLOW-6281] Create guide for GCS to GCS transfer operators

[GitBox]

wkhudgins92 commented on issue #8442: [AIRFLOW-6281] Create guide for GCS to 
GCS transfer operators
URL: https://github.com/apache/airflow/pull/8442#issuecomment-615936090
 
 
   Not sure why the build is failing–not sure if it is related to my 
documentation changes or not.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil opened a new pull request #8445: Improve language in Pod Mutation Hook docs

[GitBox]

kaxil opened a new pull request #8445: Improve language in Pod Mutation Hook 
docs
URL: https://github.com/apache/airflow/pull/8445
 
 
   `Your local Airflow settings file` doesn't seem correct. 
   
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on issue #8443: Fix formatting of Pool docs in concepts.rst

[GitBox]

kaxil commented on issue #8443: Fix formatting of Pool docs in concepts.rst
URL: https://github.com/apache/airflow/pull/8443#issuecomment-615934700
 
 
   Docs and static tests have passed


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil merged pull request #8443: Fix formatting of Pool docs in concepts.rst

[GitBox]

kaxil merged pull request #8443: Fix formatting of Pool docs in concepts.rst
URL: https://github.com/apache/airflow/pull/8443
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil opened a new pull request #8444: Make KubernetesPodOperator clear in docs

[GitBox]

kaxil opened a new pull request #8444: Make KubernetesPodOperator clear in docs
URL: https://github.com/apache/airflow/pull/8444
 
 
   Avoid confusion with 
https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
   and be specific about the name of the Operator
   
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #8436: Add Local Filesystem Secret Backend

[GitBox]

mik-laj commented on a change in pull request #8436: Add Local Filesystem 
Secret Backend
URL: https://github.com/apache/airflow/pull/8436#discussion_r410741048
 
 

 ##
 File path: airflow/secrets/local_filesystem.py
 ##
 @@ -0,0 +1,257 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Objects relating to retreiving connections and variables from local file
+"""
+import json
+import logging
+import os
+from collections import defaultdict
+from json import JSONDecodeError
+from typing import Any, Dict, List, Optional, Tuple
+
+from airflow.exceptions import AirflowException, AirflowFileParseException, 
FileSyntaxError
+from airflow.secrets.base_secrets import BaseSecretsBackend
+from airflow.utils.file import COMMENT_PATTERN
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+# constants to limit cyclical imports
+CONNECTION_PARAMETERS_NAMES = {
+"conn_id",
+"conn_type",
+"host",
+"login",
+"password",
+"schema",
+"port",
+"extra",
+"uri",
+}
+
+log = logging.getLogger(__name__)
+
+
+def _parser_env_file(content: str) -> Tuple[Dict[str, List[str]], 
List[FileSyntaxError]]:
+"""
+Parse a file in the ``.env '' format.
+
+   .. code-block:: text
+
MY_CONN_ID=my-conn-type://my-login:my-pa%2Fssword@my-host:5432/my-schema?param1=val1¶m2=val2
+
+:param content:
+:return: Tuple with mapping of key and list of values and list of syntax 
errors
+"""
+secrets: Dict[str, List[str]] = defaultdict(list)
+errors: List[FileSyntaxError] = []
+for line_no, line in enumerate(content.splitlines(), 1):
+if not line:
+# Ignore empty line
+continue
+
+if COMMENT_PATTERN.match(line):
+# Ignore comments:
+continue
+
+var_parts: List[str] = line.split("=", 2)
+if len(var_parts) != 2:
+errors.append(
+FileSyntaxError(
+line_no=line_no,
+message='Invalid line format. The line should contain at 
least one equal sign ("=").',
+)
+)
+continue
+
+key, value = var_parts
+if not key:
+errors.append(FileSyntaxError(line_no=line_no, message="Invalid 
line format. Key is empty.",))
+secrets[key].append(value)
+return secrets, errors
+
+
+def _parse_json_file(content: str) -> Tuple[Dict[str, Any], 
List[FileSyntaxError]]:
+"""
+Parrse a file in the JSON format.
+
+:param content:
+:return: Tuple with mapping of key and list of values and list of syntax 
errors
+"""
+if not content:
+return {}, [FileSyntaxError(line_no=1, message="The file is empty.")]
+try:
+secrets = json.loads(content)
+except JSONDecodeError as e:
+return {}, [FileSyntaxError(line_no=int(e.lineno), message=e.msg)]
+if not isinstance(secrets, dict):
+return {}, [FileSyntaxError(line_no=1, message="The file should 
contain the object.")]
+
+return secrets, []
+
+
+def _parse_secret_file(file_content: str, file_path: str) -> Tuple[Dict[str, 
Any], List[FileSyntaxError]]:
+"""
+Based on the file extension format, selects a parser and parse the file.
+
+:param file_content:
+:param file_path:
+:return:
+"""
+if file_path.lower().endswith(".env"):
+secrets, parse_errors = _parser_env_file(file_content)
+elif file_path.lower().endswith(".json"):
+secrets, parse_errors = _parse_json_file(file_content)
+else:
+raise AirflowException("Unsupported file format. The file must have 
the extension .env or .json")
+return secrets, parse_errors
+
+
+def _read_secret_file(file_path: str) -> Dict[str, str]:
+if not os.path.exists(file_path):
+raise AirflowException(
+f"File {file_path} was not found. Check the configuration of your 
secret backend."
+)
+
+log.debug("Reading file: %s", file_path)
+with open(file_path) as f:
+file_content = f.read()
 
 Review comment:
   environs use the .dotenv library to parse the file.  Unfortunately, this 
library does not support duplicate keys. In our case, it is necessary to handle 
multiple connections

[GitHub] [airflow] turbaszek commented on a change in pull request #8440: Add airflow connection lookup command

[GitBox]

turbaszek commented on a change in pull request #8440: Add airflow connection 
lookup command
URL: https://github.com/apache/airflow/pull/8440#discussion_r410740787
 
 

 ##
 File path: airflow/cli/commands/connection_command.py
 ##
 @@ -15,29 +15,41 @@
 # specific language governing permissions and limitations
 # under the License.
 """Connection sub-commands"""
-import reprlib
+from typing import List
 from urllib.parse import urlunparse
 
 from sqlalchemy.orm import exc
 from tabulate import tabulate
 
+from airflow.hooks.base_hook import BaseHook
 from airflow.models import Connection
 from airflow.utils import cli as cli_utils
 from airflow.utils.session import create_session
 
 
+def _tabule_connection(conns: List[Connection], tablefmt: str):
+tabulat_data = [
 
 Review comment:
   ```suggestion
   tabulate_data = [
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] turbaszek commented on a change in pull request #8440: Add airflow connection lookup command

[GitBox]

turbaszek commented on a change in pull request #8440: Add airflow connection 
lookup command
URL: https://github.com/apache/airflow/pull/8440#discussion_r410740985
 
 

 ##
 File path: airflow/cli/commands/connection_command.py
 ##
 @@ -15,29 +15,41 @@
 # specific language governing permissions and limitations
 # under the License.
 """Connection sub-commands"""
-import reprlib
+from typing import List
 from urllib.parse import urlunparse
 
 from sqlalchemy.orm import exc
 from tabulate import tabulate
 
+from airflow.hooks.base_hook import BaseHook
 from airflow.models import Connection
 from airflow.utils import cli as cli_utils
 from airflow.utils.session import create_session
 
 
+def _tabule_connection(conns: List[Connection], tablefmt: str):
+tabulat_data = [
+{
+'Conn Id': conn.conn_id,
+'Conn Type': conn.conn_type,
+'Host': conn.host,
+'Port': conn.port,
+'Is Encrypted': conn.is_encrypted,
+'Is Extra Encrypted': conn.is_encrypted,
+'Extra': conn.extra,
+} for conn in conns
+]
+
+msg = tabulate(tabulat_data, tablefmt=tablefmt, headers='keys')
+return msg
+
+
 def connections_list(args):
 """Lists all connections at the command line"""
 with create_session() as session:
-conns = session.query(Connection.conn_id, Connection.conn_type,
-  Connection.host, Connection.port,
-  Connection.is_encrypted,
-  Connection.is_extra_encrypted,
-  Connection.extra).all()
-conns = [map(reprlib.repr, conn) for conn in conns]
-msg = tabulate(conns, ['Conn Id', 'Conn Type', 'Host', 'Port',
-   'Is Encrypted', 'Is Extra Encrypted', 'Extra'],
-   tablefmt=args.output)
+conns = session.query(Connection).all()
 
 Review comment:
   I quite agree with @kaxil, adding `--filter=conn` sounds like a possible 
solution? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] turbaszek commented on a change in pull request #8440: Add airflow connection lookup command

[GitBox]

turbaszek commented on a change in pull request #8440: Add airflow connection 
lookup command
URL: https://github.com/apache/airflow/pull/8440#discussion_r410740772
 
 

 ##
 File path: airflow/cli/commands/connection_command.py
 ##
 @@ -15,29 +15,41 @@
 # specific language governing permissions and limitations
 # under the License.
 """Connection sub-commands"""
-import reprlib
+from typing import List
 from urllib.parse import urlunparse
 
 from sqlalchemy.orm import exc
 from tabulate import tabulate
 
+from airflow.hooks.base_hook import BaseHook
 from airflow.models import Connection
 from airflow.utils import cli as cli_utils
 from airflow.utils.session import create_session
 
 
+def _tabule_connection(conns: List[Connection], tablefmt: str):
 
 Review comment:
   ```suggestion
   def _tabulate_connection(conns: List[Connection], tablefmt: str):
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil opened a new pull request #8443: Fix formatting of Pool docs in concepts.rst

[GitBox]

kaxil opened a new pull request #8443: Fix formatting of Pool docs in 
concepts.rst
URL: https://github.com/apache/airflow/pull/8443
 
 
   **Previous**:
   
![image](https://user-images.githubusercontent.com/8811558/79669386-09b1d580-81b3-11ea-997e-a35805c39231.png)
   
   
   **Now**:
   
![image](https://user-images.githubusercontent.com/8811558/79669411-2d751b80-81b3-11ea-9cad-f2d87c8bba51.png)
   
   
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] turbaszek merged pull request #8273: BigQueryCheckOperator location fix

[GitBox]

turbaszek merged pull request #8273: BigQueryCheckOperator location fix
URL: https://github.com/apache/airflow/pull/8273
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] boring-cyborg[bot] commented on issue #8273: BigQueryCheckOperator location fix

[GitBox]

boring-cyborg[bot] commented on issue #8273: BigQueryCheckOperator location fix
URL: https://github.com/apache/airflow/pull/8273#issuecomment-615932102
 
 
   Awesome work, congrats on your first merged pull request!
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #8436: Add Local Filesystem Secret Backend

[GitBox]

mik-laj commented on a change in pull request #8436: Add Local Filesystem 
Secret Backend
URL: https://github.com/apache/airflow/pull/8436#discussion_r410740112
 
 

 ##
 File path: airflow/secrets/local_filesystem.py
 ##
 @@ -0,0 +1,257 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Objects relating to retreiving connections and variables from local file
+"""
+import json
+import logging
+import os
+from collections import defaultdict
+from json import JSONDecodeError
+from typing import Any, Dict, List, Optional, Tuple
+
+from airflow.exceptions import AirflowException, AirflowFileParseException, 
FileSyntaxError
+from airflow.secrets.base_secrets import BaseSecretsBackend
+from airflow.utils.file import COMMENT_PATTERN
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+# constants to limit cyclical imports
+CONNECTION_PARAMETERS_NAMES = {
+"conn_id",
+"conn_type",
+"host",
+"login",
+"password",
+"schema",
+"port",
+"extra",
+"uri",
+}
+
+log = logging.getLogger(__name__)
+
+
+def _parser_env_file(content: str) -> Tuple[Dict[str, List[str]], 
List[FileSyntaxError]]:
 
 Review comment:
   https://user-images.githubusercontent.com/12058428/79669319-ff93d500-81ba-11ea-8c71-95c15ad50f0f.png";>
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] wkhudgins92 opened a new pull request #8442: [AIRFLOW-6281] Create guide for GCS to GCS transfer operators

[GitBox]

wkhudgins92 opened a new pull request #8442: [AIRFLOW-6281] Create guide for 
GCS to GCS transfer operators
URL: https://github.com/apache/airflow/pull/8442
 
 
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6281) Create guide for GCS to GCS transfer operator

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086609#comment-17086609
 ] 

ASF GitHub Bot commented on AIRFLOW-6281:
-

wkhudgins92 commented on pull request #8442: [AIRFLOW-6281] Create guide for 
GCS to GCS transfer operators
URL: https://github.com/apache/airflow/pull/8442
 
 
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Create guide for GCS to GCS transfer operator 
> --
>
> Key: AIRFLOW-6281
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6281
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: documentation, gcp
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Minor
>  Labels: GoodForNewContributors
>
> Hello,
> A guide that describes how to use GCS to GCS transfer operators would be 
> useful.
> Other guides are available:
> https://airflow.readthedocs.io/en/latest/howto/operator/gcp/index.html
> If anyone is interested in this task, I am willing to provide all the 
> necessary tips and information.
> Best regards,
> Kamil



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] mik-laj commented on a change in pull request #8436: Add Local Filesystem Secret Backend

[GitBox]

mik-laj commented on a change in pull request #8436: Add Local Filesystem 
Secret Backend
URL: https://github.com/apache/airflow/pull/8436#discussion_r410739408
 
 

 ##
 File path: airflow/secrets/local_filesystem.py
 ##
 @@ -0,0 +1,257 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Objects relating to retreiving connections and variables from local file
+"""
+import json
+import logging
+import os
+from collections import defaultdict
+from json import JSONDecodeError
+from typing import Any, Dict, List, Optional, Tuple
+
+from airflow.exceptions import AirflowException, AirflowFileParseException, 
FileSyntaxError
+from airflow.secrets.base_secrets import BaseSecretsBackend
+from airflow.utils.file import COMMENT_PATTERN
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+# constants to limit cyclical imports
+CONNECTION_PARAMETERS_NAMES = {
+"conn_id",
+"conn_type",
+"host",
+"login",
+"password",
+"schema",
+"port",
+"extra",
+"uri",
+}
+
+log = logging.getLogger(__name__)
+
+
+def _parser_env_file(content: str) -> Tuple[Dict[str, List[str]], 
List[FileSyntaxError]]:
 
 Review comment:
   I would like to add support for env files and better error handling 
   
   Please look at this code:
   
https://github.com/apache/airflow/blob/master/airflow/cli/commands/variable_command.py#L76-L79
   A pure exception is caught here. The error message is very inaccurate. This 
is not user friendly.
   
   After that, the code in Web UI and CLI is duplicated. Now it will be the 
third-place where the logic responsible for loading variables from the file is.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-7111) Expose generate_presigned_url of boto3 to S3Hook

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086606#comment-17086606
 ] 

ASF GitHub Bot commented on AIRFLOW-7111:
-

jcarless commented on pull request #8441: [AIRFLOW-7111] Add 
generate_presigned_url method to S3Hook
URL: https://github.com/apache/airflow/pull/8441
 
 
   Boto3 generate_presigned_url exposed to S3Hook.
   ---
   
   - [ x ] Description above provides context of the change
   - [ x ] Unit tests coverage for changes (not needed for documentation 
changes)
   - [ x ] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [ x ] Relevant documentation is updated including usage instructions.
   - [ x ] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Expose generate_presigned_url of boto3 to S3Hook
> 
>
> Key: AIRFLOW-7111
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7111
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: aws
>Affects Versions: 1.10.9
>Reporter: korni
>Assignee: Jerome Carless
>Priority: Major
>  Labels: S3, S3Hook, aws, easyfix, gsoc, gsoc2020
>
> boto3 has {{generate_presigned_url which should be exposed in the Hook:}}
> {{[https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.generate_presigned_url]}}
> {{generate_presigned_url}}(_ClientMethod_, _Params=None_, _ExpiresIn=3600_, 
> _HttpMethod=None_)
> Generate a presigned url given a client, its method, and arguments
> Parameters
>  * *ClientMethod* (_string_) -- The client method to presign for
>  * *Params* (_dict_) -- The parameters normally passed to {{ClientMethod}}.
>  * *ExpiresIn* (_int_) -- The number of seconds the presigned url is valid 
> for. By default it expires in an hour (3600 seconds)
>  * *HttpMethod* (_string_) -- The http method to use on the generated url. By 
> default, the http method is whatever is used in the method's model.
> Returns The presigned url



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] jcarless opened a new pull request #8441: [AIRFLOW-7111] Add generate_presigned_url method to S3Hook

[GitBox]

jcarless opened a new pull request #8441: [AIRFLOW-7111] Add 
generate_presigned_url method to S3Hook
URL: https://github.com/apache/airflow/pull/8441
 
 
   Boto3 generate_presigned_url exposed to S3Hook.
   ---
   
   - [ x ] Description above provides context of the change
   - [ x ] Unit tests coverage for changes (not needed for documentation 
changes)
   - [ x ] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [ x ] Relevant documentation is updated including usage instructions.
   - [ x ] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #8436: Add Local Filesystem Secret Backend

[GitBox]

mik-laj commented on a change in pull request #8436: Add Local Filesystem 
Secret Backend
URL: https://github.com/apache/airflow/pull/8436#discussion_r410738042
 
 

 ##
 File path: airflow/secrets/local_filesystem.py
 ##
 @@ -0,0 +1,257 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Objects relating to retreiving connections and variables from local file
+"""
+import json
+import logging
+import os
+from collections import defaultdict
+from json import JSONDecodeError
+from typing import Any, Dict, List, Optional, Tuple
+
+from airflow.exceptions import AirflowException, AirflowFileParseException, 
FileSyntaxError
+from airflow.secrets.base_secrets import BaseSecretsBackend
+from airflow.utils.file import COMMENT_PATTERN
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+# constants to limit cyclical imports
+CONNECTION_PARAMETERS_NAMES = {
+"conn_id",
+"conn_type",
+"host",
+"login",
+"password",
+"schema",
+"port",
+"extra",
+"uri",
+}
+
+log = logging.getLogger(__name__)
+
+
+def _parser_env_file(content: str) -> Tuple[Dict[str, List[str]], 
List[FileSyntaxError]]:
+"""
+Parse a file in the ``.env '' format.
+
+   .. code-block:: text
+
MY_CONN_ID=my-conn-type://my-login:my-pa%2Fssword@my-host:5432/my-schema?param1=val1¶m2=val2
+
+:param content:
+:return: Tuple with mapping of key and list of values and list of syntax 
errors
+"""
+secrets: Dict[str, List[str]] = defaultdict(list)
+errors: List[FileSyntaxError] = []
+for line_no, line in enumerate(content.splitlines(), 1):
+if not line:
+# Ignore empty line
+continue
+
+if COMMENT_PATTERN.match(line):
+# Ignore comments:
+continue
+
+var_parts: List[str] = line.split("=", 2)
+if len(var_parts) != 2:
+errors.append(
+FileSyntaxError(
+line_no=line_no,
+message='Invalid line format. The line should contain at 
least one equal sign ("=").',
+)
+)
+continue
+
+key, value = var_parts
+if not key:
+errors.append(FileSyntaxError(line_no=line_no, message="Invalid 
line format. Key is empty.",))
+secrets[key].append(value)
+return secrets, errors
+
+
+def _parse_json_file(content: str) -> Tuple[Dict[str, Any], 
List[FileSyntaxError]]:
+"""
+Parrse a file in the JSON format.
+
+:param content:
+:return: Tuple with mapping of key and list of values and list of syntax 
errors
+"""
+if not content:
+return {}, [FileSyntaxError(line_no=1, message="The file is empty.")]
+try:
+secrets = json.loads(content)
+except JSONDecodeError as e:
+return {}, [FileSyntaxError(line_no=int(e.lineno), message=e.msg)]
+if not isinstance(secrets, dict):
+return {}, [FileSyntaxError(line_no=1, message="The file should 
contain the object.")]
+
+return secrets, []
+
+
+def _parse_secret_file(file_content: str, file_path: str) -> Tuple[Dict[str, 
Any], List[FileSyntaxError]]:
+"""
+Based on the file extension format, selects a parser and parse the file.
+
+:param file_content:
+:param file_path:
+:return:
+"""
+if file_path.lower().endswith(".env"):
+secrets, parse_errors = _parser_env_file(file_content)
+elif file_path.lower().endswith(".json"):
+secrets, parse_errors = _parse_json_file(file_content)
+else:
+raise AirflowException("Unsupported file format. The file must have 
the extension .env or .json")
+return secrets, parse_errors
+
+
+def _read_secret_file(file_path: str) -> Dict[str, str]:
+if not os.path.exists(file_path):
+raise AirflowException(
+f"File {file_path} was not found. Check the configuration of your 
secret backend."
+)
+
+log.debug("Reading file: %s", file_path)
+with open(file_path) as f:
+file_content = f.read()
 
 Review comment:
   I need to check if this library allows duplicate keys. 


This is an automated message from the Apache Git Serv

[GitHub] [airflow] kaxil commented on a change in pull request #8440: Add airflow connection lookup command

[GitBox]

kaxil commented on a change in pull request #8440: Add airflow connection 
lookup command
URL: https://github.com/apache/airflow/pull/8440#discussion_r410737868
 
 

 ##
 File path: airflow/cli/commands/connection_command.py
 ##
 @@ -15,29 +15,41 @@
 # specific language governing permissions and limitations
 # under the License.
 """Connection sub-commands"""
-import reprlib
+from typing import List
 from urllib.parse import urlunparse
 
 from sqlalchemy.orm import exc
 from tabulate import tabulate
 
+from airflow.hooks.base_hook import BaseHook
 from airflow.models import Connection
 from airflow.utils import cli as cli_utils
 from airflow.utils.session import create_session
 
 
+def _tabule_connection(conns: List[Connection], tablefmt: str):
+tabulat_data = [
+{
+'Conn Id': conn.conn_id,
+'Conn Type': conn.conn_type,
+'Host': conn.host,
+'Port': conn.port,
+'Is Encrypted': conn.is_encrypted,
+'Is Extra Encrypted': conn.is_encrypted,
+'Extra': conn.extra,
+} for conn in conns
+]
+
+msg = tabulate(tabulat_data, tablefmt=tablefmt, headers='keys')
+return msg
+
+
 def connections_list(args):
 """Lists all connections at the command line"""
 with create_session() as session:
-conns = session.query(Connection.conn_id, Connection.conn_type,
-  Connection.host, Connection.port,
-  Connection.is_encrypted,
-  Connection.is_extra_encrypted,
-  Connection.extra).all()
-conns = [map(reprlib.repr, conn) for conn in conns]
-msg = tabulate(conns, ['Conn Id', 'Conn Type', 'Host', 'Port',
-   'Is Encrypted', 'Is Extra Encrypted', 'Extra'],
-   tablefmt=args.output)
+conns = session.query(Connection).all()
 
 Review comment:
   Can we just update the **List** method instead of a new **Lookup** command ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #8436: Add Local Filesystem Secret Backend

[GitBox]

mik-laj commented on a change in pull request #8436: Add Local Filesystem 
Secret Backend
URL: https://github.com/apache/airflow/pull/8436#discussion_r410737827
 
 

 ##
 File path: airflow/secrets/local_filesystem.py
 ##
 @@ -0,0 +1,257 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Objects relating to retreiving connections and variables from local file
+"""
+import json
+import logging
+import os
+from collections import defaultdict
+from json import JSONDecodeError
+from typing import Any, Dict, List, Optional, Tuple
+
+from airflow.exceptions import AirflowException, AirflowFileParseException, 
FileSyntaxError
+from airflow.secrets.base_secrets import BaseSecretsBackend
+from airflow.utils.file import COMMENT_PATTERN
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+# constants to limit cyclical imports
+CONNECTION_PARAMETERS_NAMES = {
+"conn_id",
+"conn_type",
+"host",
+"login",
+"password",
+"schema",
+"port",
+"extra",
+"uri",
+}
+
+log = logging.getLogger(__name__)
+
+
+def _parser_env_file(content: str) -> Tuple[Dict[str, List[str]], 
List[FileSyntaxError]]:
+"""
+Parse a file in the ``.env '' format.
+
+   .. code-block:: text
+
MY_CONN_ID=my-conn-type://my-login:my-pa%2Fssword@my-host:5432/my-schema?param1=val1¶m2=val2
+
+:param content:
+:return: Tuple with mapping of key and list of values and list of syntax 
errors
+"""
+secrets: Dict[str, List[str]] = defaultdict(list)
+errors: List[FileSyntaxError] = []
+for line_no, line in enumerate(content.splitlines(), 1):
+if not line:
+# Ignore empty line
+continue
+
+if COMMENT_PATTERN.match(line):
+# Ignore comments:
+continue
+
+var_parts: List[str] = line.split("=", 2)
+if len(var_parts) != 2:
+errors.append(
+FileSyntaxError(
+line_no=line_no,
+message='Invalid line format. The line should contain at 
least one equal sign ("=").',
+)
+)
+continue
+
+key, value = var_parts
+if not key:
+errors.append(FileSyntaxError(line_no=line_no, message="Invalid 
line format. Key is empty.",))
+secrets[key].append(value)
+return secrets, errors
+
+
+def _parse_json_file(content: str) -> Tuple[Dict[str, Any], 
List[FileSyntaxError]]:
+"""
+Parrse a file in the JSON format.
+
+:param content:
+:return: Tuple with mapping of key and list of values and list of syntax 
errors
+"""
+if not content:
+return {}, [FileSyntaxError(line_no=1, message="The file is empty.")]
+try:
+secrets = json.loads(content)
+except JSONDecodeError as e:
+return {}, [FileSyntaxError(line_no=int(e.lineno), message=e.msg)]
+if not isinstance(secrets, dict):
+return {}, [FileSyntaxError(line_no=1, message="The file should 
contain the object.")]
+
+return secrets, []
+
+
+def _parse_secret_file(file_content: str, file_path: str) -> Tuple[Dict[str, 
Any], List[FileSyntaxError]]:
+"""
+Based on the file extension format, selects a parser and parse the file.
+
+:param file_content:
+:param file_path:
+:return:
+"""
+if file_path.lower().endswith(".env"):
+secrets, parse_errors = _parser_env_file(file_content)
+elif file_path.lower().endswith(".json"):
+secrets, parse_errors = _parse_json_file(file_content)
+else:
+raise AirflowException("Unsupported file format. The file must have 
the extension .env or .json")
+return secrets, parse_errors
+
+
+def _read_secret_file(file_path: str) -> Dict[str, str]:
+if not os.path.exists(file_path):
+raise AirflowException(
+f"File {file_path} was not found. Check the configuration of your 
secret backend."
+)
+
+log.debug("Reading file: %s", file_path)
+with open(file_path) as f:
+file_content = f.read()
+
+log.debug("Parsing file: %s", file_path)
+secrets, parse_errors = _parse_secret_file(file_content, file_path)
+log.debug("Parsed file file: len(parse_errors)=%d, len(secrets)=%d", 
l

[GitHub] [airflow] kaxil commented on issue #8438: Return non-zero error code when variable is missing

[GitBox]

kaxil commented on issue #8438: Return non-zero error code when variable is 
missing
URL: https://github.com/apache/airflow/pull/8438#issuecomment-615927347
 
 
   Can you add a description to the PR on why this is needed?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on issue #8436: Add Local Filesystem Secret Backend

[GitBox]

kaxil commented on issue #8436: Add Local Filesystem Secret Backend
URL: https://github.com/apache/airflow/pull/8436#issuecomment-615927107
 
 
   Whenever this PR is merged, please create a PR to backport this to 
v1-10-test branch too, would be useful to get this in 1.10.11


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil merged pull request #8435: Fix performance degradation when updating dagrun state

[GitBox]

kaxil merged pull request #8435: Fix performance degradation when updating 
dagrun state
URL: https://github.com/apache/airflow/pull/8435
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #8436: Add Local Filesystem Secret Backend

[GitBox]

kaxil commented on a change in pull request #8436: Add Local Filesystem Secret 
Backend
URL: https://github.com/apache/airflow/pull/8436#discussion_r410736676
 
 

 ##
 File path: airflow/secrets/local_filesystem.py
 ##
 @@ -0,0 +1,257 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Objects relating to retreiving connections and variables from local file
+"""
+import json
+import logging
+import os
+from collections import defaultdict
+from json import JSONDecodeError
+from typing import Any, Dict, List, Optional, Tuple
+
+from airflow.exceptions import AirflowException, AirflowFileParseException, 
FileSyntaxError
+from airflow.secrets.base_secrets import BaseSecretsBackend
+from airflow.utils.file import COMMENT_PATTERN
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+# constants to limit cyclical imports
+CONNECTION_PARAMETERS_NAMES = {
+"conn_id",
+"conn_type",
+"host",
+"login",
+"password",
+"schema",
+"port",
+"extra",
+"uri",
+}
+
+log = logging.getLogger(__name__)
+
+
+def _parser_env_file(content: str) -> Tuple[Dict[str, List[str]], 
List[FileSyntaxError]]:
 
 Review comment:
   The Variables uploaded from the UI are saved to DB. Where will this code be 
reused?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj opened a new pull request #8440: Add airflow connection lookup command

[GitBox]

mik-laj opened a new pull request #8440: Add airflow connection lookup command
URL: https://github.com/apache/airflow/pull/8440
 
 
   This command is especially useful when interacting with a secret backend.
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [X] Description above provides context of the change
   - [X] Unit tests coverage for changes (not needed for documentation changes)
   - [X] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [X] Relevant documentation is updated including usage instructions.
   - [X] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #8436: Add Local Filesystem Secret Backend

[GitBox]

kaxil commented on a change in pull request #8436: Add Local Filesystem Secret 
Backend
URL: https://github.com/apache/airflow/pull/8436#discussion_r410736096
 
 

 ##
 File path: airflow/secrets/local_filesystem.py
 ##
 @@ -0,0 +1,257 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Objects relating to retreiving connections and variables from local file
+"""
+import json
+import logging
+import os
+from collections import defaultdict
+from json import JSONDecodeError
+from typing import Any, Dict, List, Optional, Tuple
+
+from airflow.exceptions import AirflowException, AirflowFileParseException, 
FileSyntaxError
+from airflow.secrets.base_secrets import BaseSecretsBackend
+from airflow.utils.file import COMMENT_PATTERN
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+# constants to limit cyclical imports
+CONNECTION_PARAMETERS_NAMES = {
+"conn_id",
+"conn_type",
+"host",
+"login",
+"password",
+"schema",
+"port",
+"extra",
+"uri",
+}
+
+log = logging.getLogger(__name__)
+
+
+def _parser_env_file(content: str) -> Tuple[Dict[str, List[str]], 
List[FileSyntaxError]]:
+"""
+Parse a file in the ``.env '' format.
+
+   .. code-block:: text
+
MY_CONN_ID=my-conn-type://my-login:my-pa%2Fssword@my-host:5432/my-schema?param1=val1¶m2=val2
+
+:param content:
+:return: Tuple with mapping of key and list of values and list of syntax 
errors
+"""
+secrets: Dict[str, List[str]] = defaultdict(list)
+errors: List[FileSyntaxError] = []
+for line_no, line in enumerate(content.splitlines(), 1):
+if not line:
+# Ignore empty line
+continue
+
+if COMMENT_PATTERN.match(line):
+# Ignore comments:
+continue
+
+var_parts: List[str] = line.split("=", 2)
+if len(var_parts) != 2:
+errors.append(
+FileSyntaxError(
+line_no=line_no,
+message='Invalid line format. The line should contain at 
least one equal sign ("=").',
+)
+)
+continue
+
+key, value = var_parts
+if not key:
+errors.append(FileSyntaxError(line_no=line_no, message="Invalid 
line format. Key is empty.",))
+secrets[key].append(value)
+return secrets, errors
+
+
+def _parse_json_file(content: str) -> Tuple[Dict[str, Any], 
List[FileSyntaxError]]:
+"""
+Parrse a file in the JSON format.
+
+:param content:
+:return: Tuple with mapping of key and list of values and list of syntax 
errors
+"""
+if not content:
+return {}, [FileSyntaxError(line_no=1, message="The file is empty.")]
+try:
+secrets = json.loads(content)
+except JSONDecodeError as e:
+return {}, [FileSyntaxError(line_no=int(e.lineno), message=e.msg)]
+if not isinstance(secrets, dict):
+return {}, [FileSyntaxError(line_no=1, message="The file should 
contain the object.")]
+
+return secrets, []
+
+
+def _parse_secret_file(file_content: str, file_path: str) -> Tuple[Dict[str, 
Any], List[FileSyntaxError]]:
+"""
+Based on the file extension format, selects a parser and parse the file.
+
+:param file_content:
+:param file_path:
+:return:
+"""
+if file_path.lower().endswith(".env"):
+secrets, parse_errors = _parser_env_file(file_content)
+elif file_path.lower().endswith(".json"):
+secrets, parse_errors = _parse_json_file(file_content)
+else:
+raise AirflowException("Unsupported file format. The file must have 
the extension .env or .json")
+return secrets, parse_errors
+
+
+def _read_secret_file(file_path: str) -> Dict[str, str]:
+if not os.path.exists(file_path):
+raise AirflowException(
+f"File {file_path} was not found. Check the configuration of your 
secret backend."
+)
+
+log.debug("Reading file: %s", file_path)
+with open(file_path) as f:
+file_content = f.read()
+
+log.debug("Parsing file: %s", file_path)
+secrets, parse_errors = _parse_secret_file(file_content, file_path)
+log.debug("Parsed file file: len(parse_errors)=%d, len(secrets)=%d", 
len

[GitHub] [airflow] kaxil commented on a change in pull request #8436: Add Local Filesystem Secret Backend

[GitBox]

kaxil commented on a change in pull request #8436: Add Local Filesystem Secret 
Backend
URL: https://github.com/apache/airflow/pull/8436#discussion_r410736070
 
 

 ##
 File path: airflow/secrets/local_filesystem.py
 ##
 @@ -0,0 +1,257 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Objects relating to retreiving connections and variables from local file
+"""
+import json
+import logging
+import os
+from collections import defaultdict
+from json import JSONDecodeError
+from typing import Any, Dict, List, Optional, Tuple
+
+from airflow.exceptions import AirflowException, AirflowFileParseException, 
FileSyntaxError
+from airflow.secrets.base_secrets import BaseSecretsBackend
+from airflow.utils.file import COMMENT_PATTERN
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+# constants to limit cyclical imports
+CONNECTION_PARAMETERS_NAMES = {
+"conn_id",
+"conn_type",
+"host",
+"login",
+"password",
+"schema",
+"port",
+"extra",
+"uri",
+}
+
+log = logging.getLogger(__name__)
+
+
+def _parser_env_file(content: str) -> Tuple[Dict[str, List[str]], 
List[FileSyntaxError]]:
+"""
+Parse a file in the ``.env '' format.
+
+   .. code-block:: text
+
MY_CONN_ID=my-conn-type://my-login:my-pa%2Fssword@my-host:5432/my-schema?param1=val1¶m2=val2
+
+:param content:
+:return: Tuple with mapping of key and list of values and list of syntax 
errors
+"""
+secrets: Dict[str, List[str]] = defaultdict(list)
+errors: List[FileSyntaxError] = []
+for line_no, line in enumerate(content.splitlines(), 1):
+if not line:
+# Ignore empty line
+continue
+
+if COMMENT_PATTERN.match(line):
+# Ignore comments:
+continue
+
+var_parts: List[str] = line.split("=", 2)
+if len(var_parts) != 2:
+errors.append(
+FileSyntaxError(
+line_no=line_no,
+message='Invalid line format. The line should contain at 
least one equal sign ("=").',
+)
+)
+continue
+
+key, value = var_parts
+if not key:
+errors.append(FileSyntaxError(line_no=line_no, message="Invalid 
line format. Key is empty.",))
+secrets[key].append(value)
+return secrets, errors
+
+
+def _parse_json_file(content: str) -> Tuple[Dict[str, Any], 
List[FileSyntaxError]]:
+"""
+Parrse a file in the JSON format.
+
+:param content:
+:return: Tuple with mapping of key and list of values and list of syntax 
errors
+"""
+if not content:
+return {}, [FileSyntaxError(line_no=1, message="The file is empty.")]
+try:
+secrets = json.loads(content)
+except JSONDecodeError as e:
+return {}, [FileSyntaxError(line_no=int(e.lineno), message=e.msg)]
+if not isinstance(secrets, dict):
+return {}, [FileSyntaxError(line_no=1, message="The file should 
contain the object.")]
+
+return secrets, []
+
+
+def _parse_secret_file(file_content: str, file_path: str) -> Tuple[Dict[str, 
Any], List[FileSyntaxError]]:
+"""
+Based on the file extension format, selects a parser and parse the file.
+
+:param file_content:
+:param file_path:
+:return:
+"""
+if file_path.lower().endswith(".env"):
+secrets, parse_errors = _parser_env_file(file_content)
+elif file_path.lower().endswith(".json"):
+secrets, parse_errors = _parse_json_file(file_content)
+else:
+raise AirflowException("Unsupported file format. The file must have 
the extension .env or .json")
+return secrets, parse_errors
+
+
+def _read_secret_file(file_path: str) -> Dict[str, str]:
+if not os.path.exists(file_path):
+raise AirflowException(
+f"File {file_path} was not found. Check the configuration of your 
secret backend."
+)
+
+log.debug("Reading file: %s", file_path)
+with open(file_path) as f:
+file_content = f.read()
+
+log.debug("Parsing file: %s", file_path)
+secrets, parse_errors = _parse_secret_file(file_content, file_path)
+log.debug("Parsed file file: len(parse_errors)=%d, len(secrets)=%d", 
len

[GitHub] [airflow] kaxil commented on a change in pull request #8436: Add Local Filesystem Secret Backend

[GitBox]

kaxil commented on a change in pull request #8436: Add Local Filesystem Secret 
Backend
URL: https://github.com/apache/airflow/pull/8436#discussion_r410735705
 
 

 ##
 File path: airflow/secrets/local_filesystem.py
 ##
 @@ -0,0 +1,257 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Objects relating to retreiving connections and variables from local file
+"""
+import json
+import logging
+import os
+from collections import defaultdict
+from json import JSONDecodeError
+from typing import Any, Dict, List, Optional, Tuple
+
+from airflow.exceptions import AirflowException, AirflowFileParseException, 
FileSyntaxError
+from airflow.secrets.base_secrets import BaseSecretsBackend
+from airflow.utils.file import COMMENT_PATTERN
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+# constants to limit cyclical imports
+CONNECTION_PARAMETERS_NAMES = {
+"conn_id",
+"conn_type",
+"host",
+"login",
+"password",
+"schema",
+"port",
+"extra",
+"uri",
+}
+
+log = logging.getLogger(__name__)
+
+
+def _parser_env_file(content: str) -> Tuple[Dict[str, List[str]], 
List[FileSyntaxError]]:
+"""
+Parse a file in the ``.env '' format.
+
+   .. code-block:: text
+
MY_CONN_ID=my-conn-type://my-login:my-pa%2Fssword@my-host:5432/my-schema?param1=val1¶m2=val2
+
+:param content:
+:return: Tuple with mapping of key and list of values and list of syntax 
errors
+"""
+secrets: Dict[str, List[str]] = defaultdict(list)
+errors: List[FileSyntaxError] = []
+for line_no, line in enumerate(content.splitlines(), 1):
+if not line:
+# Ignore empty line
+continue
+
+if COMMENT_PATTERN.match(line):
+# Ignore comments:
+continue
+
+var_parts: List[str] = line.split("=", 2)
+if len(var_parts) != 2:
+errors.append(
+FileSyntaxError(
+line_no=line_no,
+message='Invalid line format. The line should contain at 
least one equal sign ("=").',
+)
+)
+continue
+
+key, value = var_parts
+if not key:
+errors.append(FileSyntaxError(line_no=line_no, message="Invalid 
line format. Key is empty.",))
+secrets[key].append(value)
+return secrets, errors
+
+
+def _parse_json_file(content: str) -> Tuple[Dict[str, Any], 
List[FileSyntaxError]]:
+"""
+Parrse a file in the JSON format.
+
+:param content:
+:return: Tuple with mapping of key and list of values and list of syntax 
errors
+"""
+if not content:
+return {}, [FileSyntaxError(line_no=1, message="The file is empty.")]
+try:
+secrets = json.loads(content)
+except JSONDecodeError as e:
+return {}, [FileSyntaxError(line_no=int(e.lineno), message=e.msg)]
+if not isinstance(secrets, dict):
+return {}, [FileSyntaxError(line_no=1, message="The file should 
contain the object.")]
+
+return secrets, []
+
+
+def _parse_secret_file(file_content: str, file_path: str) -> Tuple[Dict[str, 
Any], List[FileSyntaxError]]:
+"""
+Based on the file extension format, selects a parser and parse the file.
+
+:param file_content:
+:param file_path:
+:return:
+"""
+if file_path.lower().endswith(".env"):
+secrets, parse_errors = _parser_env_file(file_content)
+elif file_path.lower().endswith(".json"):
+secrets, parse_errors = _parse_json_file(file_content)
+else:
+raise AirflowException("Unsupported file format. The file must have 
the extension .env or .json")
+return secrets, parse_errors
+
+
+def _read_secret_file(file_path: str) -> Dict[str, str]:
+if not os.path.exists(file_path):
+raise AirflowException(
+f"File {file_path} was not found. Check the configuration of your 
secret backend."
+)
+
+log.debug("Reading file: %s", file_path)
+with open(file_path) as f:
+file_content = f.read()
+
+log.debug("Parsing file: %s", file_path)
+secrets, parse_errors = _parse_secret_file(file_content, file_path)
+log.debug("Parsed file file: len(parse_errors)=%d, len(secrets)=%d", 
len

[GitHub] [airflow] kaxil commented on a change in pull request #8436: Add Local Filesystem Secret Backend

[GitBox]

kaxil commented on a change in pull request #8436: Add Local Filesystem Secret 
Backend
URL: https://github.com/apache/airflow/pull/8436#discussion_r410735536
 
 

 ##
 File path: airflow/secrets/local_filesystem.py
 ##
 @@ -0,0 +1,257 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Objects relating to retreiving connections and variables from local file
+"""
+import json
+import logging
+import os
+from collections import defaultdict
+from json import JSONDecodeError
+from typing import Any, Dict, List, Optional, Tuple
+
+from airflow.exceptions import AirflowException, AirflowFileParseException, 
FileSyntaxError
+from airflow.secrets.base_secrets import BaseSecretsBackend
+from airflow.utils.file import COMMENT_PATTERN
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+# constants to limit cyclical imports
+CONNECTION_PARAMETERS_NAMES = {
+"conn_id",
+"conn_type",
+"host",
+"login",
+"password",
+"schema",
+"port",
+"extra",
+"uri",
+}
+
+log = logging.getLogger(__name__)
+
+
+def _parser_env_file(content: str) -> Tuple[Dict[str, List[str]], 
List[FileSyntaxError]]:
+"""
+Parse a file in the ``.env '' format.
+
+   .. code-block:: text
+
MY_CONN_ID=my-conn-type://my-login:my-pa%2Fssword@my-host:5432/my-schema?param1=val1¶m2=val2
+
+:param content:
+:return: Tuple with mapping of key and list of values and list of syntax 
errors
+"""
+secrets: Dict[str, List[str]] = defaultdict(list)
+errors: List[FileSyntaxError] = []
+for line_no, line in enumerate(content.splitlines(), 1):
+if not line:
+# Ignore empty line
+continue
+
+if COMMENT_PATTERN.match(line):
+# Ignore comments:
+continue
+
+var_parts: List[str] = line.split("=", 2)
+if len(var_parts) != 2:
+errors.append(
+FileSyntaxError(
+line_no=line_no,
+message='Invalid line format. The line should contain at 
least one equal sign ("=").',
+)
+)
+continue
+
+key, value = var_parts
+if not key:
+errors.append(FileSyntaxError(line_no=line_no, message="Invalid 
line format. Key is empty.",))
+secrets[key].append(value)
+return secrets, errors
+
+
+def _parse_json_file(content: str) -> Tuple[Dict[str, Any], 
List[FileSyntaxError]]:
+"""
+Parrse a file in the JSON format.
+
+:param content:
+:return: Tuple with mapping of key and list of values and list of syntax 
errors
+"""
+if not content:
+return {}, [FileSyntaxError(line_no=1, message="The file is empty.")]
+try:
+secrets = json.loads(content)
+except JSONDecodeError as e:
+return {}, [FileSyntaxError(line_no=int(e.lineno), message=e.msg)]
+if not isinstance(secrets, dict):
+return {}, [FileSyntaxError(line_no=1, message="The file should 
contain the object.")]
+
+return secrets, []
+
+
+def _parse_secret_file(file_content: str, file_path: str) -> Tuple[Dict[str, 
Any], List[FileSyntaxError]]:
+"""
+Based on the file extension format, selects a parser and parse the file.
+
+:param file_content:
+:param file_path:
+:return:
+"""
+if file_path.lower().endswith(".env"):
+secrets, parse_errors = _parser_env_file(file_content)
+elif file_path.lower().endswith(".json"):
+secrets, parse_errors = _parse_json_file(file_content)
+else:
+raise AirflowException("Unsupported file format. The file must have 
the extension .env or .json")
+return secrets, parse_errors
+
+
+def _read_secret_file(file_path: str) -> Dict[str, str]:
+if not os.path.exists(file_path):
+raise AirflowException(
+f"File {file_path} was not found. Check the configuration of your 
secret backend."
+)
+
+log.debug("Reading file: %s", file_path)
+with open(file_path) as f:
+file_content = f.read()
+
+log.debug("Parsing file: %s", file_path)
+secrets, parse_errors = _parse_secret_file(file_content, file_path)
+log.debug("Parsed file file: len(parse_errors)=%d, len(secrets)=%d", 
len

[GitHub] [airflow] kaxil commented on a change in pull request #8436: Add Local Filesystem Secret Backend

[GitBox]

kaxil commented on a change in pull request #8436: Add Local Filesystem Secret 
Backend
URL: https://github.com/apache/airflow/pull/8436#discussion_r410735663
 
 

 ##
 File path: airflow/secrets/local_filesystem.py
 ##
 @@ -0,0 +1,257 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Objects relating to retreiving connections and variables from local file
+"""
+import json
+import logging
+import os
+from collections import defaultdict
+from json import JSONDecodeError
+from typing import Any, Dict, List, Optional, Tuple
+
+from airflow.exceptions import AirflowException, AirflowFileParseException, 
FileSyntaxError
+from airflow.secrets.base_secrets import BaseSecretsBackend
+from airflow.utils.file import COMMENT_PATTERN
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+# constants to limit cyclical imports
+CONNECTION_PARAMETERS_NAMES = {
+"conn_id",
+"conn_type",
+"host",
+"login",
+"password",
+"schema",
+"port",
+"extra",
+"uri",
+}
+
+log = logging.getLogger(__name__)
+
+
+def _parser_env_file(content: str) -> Tuple[Dict[str, List[str]], 
List[FileSyntaxError]]:
+"""
+Parse a file in the ``.env '' format.
+
+   .. code-block:: text
+
MY_CONN_ID=my-conn-type://my-login:my-pa%2Fssword@my-host:5432/my-schema?param1=val1¶m2=val2
+
+:param content:
+:return: Tuple with mapping of key and list of values and list of syntax 
errors
+"""
+secrets: Dict[str, List[str]] = defaultdict(list)
+errors: List[FileSyntaxError] = []
+for line_no, line in enumerate(content.splitlines(), 1):
+if not line:
+# Ignore empty line
+continue
+
+if COMMENT_PATTERN.match(line):
+# Ignore comments:
+continue
+
+var_parts: List[str] = line.split("=", 2)
+if len(var_parts) != 2:
+errors.append(
+FileSyntaxError(
+line_no=line_no,
+message='Invalid line format. The line should contain at 
least one equal sign ("=").',
+)
+)
+continue
+
+key, value = var_parts
+if not key:
+errors.append(FileSyntaxError(line_no=line_no, message="Invalid 
line format. Key is empty.",))
+secrets[key].append(value)
+return secrets, errors
+
+
+def _parse_json_file(content: str) -> Tuple[Dict[str, Any], 
List[FileSyntaxError]]:
+"""
+Parrse a file in the JSON format.
+
+:param content:
+:return: Tuple with mapping of key and list of values and list of syntax 
errors
+"""
+if not content:
+return {}, [FileSyntaxError(line_no=1, message="The file is empty.")]
+try:
+secrets = json.loads(content)
+except JSONDecodeError as e:
+return {}, [FileSyntaxError(line_no=int(e.lineno), message=e.msg)]
+if not isinstance(secrets, dict):
+return {}, [FileSyntaxError(line_no=1, message="The file should 
contain the object.")]
+
+return secrets, []
+
+
+def _parse_secret_file(file_content: str, file_path: str) -> Tuple[Dict[str, 
Any], List[FileSyntaxError]]:
+"""
+Based on the file extension format, selects a parser and parse the file.
+
+:param file_content:
+:param file_path:
+:return:
+"""
+if file_path.lower().endswith(".env"):
+secrets, parse_errors = _parser_env_file(file_content)
+elif file_path.lower().endswith(".json"):
+secrets, parse_errors = _parse_json_file(file_content)
+else:
+raise AirflowException("Unsupported file format. The file must have 
the extension .env or .json")
+return secrets, parse_errors
+
+
+def _read_secret_file(file_path: str) -> Dict[str, str]:
+if not os.path.exists(file_path):
+raise AirflowException(
+f"File {file_path} was not found. Check the configuration of your 
secret backend."
+)
+
+log.debug("Reading file: %s", file_path)
+with open(file_path) as f:
+file_content = f.read()
+
+log.debug("Parsing file: %s", file_path)
+secrets, parse_errors = _parse_secret_file(file_content, file_path)
+log.debug("Parsed file file: len(parse_errors)=%d, len(secrets)=%d", 
len

[GitHub] [airflow] codecov-io edited a comment on issue #8437: Not pulling base python images does not work for bugfixes

[GitBox]

codecov-io edited a comment on issue #8437: Not pulling base python images does 
not work for bugfixes
URL: https://github.com/apache/airflow/pull/8437#issuecomment-615924689
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/8437?src=pr&el=h1) 
Report
   > Merging 
[#8437](https://codecov.io/gh/apache/airflow/pull/8437?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/d61a476da3a649bf2c1d347b9cb3abc62eae3ce9&el=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/8437/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/8437?src=pr&el=tree)
   
   ```diff
   @@  Coverage Diff   @@
   ##   master   #8437   +/-   ##
   ==
 Coverage6.23%   6.23%   
   ==
 Files 946 946   
 Lines   45663   45663   
   ==
 Hits 28462846   
 Misses  42817   42817   
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/8437?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/8437?src=pr&el=footer). 
Last update 
[d61a476...c56af80](https://codecov.io/gh/apache/airflow/pull/8437?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #8437: Not pulling base python images does not work for bugfixes

[GitBox]

codecov-io commented on issue #8437: Not pulling base python images does not 
work for bugfixes
URL: https://github.com/apache/airflow/pull/8437#issuecomment-615924689
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/8437?src=pr&el=h1) 
Report
   > Merging 
[#8437](https://codecov.io/gh/apache/airflow/pull/8437?src=pr&el=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/d61a476da3a649bf2c1d347b9cb3abc62eae3ce9&el=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/8437/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/8437?src=pr&el=tree)
   
   ```diff
   @@  Coverage Diff   @@
   ##   master   #8437   +/-   ##
   ==
 Coverage6.23%   6.23%   
   ==
 Files 946 946   
 Lines   45663   45663   
   ==
 Hits 28462846   
 Misses  42817   42817   
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/8437?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/8437?src=pr&el=footer). 
Last update 
[d61a476...c56af80](https://codecov.io/gh/apache/airflow/pull/8437?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #8436: Add Local Filesystem Secret Backend

[GitBox]

kaxil commented on a change in pull request #8436: Add Local Filesystem Secret 
Backend
URL: https://github.com/apache/airflow/pull/8436#discussion_r410735097
 
 

 ##
 File path: airflow/secrets/local_filesystem.py
 ##
 @@ -0,0 +1,257 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Objects relating to retreiving connections and variables from local file
+"""
+import json
+import logging
+import os
+from collections import defaultdict
+from json import JSONDecodeError
+from typing import Any, Dict, List, Optional, Tuple
+
+from airflow.exceptions import AirflowException, AirflowFileParseException, 
FileSyntaxError
+from airflow.secrets.base_secrets import BaseSecretsBackend
+from airflow.utils.file import COMMENT_PATTERN
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+# constants to limit cyclical imports
+CONNECTION_PARAMETERS_NAMES = {
+"conn_id",
+"conn_type",
+"host",
+"login",
+"password",
+"schema",
+"port",
+"extra",
+"uri",
+}
+
+log = logging.getLogger(__name__)
+
+
+def _parser_env_file(content: str) -> Tuple[Dict[str, List[str]], 
List[FileSyntaxError]]:
+"""
+Parse a file in the ``.env '' format.
+
+   .. code-block:: text
+
MY_CONN_ID=my-conn-type://my-login:my-pa%2Fssword@my-host:5432/my-schema?param1=val1¶m2=val2
+
+:param content:
+:return: Tuple with mapping of key and list of values and list of syntax 
errors
+"""
+secrets: Dict[str, List[str]] = defaultdict(list)
+errors: List[FileSyntaxError] = []
+for line_no, line in enumerate(content.splitlines(), 1):
+if not line:
+# Ignore empty line
+continue
+
+if COMMENT_PATTERN.match(line):
+# Ignore comments:
+continue
+
+var_parts: List[str] = line.split("=", 2)
+if len(var_parts) != 2:
+errors.append(
+FileSyntaxError(
+line_no=line_no,
+message='Invalid line format. The line should contain at 
least one equal sign ("=").',
+)
+)
+continue
+
+key, value = var_parts
+if not key:
+errors.append(FileSyntaxError(line_no=line_no, message="Invalid 
line format. Key is empty.",))
+secrets[key].append(value)
+return secrets, errors
+
+
+def _parse_json_file(content: str) -> Tuple[Dict[str, Any], 
List[FileSyntaxError]]:
+"""
+Parrse a file in the JSON format.
+
+:param content:
+:return: Tuple with mapping of key and list of values and list of syntax 
errors
+"""
+if not content:
+return {}, [FileSyntaxError(line_no=1, message="The file is empty.")]
+try:
+secrets = json.loads(content)
+except JSONDecodeError as e:
+return {}, [FileSyntaxError(line_no=int(e.lineno), message=e.msg)]
+if not isinstance(secrets, dict):
+return {}, [FileSyntaxError(line_no=1, message="The file should 
contain the object.")]
+
+return secrets, []
+
+
+def _parse_secret_file(file_content: str, file_path: str) -> Tuple[Dict[str, 
Any], List[FileSyntaxError]]:
+"""
+Based on the file extension format, selects a parser and parse the file.
+
+:param file_content:
+:param file_path:
+:return:
+"""
+if file_path.lower().endswith(".env"):
+secrets, parse_errors = _parser_env_file(file_content)
+elif file_path.lower().endswith(".json"):
+secrets, parse_errors = _parse_json_file(file_content)
+else:
+raise AirflowException("Unsupported file format. The file must have 
the extension .env or .json")
+return secrets, parse_errors
+
+
+def _read_secret_file(file_path: str) -> Dict[str, str]:
+if not os.path.exists(file_path):
+raise AirflowException(
+f"File {file_path} was not found. Check the configuration of your 
secret backend."
+)
+
+log.debug("Reading file: %s", file_path)
+with open(file_path) as f:
+file_content = f.read()
 
 Review comment:
   I think we should pass the file path directly to `_parser_env_file` and 
`_parse_json_file` and let both the method handle opening of files. 
   
   How about using https://pypi

[GitHub] [airflow] kaxil commented on a change in pull request #8436: Add Local Filesystem Secret Backend

[GitBox]

kaxil commented on a change in pull request #8436: Add Local Filesystem Secret 
Backend
URL: https://github.com/apache/airflow/pull/8436#discussion_r410735097
 
 

 ##
 File path: airflow/secrets/local_filesystem.py
 ##
 @@ -0,0 +1,257 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Objects relating to retreiving connections and variables from local file
+"""
+import json
+import logging
+import os
+from collections import defaultdict
+from json import JSONDecodeError
+from typing import Any, Dict, List, Optional, Tuple
+
+from airflow.exceptions import AirflowException, AirflowFileParseException, 
FileSyntaxError
+from airflow.secrets.base_secrets import BaseSecretsBackend
+from airflow.utils.file import COMMENT_PATTERN
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+# constants to limit cyclical imports
+CONNECTION_PARAMETERS_NAMES = {
+"conn_id",
+"conn_type",
+"host",
+"login",
+"password",
+"schema",
+"port",
+"extra",
+"uri",
+}
+
+log = logging.getLogger(__name__)
+
+
+def _parser_env_file(content: str) -> Tuple[Dict[str, List[str]], 
List[FileSyntaxError]]:
+"""
+Parse a file in the ``.env '' format.
+
+   .. code-block:: text
+
MY_CONN_ID=my-conn-type://my-login:my-pa%2Fssword@my-host:5432/my-schema?param1=val1¶m2=val2
+
+:param content:
+:return: Tuple with mapping of key and list of values and list of syntax 
errors
+"""
+secrets: Dict[str, List[str]] = defaultdict(list)
+errors: List[FileSyntaxError] = []
+for line_no, line in enumerate(content.splitlines(), 1):
+if not line:
+# Ignore empty line
+continue
+
+if COMMENT_PATTERN.match(line):
+# Ignore comments:
+continue
+
+var_parts: List[str] = line.split("=", 2)
+if len(var_parts) != 2:
+errors.append(
+FileSyntaxError(
+line_no=line_no,
+message='Invalid line format. The line should contain at 
least one equal sign ("=").',
+)
+)
+continue
+
+key, value = var_parts
+if not key:
+errors.append(FileSyntaxError(line_no=line_no, message="Invalid 
line format. Key is empty.",))
+secrets[key].append(value)
+return secrets, errors
+
+
+def _parse_json_file(content: str) -> Tuple[Dict[str, Any], 
List[FileSyntaxError]]:
+"""
+Parrse a file in the JSON format.
+
+:param content:
+:return: Tuple with mapping of key and list of values and list of syntax 
errors
+"""
+if not content:
+return {}, [FileSyntaxError(line_no=1, message="The file is empty.")]
+try:
+secrets = json.loads(content)
+except JSONDecodeError as e:
+return {}, [FileSyntaxError(line_no=int(e.lineno), message=e.msg)]
+if not isinstance(secrets, dict):
+return {}, [FileSyntaxError(line_no=1, message="The file should 
contain the object.")]
+
+return secrets, []
+
+
+def _parse_secret_file(file_content: str, file_path: str) -> Tuple[Dict[str, 
Any], List[FileSyntaxError]]:
+"""
+Based on the file extension format, selects a parser and parse the file.
+
+:param file_content:
+:param file_path:
+:return:
+"""
+if file_path.lower().endswith(".env"):
+secrets, parse_errors = _parser_env_file(file_content)
+elif file_path.lower().endswith(".json"):
+secrets, parse_errors = _parse_json_file(file_content)
+else:
+raise AirflowException("Unsupported file format. The file must have 
the extension .env or .json")
+return secrets, parse_errors
+
+
+def _read_secret_file(file_path: str) -> Dict[str, str]:
+if not os.path.exists(file_path):
+raise AirflowException(
+f"File {file_path} was not found. Check the configuration of your 
secret backend."
+)
+
+log.debug("Reading file: %s", file_path)
+with open(file_path) as f:
+file_content = f.read()
 
 Review comment:
   I think we should pass the file path directly to `_parser_env_file` and 
`_parse_json_file` and let both the method handle opening of files. 
   
   How about using https://pypi

[GitHub] [airflow] kaxil commented on a change in pull request #8436: Add Local Filesystem Secret Backend

[GitBox]

kaxil commented on a change in pull request #8436: Add Local Filesystem Secret 
Backend
URL: https://github.com/apache/airflow/pull/8436#discussion_r410735097
 
 

 ##
 File path: airflow/secrets/local_filesystem.py
 ##
 @@ -0,0 +1,257 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Objects relating to retreiving connections and variables from local file
+"""
+import json
+import logging
+import os
+from collections import defaultdict
+from json import JSONDecodeError
+from typing import Any, Dict, List, Optional, Tuple
+
+from airflow.exceptions import AirflowException, AirflowFileParseException, 
FileSyntaxError
+from airflow.secrets.base_secrets import BaseSecretsBackend
+from airflow.utils.file import COMMENT_PATTERN
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+# constants to limit cyclical imports
+CONNECTION_PARAMETERS_NAMES = {
+"conn_id",
+"conn_type",
+"host",
+"login",
+"password",
+"schema",
+"port",
+"extra",
+"uri",
+}
+
+log = logging.getLogger(__name__)
+
+
+def _parser_env_file(content: str) -> Tuple[Dict[str, List[str]], 
List[FileSyntaxError]]:
+"""
+Parse a file in the ``.env '' format.
+
+   .. code-block:: text
+
MY_CONN_ID=my-conn-type://my-login:my-pa%2Fssword@my-host:5432/my-schema?param1=val1¶m2=val2
+
+:param content:
+:return: Tuple with mapping of key and list of values and list of syntax 
errors
+"""
+secrets: Dict[str, List[str]] = defaultdict(list)
+errors: List[FileSyntaxError] = []
+for line_no, line in enumerate(content.splitlines(), 1):
+if not line:
+# Ignore empty line
+continue
+
+if COMMENT_PATTERN.match(line):
+# Ignore comments:
+continue
+
+var_parts: List[str] = line.split("=", 2)
+if len(var_parts) != 2:
+errors.append(
+FileSyntaxError(
+line_no=line_no,
+message='Invalid line format. The line should contain at 
least one equal sign ("=").',
+)
+)
+continue
+
+key, value = var_parts
+if not key:
+errors.append(FileSyntaxError(line_no=line_no, message="Invalid 
line format. Key is empty.",))
+secrets[key].append(value)
+return secrets, errors
+
+
+def _parse_json_file(content: str) -> Tuple[Dict[str, Any], 
List[FileSyntaxError]]:
+"""
+Parrse a file in the JSON format.
+
+:param content:
+:return: Tuple with mapping of key and list of values and list of syntax 
errors
+"""
+if not content:
+return {}, [FileSyntaxError(line_no=1, message="The file is empty.")]
+try:
+secrets = json.loads(content)
+except JSONDecodeError as e:
+return {}, [FileSyntaxError(line_no=int(e.lineno), message=e.msg)]
+if not isinstance(secrets, dict):
+return {}, [FileSyntaxError(line_no=1, message="The file should 
contain the object.")]
+
+return secrets, []
+
+
+def _parse_secret_file(file_content: str, file_path: str) -> Tuple[Dict[str, 
Any], List[FileSyntaxError]]:
+"""
+Based on the file extension format, selects a parser and parse the file.
+
+:param file_content:
+:param file_path:
+:return:
+"""
+if file_path.lower().endswith(".env"):
+secrets, parse_errors = _parser_env_file(file_content)
+elif file_path.lower().endswith(".json"):
+secrets, parse_errors = _parse_json_file(file_content)
+else:
+raise AirflowException("Unsupported file format. The file must have 
the extension .env or .json")
+return secrets, parse_errors
+
+
+def _read_secret_file(file_path: str) -> Dict[str, str]:
+if not os.path.exists(file_path):
+raise AirflowException(
+f"File {file_path} was not found. Check the configuration of your 
secret backend."
+)
+
+log.debug("Reading file: %s", file_path)
+with open(file_path) as f:
+file_content = f.read()
 
 Review comment:
   I think we should pass the file path directly to `_parser_env_file` and 
`_parse_json_file` and let both the method handle opening of files. 
   
   How about using https://pypi

[GitHub] [airflow] danfrankj commented on issue #8435: Fix performance degradation when updating dagrun state

[GitBox]

danfrankj commented on issue #8435: Fix performance degradation when updating 
dagrun state
URL: https://github.com/apache/airflow/pull/8435#issuecomment-615923750
 
 
   @BasPH we're at ~5000 tasks so when we upgraded recently, it ground to a 
halt. 
   
   Also, pretty sure the CI failure here is unrelated?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] danfrankj commented on issue #8435: Fix performance degradation when updating dagrun state

[GitBox]

danfrankj commented on issue #8435: Fix performance degradation when updating 
dagrun state
URL: https://github.com/apache/airflow/pull/8435#issuecomment-615923853
 
 
   > Whoops, good catch
   
   Catch credit goes to @cmlad :) 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #8436: Add Local Filesystem Secret Backend

[GitBox]

mik-laj commented on a change in pull request #8436: Add Local Filesystem 
Secret Backend
URL: https://github.com/apache/airflow/pull/8436#discussion_r410734516
 
 

 ##
 File path: airflow/secrets/local_filesystem.py
 ##
 @@ -0,0 +1,257 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Objects relating to retreiving connections and variables from local file
+"""
+import json
+import logging
+import os
+from collections import defaultdict
+from json import JSONDecodeError
+from typing import Any, Dict, List, Optional, Tuple
+
+from airflow.exceptions import AirflowException, AirflowFileParseException, 
FileSyntaxError
+from airflow.secrets.base_secrets import BaseSecretsBackend
+from airflow.utils.file import COMMENT_PATTERN
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+# constants to limit cyclical imports
+CONNECTION_PARAMETERS_NAMES = {
+"conn_id",
+"conn_type",
+"host",
+"login",
+"password",
+"schema",
+"port",
+"extra",
+"uri",
+}
+
+log = logging.getLogger(__name__)
+
+
+def _parser_env_file(content: str) -> Tuple[Dict[str, List[str]], 
List[FileSyntaxError]]:
 
 Review comment:
   I want to refactor CLI and Web UI to use the same code. in the next PR. For 
Web UI, it would be useful for the user to paste the contents of the file 
instead of uploading the file.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #8436: Add Local Filesystem Secret Backend

[GitBox]

mik-laj commented on a change in pull request #8436: Add Local Filesystem 
Secret Backend
URL: https://github.com/apache/airflow/pull/8436#discussion_r410734516
 
 

 ##
 File path: airflow/secrets/local_filesystem.py
 ##
 @@ -0,0 +1,257 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Objects relating to retreiving connections and variables from local file
+"""
+import json
+import logging
+import os
+from collections import defaultdict
+from json import JSONDecodeError
+from typing import Any, Dict, List, Optional, Tuple
+
+from airflow.exceptions import AirflowException, AirflowFileParseException, 
FileSyntaxError
+from airflow.secrets.base_secrets import BaseSecretsBackend
+from airflow.utils.file import COMMENT_PATTERN
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+# constants to limit cyclical imports
+CONNECTION_PARAMETERS_NAMES = {
+"conn_id",
+"conn_type",
+"host",
+"login",
+"password",
+"schema",
+"port",
+"extra",
+"uri",
+}
+
+log = logging.getLogger(__name__)
+
+
+def _parser_env_file(content: str) -> Tuple[Dict[str, List[str]], 
List[FileSyntaxError]]:
 
 Review comment:
   I want to refactor CLI and Web UI to use the same code. I want to refactor 
CLI and Web UI to use the same code. in the next PR. For Web UI, it would be 
useful for the user to paste the contents of the file instead of uploading the 
file.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj opened a new issue #8439: Clean up the test_variable_commnand.py file.

[GitBox]

mik-laj opened a new issue #8439: Clean up the test_variable_commnand.py file.
URL: https://github.com/apache/airflow/issues/8439
 
 
   Hello,
   
   There is some mess in this file. All commands are tested in one method, 
which makes it difficult to develop these commands.
   
https://raw.githubusercontent.com/apache/airflow/17fc9c541ea4aecbd5ae566a11f6601b4038b197/tests/cli/commands/test_variable_command.py
   I would be happy if each command was tested in a separate method and if 
there were no side effects between tests.
   
   Additional points can be obtained if you add an assertion that will check 
the content of the file during export and import.
   
   Best regards,
   Kamil


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #8436: Add Local Filesystem Secret Backend

[GitBox]

kaxil commented on a change in pull request #8436: Add Local Filesystem Secret 
Backend
URL: https://github.com/apache/airflow/pull/8436#discussion_r410734028
 
 

 ##
 File path: airflow/secrets/local_filesystem.py
 ##
 @@ -0,0 +1,257 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Objects relating to retreiving connections and variables from local file
+"""
+import json
+import logging
+import os
+from collections import defaultdict
+from json import JSONDecodeError
+from typing import Any, Dict, List, Optional, Tuple
+
+from airflow.exceptions import AirflowException, AirflowFileParseException, 
FileSyntaxError
+from airflow.secrets.base_secrets import BaseSecretsBackend
+from airflow.utils.file import COMMENT_PATTERN
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+# constants to limit cyclical imports
+CONNECTION_PARAMETERS_NAMES = {
+"conn_id",
+"conn_type",
+"host",
+"login",
+"password",
+"schema",
+"port",
+"extra",
+"uri",
+}
+
+log = logging.getLogger(__name__)
+
+
+def _parser_env_file(content: str) -> Tuple[Dict[str, List[str]], 
List[FileSyntaxError]]:
 
 Review comment:
   If the name is `_parser_env_file`, how about we take the file as an input?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #8436: Add Local Filesystem Secret Backend

[GitBox]

kaxil commented on a change in pull request #8436: Add Local Filesystem Secret 
Backend
URL: https://github.com/apache/airflow/pull/8436#discussion_r410733798
 
 

 ##
 File path: airflow/secrets/local_filesystem.py
 ##
 @@ -0,0 +1,257 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Objects relating to retreiving connections and variables from local file
+"""
+import json
+import logging
+import os
+from collections import defaultdict
+from json import JSONDecodeError
+from typing import Any, Dict, List, Optional, Tuple
+
+from airflow.exceptions import AirflowException, AirflowFileParseException, 
FileSyntaxError
+from airflow.secrets.base_secrets import BaseSecretsBackend
+from airflow.utils.file import COMMENT_PATTERN
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+# constants to limit cyclical imports
+CONNECTION_PARAMETERS_NAMES = {
+"conn_id",
+"conn_type",
+"host",
+"login",
+"password",
+"schema",
+"port",
+"extra",
+"uri",
+}
+
+log = logging.getLogger(__name__)
+
+
+def _parser_env_file(content: str) -> Tuple[Dict[str, List[str]], 
List[FileSyntaxError]]:
+"""
+Parse a file in the ``.env '' format.
+
+   .. code-block:: text
+
MY_CONN_ID=my-conn-type://my-login:my-pa%2Fssword@my-host:5432/my-schema?param1=val1¶m2=val2
+
+:param content:
 
 Review comment:
   Description is missing


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #8436: Add Local Filesystem Secret Backend

[GitBox]

kaxil commented on a change in pull request #8436: Add Local Filesystem Secret 
Backend
URL: https://github.com/apache/airflow/pull/8436#discussion_r410733413
 
 

 ##
 File path: docs/howto/use-alternative-secrets-backend.rst
 ##
 @@ -52,7 +52,105 @@ Set ``backend`` to the fully qualified class name of the 
backend you want to ena
 You can provide ``backend_kwargs`` with json and it will be passed as kwargs 
to the ``__init__`` method of
 your secrets backend.
 
-See :ref:`AWS SSM Parameter Store ` for an 
example configuration.
+.. _local_disk_secrets:
+
+Local Filesystem Secrets Backend
+
+
+This backend is especially useful in the following use cases:
+
+* **development**: It ensures data synchronization between all terminal 
windows (same as databases),
+  and at the same time the values are retained after database restart (same as 
environment variable)
+* **Kubernetes**: It allows you to store secrets in `Kubernetes Secrets 
`__
+  or you can synchronize values using the sidecar container and
+  `a shared volume 
`__
+* **JSON**: If you're tired of defining all connections using a URI - creating 
JSON object is easier than using URI.
+
+To use variable and connection from local file, specify 
:py:class:`~airflow.secrets.local_filesystem.LocalFilesystemBackend`
+as the ``backend`` in  ``[secrets]`` section of ``airflow.cfg``.
+
+Available parameters to ``backend_kwargs``:
+
+* ``variable_file_path``: File location with variables data.
+* ``connection_file_path``: File location with connection data.
+
+Here is a sample configuration:
+
+.. code-block:: ini
+
+[secrets]
+backend = airflow.secrets.local_filesystem.LocalFilesystemBackend
+backend_kwargs = {"variable_file_path": "/files/var.json", 
"connection_file_path": "/files/conn.json"}
+
+Both ``JSON`` and ``.env`` files are supported. All parameters are optional. 
If the file path is not passed,
+the backend returns an empty collection.
+
+Storing and Retrieving Connections
+""
+
+If you have set ``connection_file_path`` as ``/files/my_conn.json``, then the 
backend will read the
+file ``/files/my_conn.json`` when it looks for connections.
+
+The file can be defined in ``JSON`` or ``env`` format.
+
+The JSON file must contain an object where the key contains the connection ID 
and the value contains
+the definitions of one or more connections. The connection can be defined as a 
URL (string) or object.
+For a guide about defining a connection as a URI, see:: 
:doc:`generating_connection_uri`.
+For a description of the connection object parameters see 
:class:`~airflow.models.connection.Connection`.
+The following is a sample JSON file.
+
+.. code-block:: json
+
+{
+"CONN_A": "mysq://host_a",
+"CONN_B": [
+"mysq://host_a",
+"mysq://host_a"
+],
+"CONN_C": {
+"conn_type": "scheme",
+"host": "host",
+"schema": "lschema",
+"login": "Login",
+"password": "None",
+"port": "1234"
+}
+}
+
+You can also define connections using a ``.env`` file. Then the key is the 
connection ID, and
+the value should describe the connection using the URI. If the connection ID 
is repeated, all values will
+be returned. The following is a sample file.
+
+  .. code-block:: text
+
+mysql_conn_id=mysql//log:password@13.1.21.1:3306/mysqldbrd
+
google_custom_key=google-cloud-platform://?extra__google_cloud_platform__key_path=%2Fkeys%2Fkey.json
+
+Storing and Retrieving Variables
+
+
+If you have set ``variable_file_path`` as ``/files/my_var.json``, then the 
backend will read the
+file ``/files/my_var.json`` when it looks for variables.
+
+The file can be defined in ``JSON`` or ``env`` format.
+
+The JSON file must contain an object where the key contains the variable key 
and the value contains
+the variable value. The following is a sample JSON file.
+
+  .. code-block:: json
+
+{
+"VAR_A": "some_value",
+"var_b": "differnet_value"
+}
+
+You can also define variable using a ``.env`` file. Then the key is the 
variable key, and variable should
+describe the variable value. The following is a sample  file.
 
 Review comment:
   ```suggestion
   describe the variable value. The following is a sample file.
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on a change in pull request #8436: Add Local Filesystem Secret Backend

[GitBox]

kaxil commented on a change in pull request #8436: Add Local Filesystem Secret 
Backend
URL: https://github.com/apache/airflow/pull/8436#discussion_r410733299
 
 

 ##
 File path: docs/howto/use-alternative-secrets-backend.rst
 ##
 @@ -52,7 +52,105 @@ Set ``backend`` to the fully qualified class name of the 
backend you want to ena
 You can provide ``backend_kwargs`` with json and it will be passed as kwargs 
to the ``__init__`` method of
 your secrets backend.
 
-See :ref:`AWS SSM Parameter Store ` for an 
example configuration.
+.. _local_disk_secrets:
+
+Local Filesystem Secrets Backend
+
+
+This backend is especially useful in the following use cases:
+
+* **development**: It ensures data synchronization between all terminal 
windows (same as databases),
+  and at the same time the values are retained after database restart (same as 
environment variable)
+* **Kubernetes**: It allows you to store secrets in `Kubernetes Secrets 
`__
+  or you can synchronize values using the sidecar container and
+  `a shared volume 
`__
+* **JSON**: If you're tired of defining all connections using a URI - creating 
JSON object is easier than using URI.
+
+To use variable and connection from local file, specify 
:py:class:`~airflow.secrets.local_filesystem.LocalFilesystemBackend`
+as the ``backend`` in  ``[secrets]`` section of ``airflow.cfg``.
+
+Available parameters to ``backend_kwargs``:
+
+* ``variable_file_path``: File location with variables data.
+* ``connection_file_path``: File location with connection data.
+
+Here is a sample configuration:
+
+.. code-block:: ini
+
+[secrets]
+backend = airflow.secrets.local_filesystem.LocalFilesystemBackend
+backend_kwargs = {"variable_file_path": "/files/var.json", 
"connection_file_path": "/files/conn.json"}
+
+Both ``JSON`` and ``.env`` files are supported. All parameters are optional. 
If the file path is not passed,
+the backend returns an empty collection.
+
+Storing and Retrieving Connections
+""
+
+If you have set ``connection_file_path`` as ``/files/my_conn.json``, then the 
backend will read the
+file ``/files/my_conn.json`` when it looks for connections.
+
+The file can be defined in ``JSON`` or ``env`` format.
+
+The JSON file must contain an object where the key contains the connection ID 
and the value contains
+the definitions of one or more connections. The connection can be defined as a 
URL (string) or object.
+For a guide about defining a connection as a URI, see:: 
:doc:`generating_connection_uri`.
+For a description of the connection object parameters see 
:class:`~airflow.models.connection.Connection`.
+The following is a sample JSON file.
+
+.. code-block:: json
+
+{
+"CONN_A": "mysq://host_a",
+"CONN_B": [
+"mysq://host_a",
+"mysq://host_a"
+],
+"CONN_C": {
+"conn_type": "scheme",
+"host": "host",
+"schema": "lschema",
+"login": "Login",
+"password": "None",
+"port": "1234"
+}
+}
+
+You can also define connections using a ``.env`` file. Then the key is the 
connection ID, and
+the value should describe the connection using the URI. If the connection ID 
is repeated, all values will
+be returned. The following is a sample file.
+
+  .. code-block:: text
+
+mysql_conn_id=mysql//log:password@13.1.21.1:3306/mysqldbrd
 
 Review comment:
   ```suggestion
   mysql_conn_id=mysql://log:password@13.1.21.1:3306/mysqldbrd
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj opened a new pull request #8438: Return non-zero error code when variable is missing

[GitBox]

mik-laj opened a new pull request #8438: Return non-zero error code when 
variable is missing
URL: https://github.com/apache/airflow/pull/8438
 
 
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [X] Description above provides context of the change
   - [X] Unit tests coverage for changes (not needed for documentation changes)
   - [X] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [X] Relevant documentation is updated including usage instructions.
   - [X] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] BasPH commented on issue #8435: Fix performance degradation when updating dagrun state

[GitBox]

BasPH commented on issue #8435: Fix performance degradation when updating 
dagrun state
URL: https://github.com/apache/airflow/pull/8435#issuecomment-615921944
 
 
   @danfrankj out of interest: how big is your DAG?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #8436: Add Local Filesystem Secret Backend

[GitBox]

mik-laj commented on a change in pull request #8436: Add Local Filesystem 
Secret Backend
URL: https://github.com/apache/airflow/pull/8436#discussion_r410733255
 
 

 ##
 File path: airflow/secrets/base_secrets.py
 ##
 @@ -43,7 +43,7 @@ def build_path(path_prefix: str, secret_id: str, sep: str = 
"/") -> str:
 """
 return f"{path_prefix}{sep}{secret_id}"
 
-def get_conn_uri(self, conn_id: str) -> Optional[str]:
+def get_conn_uri(self, conn_id: str) -> Optional[Union[str, List[str]]]:
 
 Review comment:
   I will revert it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #8437: Not pulling base python images does not work for bugfixes

[GitBox]

potiuk commented on issue #8437: Not pulling base python images does not work 
for bugfixes
URL: https://github.com/apache/airflow/pull/8437#issuecomment-615919400
 
 
   Small one. There was another  bugfix update for base python image and I 
found out that not pulling those images is bad as you will use an old, 
previosly downloaded python image :( which means full rebuild. I have changed 
it now so when "force pull" is used the base python image is also downloaded. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk opened a new pull request #8437: Not pulling base python images does not work for bugfixes

[GitBox]

potiuk opened a new pull request #8437: Not pulling base python images does not 
work for bugfixes
URL: https://github.com/apache/airflow/pull/8437
 
 
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5156) Add other authentication mechanisms to HttpHook

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086556#comment-17086556
 ] 

ASF subversion and git services commented on AIRFLOW-5156:
--

Commit ddd005e3b97e82ce715dc6604ff60ed5768de6ea in airflow's branch 
refs/heads/master from S S Rohit
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=ddd005e ]

[AIRFLOW-5156] Fixed doc strigns for HttpHook (#8434)



> Add other authentication mechanisms to HttpHook
> ---
>
> Key: AIRFLOW-5156
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5156
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: hooks
>Affects Versions: 1.10.4
>Reporter: Joshua Kornblum
>Assignee: Rohit S S
>Priority: Minor
>
> It looks like the only supported authentication for HttpHooks is basic auth.
> The hook code shows 
> {quote}_if conn.login:_
>   _session.auth = (conn.login, conn.password)_
> {quote}
> requests library supports any auth that inherits AuthBase – in my scenario we 
> need ntlmauth for API on IIS server. 
> [https://2.python-requests.org/en/master/user/advanced/#custom-authentication]
> I would suggest option to pass auth object in constructor then add to if/else 
> control flow like
> {quote}_if self.auth is not None:_
>   _session.auth = self.auth_
> _elif conn.login:_
>   _session.auth = (conn.login, conn.password)_
> {quote}
> One would have to fetch the connection themselves and then fill out auth and 
> then pass that to hook which is flexible although a little awkard.
> {quote}api_conn = BaseHook().get_connection('my_api')
> auth = HttpNtlmAuth(api_conn.login, api_conn.password)
> HttpSensor(task_id='sensing', auth=auth, )
> {quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] mik-laj opened a new pull request #8436: Add Local Filesystem Secret Backend

[GitBox]

mik-laj opened a new pull request #8436: Add Local Filesystem Secret Backend
URL: https://github.com/apache/airflow/pull/8436
 
 
   This backend is especially useful in the following use cases:
   
   * during **development**: It ensures data synchronization between all 
terminal windows (same as databases),
 and at the same time the values are retained after database restart (same 
as environment variable)
   * for **Kubernetes**: It allows you to store secrets in [Kubernetes 
Secrets](https://kubernetes.io/docs/concepts/configuration/secret/)
 or you can synchronize values using the sidecar container and
 [a shared 
volume](https://kubernetes.io/docs/tasks/access-application-cluster/communicate-containers-same-pod-shared-volume/)
 between two containers
   * if you're tired of defining all connections using a URI.
   
   
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [X] Description above provides context of the change
   - [X] Unit tests coverage for changes (not needed for documentation changes)
   - [X] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [X] Relevant documentation is updated including usage instructions.
   - [X] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #8432: Provide GCP credentials in Bash/Python operators

[GitBox]

potiuk commented on issue #8432: Provide GCP credentials in Bash/Python 
operators
URL: https://github.com/apache/airflow/pull/8432#issuecomment-615914795
 
 
   > I will present something generic for moments, which will not require the 
transfer of a complex object. It's as easy to use as it is now, but it will be 
more generic. It will be something similar to the code below.
   > 
   
   That looks much better indeed. Let's see how this will look like eventually 
:) 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-5156) Add other authentication mechanisms to HttpHook

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086554#comment-17086554
 ] 

ASF GitHub Bot commented on AIRFLOW-5156:
-

potiuk commented on pull request #8434: [AIRFLOW-5156] Fixed doc strigns for 
HttpHook
URL: https://github.com/apache/airflow/pull/8434
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add other authentication mechanisms to HttpHook
> ---
>
> Key: AIRFLOW-5156
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5156
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: hooks
>Affects Versions: 1.10.4
>Reporter: Joshua Kornblum
>Assignee: Rohit S S
>Priority: Minor
>
> It looks like the only supported authentication for HttpHooks is basic auth.
> The hook code shows 
> {quote}_if conn.login:_
>   _session.auth = (conn.login, conn.password)_
> {quote}
> requests library supports any auth that inherits AuthBase – in my scenario we 
> need ntlmauth for API on IIS server. 
> [https://2.python-requests.org/en/master/user/advanced/#custom-authentication]
> I would suggest option to pass auth object in constructor then add to if/else 
> control flow like
> {quote}_if self.auth is not None:_
>   _session.auth = self.auth_
> _elif conn.login:_
>   _session.auth = (conn.login, conn.password)_
> {quote}
> One would have to fetch the connection themselves and then fill out auth and 
> then pass that to hook which is flexible although a little awkard.
> {quote}api_conn = BaseHook().get_connection('my_api')
> auth = HttpNtlmAuth(api_conn.login, api_conn.password)
> HttpSensor(task_id='sensing', auth=auth, )
> {quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] mik-laj edited a comment on issue #8432: Provide GCP credentials in Bash/Python operators

[GitBox]

mik-laj edited a comment on issue #8432: Provide GCP credentials in Bash/Python 
operators
URL: https://github.com/apache/airflow/pull/8432#issuecomment-615910786
 
 
   I will present something generic for moments, which will not require the 
transfer of a complex object. It's as easy to use as it is now, but it will be 
more generic. It will be something similar to the code below.
   ```
   exit_stack.enter_context(  # pylint: disable=no-member
   BaseHook.get_connection(conn).get_hook(
   conn_id=self.conn_id,
   ).provide_authorization()
   )
   ```
   New services will continue to be added by composition rather than 
inheritance.
   
   Thank you very much for the comments. This shows that this feature can be 
useful.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk merged pull request #8434: [AIRFLOW-5156] Fixed doc strigns for HttpHook

[GitBox]

potiuk merged pull request #8434: [AIRFLOW-5156] Fixed doc strigns for HttpHook
URL: https://github.com/apache/airflow/pull/8434
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #8432: Provide GCP credentials in Bash/Python operators

[GitBox]

mik-laj commented on issue #8432: Provide GCP credentials in Bash/Python 
operators
URL: https://github.com/apache/airflow/pull/8432#issuecomment-615910786
 
 
   I will present something generic for moments, which will not require the 
transfer of a complex object. It's as easy to use as it is now, but it will be 
more generic. It will be something similar to the code below.
   ```
   exit_stack.enter_context(  # pylint: disable=no-member
   BaseHook.get_connection(conn).get_hook(
   gcp_conn_id=self.gcp_conn_id, 
delegate_to=self.gcp_gcp_delegate_to
   ).provide_authorization()
   )
   ```
   New services will continue to be added by composition rather than 
inheritance.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj edited a comment on issue #8432: Provide GCP credentials in Bash/Python operators

[GitBox]

mik-laj edited a comment on issue #8432: Provide GCP credentials in Bash/Python 
operators
URL: https://github.com/apache/airflow/pull/8432#issuecomment-615910786
 
 
   I will present something generic for moments, which will not require the 
transfer of a complex object. It's as easy to use as it is now, but it will be 
more generic. It will be something similar to the code below.
   ```
   exit_stack.enter_context(  # pylint: disable=no-member
   BaseHook.get_connection(conn).get_hook(
   gcp_conn_id=self.gcp_conn_id, 
delegate_to=self.gcp_gcp_delegate_to
   ).provide_authorization()
   )
   ```
   New services will continue to be added by composition rather than 
inheritance.
   
   Thank you very much for the comments. This shows that this feature can be 
useful.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #8393: Bring back CI optimisations

[GitBox]

potiuk commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r410725421
 
 

 ##
 File path: .github/workflows/ci.yml
 ##
 @@ -17,272 +17,275 @@
 #
 
 ---
-name: Airflow
+name: CI Build
 on:
+  schedule:
+- cron: '10 2 * * *'
   push:
-branches: ['*']
+branches: ['master', 'v1-10-test', 'v1-10-stable']
   pull_request:
-branches: [master]
+branches: ['master', 'v1-10-test', 'v1-10-stable']
 env:
-  BUILD_ID: ${{github.sha }}
   MOUNT_LOCAL_SOURCES: "false"
-  MOUNT_HOST_AIRFLOW_VOLUME: "true"
   FORCE_ANSWER_TO_QUESTIONS: "yes"
   SKIP_CHECK_REMOTE_IMAGE: "true"
+  SKIP_CI_IMAGE_CHECK: "true"
   DB_RESET: "true"
   VERBOSE: "true"
-  CI: "true"
-  # Should be a target branch
-  TRAVIS_BRANCH: "master"
-  TRAVIS: "true"
+  UPGRADE_TO_LATEST_REQUIREMENTS: "false"
+  ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap presto rabbitmq 
redis"
+  PYTHON_MAJOR_MINOR_VERSION: 3.6
+  # TODO: Enable cache registry once a master build succeeds uploading the 
images to Github registry cache
+  DISABLE_CACHE_REGISTRY: "true"
+  CACHE_REGISTRY: "docker.pkg.github.com"
+  CACHE_IMAGE_PREFIX: ${{ github.repository }}
+  CACHE_REGISTRY_USERNAME: ${{ github.actor }}
+  CACHE_REGISTRY_PASSWORD: ${{ secrets.GITHUB_TOKEN }}
+
 jobs:
-  statics:
-name: Static checks
+
+  static-checks-1:
+name: "Checks: pylint, mypy"
 runs-on: ubuntu-latest
 env:
-  PYTHON_VERSION: 3.6
-  AIRFLOW_MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
-  TRAVIS_JOB_NAME: "Static"
+  MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
+  CI_JOB_TYPE: "Pylint checks for tests"
 steps:
   - uses: actions/checkout@master
-  - name: free disk space
-run: |
-  sudo swapoff -a
-  sudo rm -f /swapfile
-  sudo apt clean
-  docker rmi $(docker image ls -aq)
-  df -h
   - uses: actions/setup-python@v1
 with:
-  python-version: ${{ env.PYTHON_VERSION }}
-  - name: "Before install"
-run: ./scripts/ci/ci_before_install.sh
-  - name: "Static checks"
-if: success()
+  python-version: '3.x'
+  - name: Cache pre-commit env
+uses: actions/cache@v1
 env:
-  PYTHON_VERSION: 3.6
-run: ./scripts/ci/ci_run_all_static_checks.sh
+  cache-name: cache-pre-commit
+with:
+  path: ~/.cache/pre-commit
+  key: ${{ env.cache-name }}-${{ github.job }}-${{ 
hashFiles('.pre-commit-config.yaml') }}
+  - name: "Free space"
+run: ./scripts/ci/ci_free_space_on_ci.sh
+  - name: "Build CI image"
+run: ./scripts/ci/ci_prepare_image_on_ci.sh
+  - name: "Pylint checks for tests"
+run: |
+  python -m pip install pre-commit \
+  --constraint 
requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt
+  ./scripts/ci/ci_run_static_checks.sh pylint-tests mypy
 
-  statics-tests:
-name: Pylint for tests
+  static-checks-2:
+name: "Checks: All other"
 runs-on: ubuntu-latest
 env:
-  PYTHON_VERSION: 3.6
-  AIRFLOW_MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
-  TRAVIS_JOB_NAME: "Static"
+  SKIP: pylint-tests,mypy
+  MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
+  CI_JOB_TYPE: "Static checks"
 steps:
   - uses: actions/checkout@master
   - uses: actions/setup-python@v1
 with:
-  python-version: ${{ env.PYTHON_VERSION }}
-  - name: "Before install"
-run: ./scripts/ci/ci_before_install.sh
-  - name: "Static checks tests"
-if: success()
+  python-version: '3.x'
+  - name: Cache pre-commit env
+uses: actions/cache@v1
 env:
-  PYTHON_VERSION: 3.6
-run: ./scripts/ci/ci_run_static_checks_pylint_tests.sh
+  cache-name: cache-pre-commit
+with:
+  path: ~/.cache/pre-commit
+  key: ${{ env.cache-name }}-${{ github.job }}-${{ 
hashFiles('.pre-commit-config.yaml') }}
+  - name: "Free space"
+run: ./scripts/ci/ci_free_space_on_ci.sh
+  - name: "Build CI image"
+run: ./scripts/ci/ci_prepare_image_on_ci.sh
+  - name: "Static checks"
+run: |
+  python -m pip install pre-commit \
+  --constraint 
requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt
+  ./scripts/ci/ci_run_static_checks.sh
 
   docs:
 name: Build docs
 runs-on: ubuntu-latest
 env:
-  TRAVIS_JOB_NAME: "Build documentation"
-  PYTHON_VERSION: 3.6
+  CI_JOB_TYPE: "Documentation"
 steps:
   - uses: actions/checkout@master
-  - name: "Build documentation"
+  - name: "Build CI image ${{ matrix.python-version }}"
+run: ./scripts/ci/ci_prepare_image_on_ci.sh
+  - name: "Build docs"
 run: ./scripts/ci/ci_docs.sh
 
-  tests-p36-postgres-integrations:
-name: "Tests [Postgres9.6][Py3.

[GitHub] [airflow] danfrankj opened a new pull request #8435: Fix performance degradation when updating dagrun state

[GitBox]

danfrankj opened a new pull request #8435: Fix performance degradation when 
updating dagrun state
URL: https://github.com/apache/airflow/pull/8435
 
 
   The set comprehension `{t.task_id for t in dag.leaves}` in the conditional 
is re-evaluated every for each loop in the outer list comprehension resulting 
in an O(n^2) computation and significant slowdowns on large dags. 
   
   Introduced here: 
https://github.com/apache/airflow/commit/8f6ca5305d8f1ae068903972bc6ad8893693514c#diff-32aa8dbb910719ef24a39cab5d0f2a97R302
 
   
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x ] Description above provides context of the change
   - [not sure - please advise ] Unit tests coverage for changes (not needed 
for documentation changes)
   - [x ] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [not sure - please advise ] Relevant documentation is updated including 
usage instructions.
   - [ ] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #8393: Bring back CI optimisations

[GitBox]

potiuk commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r410724983
 
 

 ##
 File path: .github/workflows/ci.yml
 ##
 @@ -17,272 +17,275 @@
 #
 
 ---
-name: Airflow
+name: CI Build
 on:
+  schedule:
+- cron: '10 2 * * *'
   push:
-branches: ['*']
+branches: ['master', 'v1-10-test', 'v1-10-stable']
   pull_request:
-branches: [master]
+branches: ['master', 'v1-10-test', 'v1-10-stable']
 env:
-  BUILD_ID: ${{github.sha }}
   MOUNT_LOCAL_SOURCES: "false"
-  MOUNT_HOST_AIRFLOW_VOLUME: "true"
   FORCE_ANSWER_TO_QUESTIONS: "yes"
   SKIP_CHECK_REMOTE_IMAGE: "true"
+  SKIP_CI_IMAGE_CHECK: "true"
   DB_RESET: "true"
   VERBOSE: "true"
-  CI: "true"
-  # Should be a target branch
-  TRAVIS_BRANCH: "master"
-  TRAVIS: "true"
+  UPGRADE_TO_LATEST_REQUIREMENTS: "false"
+  ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap presto rabbitmq 
redis"
+  PYTHON_MAJOR_MINOR_VERSION: 3.6
+  # TODO: Enable cache registry once a master build succeeds uploading the 
images to Github registry cache
+  DISABLE_CACHE_REGISTRY: "true"
+  CACHE_REGISTRY: "docker.pkg.github.com"
+  CACHE_IMAGE_PREFIX: ${{ github.repository }}
+  CACHE_REGISTRY_USERNAME: ${{ github.actor }}
+  CACHE_REGISTRY_PASSWORD: ${{ secrets.GITHUB_TOKEN }}
+
 jobs:
-  statics:
-name: Static checks
+
+  static-checks-1:
+name: "Checks: pylint, mypy"
 runs-on: ubuntu-latest
 env:
-  PYTHON_VERSION: 3.6
-  AIRFLOW_MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
-  TRAVIS_JOB_NAME: "Static"
+  MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
+  CI_JOB_TYPE: "Pylint checks for tests"
 steps:
   - uses: actions/checkout@master
-  - name: free disk space
-run: |
-  sudo swapoff -a
-  sudo rm -f /swapfile
-  sudo apt clean
-  docker rmi $(docker image ls -aq)
-  df -h
   - uses: actions/setup-python@v1
 with:
-  python-version: ${{ env.PYTHON_VERSION }}
-  - name: "Before install"
-run: ./scripts/ci/ci_before_install.sh
-  - name: "Static checks"
-if: success()
+  python-version: '3.x'
+  - name: Cache pre-commit env
+uses: actions/cache@v1
 env:
-  PYTHON_VERSION: 3.6
-run: ./scripts/ci/ci_run_all_static_checks.sh
+  cache-name: cache-pre-commit
+with:
+  path: ~/.cache/pre-commit
+  key: ${{ env.cache-name }}-${{ github.job }}-${{ 
hashFiles('.pre-commit-config.yaml') }}
+  - name: "Free space"
+run: ./scripts/ci/ci_free_space_on_ci.sh
+  - name: "Build CI image"
+run: ./scripts/ci/ci_prepare_image_on_ci.sh
+  - name: "Pylint checks for tests"
+run: |
+  python -m pip install pre-commit \
+  --constraint 
requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt
+  ./scripts/ci/ci_run_static_checks.sh pylint-tests mypy
 
-  statics-tests:
-name: Pylint for tests
+  static-checks-2:
+name: "Checks: All other"
 runs-on: ubuntu-latest
 env:
-  PYTHON_VERSION: 3.6
-  AIRFLOW_MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
-  TRAVIS_JOB_NAME: "Static"
+  SKIP: pylint-tests,mypy
+  MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
+  CI_JOB_TYPE: "Static checks"
 steps:
   - uses: actions/checkout@master
   - uses: actions/setup-python@v1
 with:
-  python-version: ${{ env.PYTHON_VERSION }}
-  - name: "Before install"
-run: ./scripts/ci/ci_before_install.sh
-  - name: "Static checks tests"
-if: success()
+  python-version: '3.x'
+  - name: Cache pre-commit env
+uses: actions/cache@v1
 env:
-  PYTHON_VERSION: 3.6
-run: ./scripts/ci/ci_run_static_checks_pylint_tests.sh
+  cache-name: cache-pre-commit
+with:
+  path: ~/.cache/pre-commit
+  key: ${{ env.cache-name }}-${{ github.job }}-${{ 
hashFiles('.pre-commit-config.yaml') }}
+  - name: "Free space"
+run: ./scripts/ci/ci_free_space_on_ci.sh
+  - name: "Build CI image"
+run: ./scripts/ci/ci_prepare_image_on_ci.sh
+  - name: "Static checks"
+run: |
+  python -m pip install pre-commit \
+  --constraint 
requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt
+  ./scripts/ci/ci_run_static_checks.sh
 
   docs:
 name: Build docs
 runs-on: ubuntu-latest
 env:
-  TRAVIS_JOB_NAME: "Build documentation"
-  PYTHON_VERSION: 3.6
+  CI_JOB_TYPE: "Documentation"
 steps:
   - uses: actions/checkout@master
-  - name: "Build documentation"
+  - name: "Build CI image ${{ matrix.python-version }}"
+run: ./scripts/ci/ci_prepare_image_on_ci.sh
+  - name: "Build docs"
 run: ./scripts/ci/ci_docs.sh
 
-  tests-p36-postgres-integrations:
-name: "Tests [Postgres9.6][Py3.

[GitHub] [airflow] potiuk commented on a change in pull request #8393: Bring back CI optimisations

[GitBox]

potiuk commented on a change in pull request #8393: Bring back CI optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r410724908
 
 

 ##
 File path: .github/workflows/ci.yml
 ##
 @@ -17,272 +17,275 @@
 #
 
 ---
-name: Airflow
+name: CI Build
 on:
+  schedule:
+- cron: '10 2 * * *'
   push:
-branches: ['*']
+branches: ['master', 'v1-10-test', 'v1-10-stable']
   pull_request:
-branches: [master]
+branches: ['master', 'v1-10-test', 'v1-10-stable']
 env:
-  BUILD_ID: ${{github.sha }}
   MOUNT_LOCAL_SOURCES: "false"
-  MOUNT_HOST_AIRFLOW_VOLUME: "true"
   FORCE_ANSWER_TO_QUESTIONS: "yes"
   SKIP_CHECK_REMOTE_IMAGE: "true"
+  SKIP_CI_IMAGE_CHECK: "true"
   DB_RESET: "true"
   VERBOSE: "true"
-  CI: "true"
-  # Should be a target branch
-  TRAVIS_BRANCH: "master"
-  TRAVIS: "true"
+  UPGRADE_TO_LATEST_REQUIREMENTS: "false"
+  ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap presto rabbitmq 
redis"
+  PYTHON_MAJOR_MINOR_VERSION: 3.6
+  # TODO: Enable cache registry once a master build succeeds uploading the 
images to Github registry cache
+  DISABLE_CACHE_REGISTRY: "true"
+  CACHE_REGISTRY: "docker.pkg.github.com"
+  CACHE_IMAGE_PREFIX: ${{ github.repository }}
+  CACHE_REGISTRY_USERNAME: ${{ github.actor }}
+  CACHE_REGISTRY_PASSWORD: ${{ secrets.GITHUB_TOKEN }}
+
 jobs:
-  statics:
-name: Static checks
+
+  static-checks-1:
+name: "Checks: pylint, mypy"
 runs-on: ubuntu-latest
 env:
-  PYTHON_VERSION: 3.6
-  AIRFLOW_MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
-  TRAVIS_JOB_NAME: "Static"
+  MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
+  CI_JOB_TYPE: "Pylint checks for tests"
 steps:
   - uses: actions/checkout@master
-  - name: free disk space
-run: |
-  sudo swapoff -a
-  sudo rm -f /swapfile
-  sudo apt clean
-  docker rmi $(docker image ls -aq)
-  df -h
   - uses: actions/setup-python@v1
 with:
-  python-version: ${{ env.PYTHON_VERSION }}
-  - name: "Before install"
-run: ./scripts/ci/ci_before_install.sh
-  - name: "Static checks"
-if: success()
+  python-version: '3.x'
+  - name: Cache pre-commit env
+uses: actions/cache@v1
 env:
-  PYTHON_VERSION: 3.6
-run: ./scripts/ci/ci_run_all_static_checks.sh
+  cache-name: cache-pre-commit
+with:
+  path: ~/.cache/pre-commit
 
 Review comment:
   Yep. it speeds up (by about a minute) pre-commit initialization. When you 
run pre-commit first time it builds local virtualenvs for all the pre-commits 
requiring it. It takes about a minute and the .venvs are stored in 
~/.cache/pre-commit . This way we save about a minute for static checks.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow-site] zhongjiajie commented on issue #266: Improve documentation on DockerOperator

[GitBox]

zhongjiajie commented on issue #266: Improve documentation on DockerOperator
URL: https://github.com/apache/airflow-site/pull/266#issuecomment-615897688
 
 
   Hi, @dedunumax the docs-archive was generate from 
https://github.com/apache/airflow/tree/master/docs, so I think we should not 
directly change it, we should change source in 
https://github.com/apache/airflow/tree/master/docs. Your change will be release 
in feature, maybe next release.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] turbaszek commented on a change in pull request #8393: Bring back CI optimisations

[GitBox]

turbaszek commented on a change in pull request #8393: Bring back CI 
optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r410708187
 
 

 ##
 File path: .github/workflows/ci.yml
 ##
 @@ -17,272 +17,275 @@
 #
 
 ---
-name: Airflow
+name: CI Build
 on:
+  schedule:
+- cron: '10 2 * * *'
   push:
-branches: ['*']
+branches: ['master', 'v1-10-test', 'v1-10-stable']
   pull_request:
-branches: [master]
+branches: ['master', 'v1-10-test', 'v1-10-stable']
 env:
-  BUILD_ID: ${{github.sha }}
   MOUNT_LOCAL_SOURCES: "false"
-  MOUNT_HOST_AIRFLOW_VOLUME: "true"
   FORCE_ANSWER_TO_QUESTIONS: "yes"
   SKIP_CHECK_REMOTE_IMAGE: "true"
+  SKIP_CI_IMAGE_CHECK: "true"
   DB_RESET: "true"
   VERBOSE: "true"
-  CI: "true"
-  # Should be a target branch
-  TRAVIS_BRANCH: "master"
-  TRAVIS: "true"
+  UPGRADE_TO_LATEST_REQUIREMENTS: "false"
+  ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap presto rabbitmq 
redis"
+  PYTHON_MAJOR_MINOR_VERSION: 3.6
+  # TODO: Enable cache registry once a master build succeeds uploading the 
images to Github registry cache
+  DISABLE_CACHE_REGISTRY: "true"
+  CACHE_REGISTRY: "docker.pkg.github.com"
+  CACHE_IMAGE_PREFIX: ${{ github.repository }}
+  CACHE_REGISTRY_USERNAME: ${{ github.actor }}
+  CACHE_REGISTRY_PASSWORD: ${{ secrets.GITHUB_TOKEN }}
+
 jobs:
-  statics:
-name: Static checks
+
+  static-checks-1:
+name: "Checks: pylint, mypy"
 runs-on: ubuntu-latest
 env:
-  PYTHON_VERSION: 3.6
-  AIRFLOW_MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
-  TRAVIS_JOB_NAME: "Static"
+  MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
+  CI_JOB_TYPE: "Pylint checks for tests"
 steps:
   - uses: actions/checkout@master
-  - name: free disk space
-run: |
-  sudo swapoff -a
-  sudo rm -f /swapfile
-  sudo apt clean
-  docker rmi $(docker image ls -aq)
-  df -h
   - uses: actions/setup-python@v1
 with:
-  python-version: ${{ env.PYTHON_VERSION }}
-  - name: "Before install"
-run: ./scripts/ci/ci_before_install.sh
-  - name: "Static checks"
-if: success()
+  python-version: '3.x'
+  - name: Cache pre-commit env
+uses: actions/cache@v1
 env:
-  PYTHON_VERSION: 3.6
-run: ./scripts/ci/ci_run_all_static_checks.sh
+  cache-name: cache-pre-commit
+with:
+  path: ~/.cache/pre-commit
+  key: ${{ env.cache-name }}-${{ github.job }}-${{ 
hashFiles('.pre-commit-config.yaml') }}
+  - name: "Free space"
+run: ./scripts/ci/ci_free_space_on_ci.sh
+  - name: "Build CI image"
+run: ./scripts/ci/ci_prepare_image_on_ci.sh
+  - name: "Pylint checks for tests"
+run: |
+  python -m pip install pre-commit \
+  --constraint 
requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt
+  ./scripts/ci/ci_run_static_checks.sh pylint-tests mypy
 
-  statics-tests:
-name: Pylint for tests
+  static-checks-2:
+name: "Checks: All other"
 runs-on: ubuntu-latest
 env:
-  PYTHON_VERSION: 3.6
-  AIRFLOW_MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
-  TRAVIS_JOB_NAME: "Static"
+  SKIP: pylint-tests,mypy
+  MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
+  CI_JOB_TYPE: "Static checks"
 steps:
   - uses: actions/checkout@master
   - uses: actions/setup-python@v1
 with:
-  python-version: ${{ env.PYTHON_VERSION }}
-  - name: "Before install"
-run: ./scripts/ci/ci_before_install.sh
-  - name: "Static checks tests"
-if: success()
+  python-version: '3.x'
+  - name: Cache pre-commit env
+uses: actions/cache@v1
 env:
-  PYTHON_VERSION: 3.6
-run: ./scripts/ci/ci_run_static_checks_pylint_tests.sh
+  cache-name: cache-pre-commit
+with:
+  path: ~/.cache/pre-commit
+  key: ${{ env.cache-name }}-${{ github.job }}-${{ 
hashFiles('.pre-commit-config.yaml') }}
+  - name: "Free space"
+run: ./scripts/ci/ci_free_space_on_ci.sh
+  - name: "Build CI image"
+run: ./scripts/ci/ci_prepare_image_on_ci.sh
+  - name: "Static checks"
+run: |
+  python -m pip install pre-commit \
+  --constraint 
requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt
+  ./scripts/ci/ci_run_static_checks.sh
 
   docs:
 name: Build docs
 runs-on: ubuntu-latest
 env:
-  TRAVIS_JOB_NAME: "Build documentation"
-  PYTHON_VERSION: 3.6
+  CI_JOB_TYPE: "Documentation"
 steps:
   - uses: actions/checkout@master
-  - name: "Build documentation"
+  - name: "Build CI image ${{ matrix.python-version }}"
+run: ./scripts/ci/ci_prepare_image_on_ci.sh
+  - name: "Build docs"
 run: ./scripts/ci/ci_docs.sh
 
-  tests-p36-postgres-integrations:
-name: "Tests [Postgres9.6][

[GitHub] [airflow] turbaszek commented on a change in pull request #8393: Bring back CI optimisations

[GitBox]

turbaszek commented on a change in pull request #8393: Bring back CI 
optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r410708095
 
 

 ##
 File path: .github/workflows/ci.yml
 ##
 @@ -17,272 +17,275 @@
 #
 
 ---
-name: Airflow
+name: CI Build
 on:
+  schedule:
+- cron: '10 2 * * *'
   push:
-branches: ['*']
+branches: ['master', 'v1-10-test', 'v1-10-stable']
   pull_request:
-branches: [master]
+branches: ['master', 'v1-10-test', 'v1-10-stable']
 env:
-  BUILD_ID: ${{github.sha }}
   MOUNT_LOCAL_SOURCES: "false"
-  MOUNT_HOST_AIRFLOW_VOLUME: "true"
   FORCE_ANSWER_TO_QUESTIONS: "yes"
   SKIP_CHECK_REMOTE_IMAGE: "true"
+  SKIP_CI_IMAGE_CHECK: "true"
   DB_RESET: "true"
   VERBOSE: "true"
-  CI: "true"
-  # Should be a target branch
-  TRAVIS_BRANCH: "master"
-  TRAVIS: "true"
+  UPGRADE_TO_LATEST_REQUIREMENTS: "false"
+  ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap presto rabbitmq 
redis"
+  PYTHON_MAJOR_MINOR_VERSION: 3.6
+  # TODO: Enable cache registry once a master build succeeds uploading the 
images to Github registry cache
+  DISABLE_CACHE_REGISTRY: "true"
+  CACHE_REGISTRY: "docker.pkg.github.com"
+  CACHE_IMAGE_PREFIX: ${{ github.repository }}
+  CACHE_REGISTRY_USERNAME: ${{ github.actor }}
+  CACHE_REGISTRY_PASSWORD: ${{ secrets.GITHUB_TOKEN }}
+
 jobs:
-  statics:
-name: Static checks
+
+  static-checks-1:
+name: "Checks: pylint, mypy"
 runs-on: ubuntu-latest
 env:
-  PYTHON_VERSION: 3.6
-  AIRFLOW_MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
-  TRAVIS_JOB_NAME: "Static"
+  MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
+  CI_JOB_TYPE: "Pylint checks for tests"
 steps:
   - uses: actions/checkout@master
-  - name: free disk space
-run: |
-  sudo swapoff -a
-  sudo rm -f /swapfile
-  sudo apt clean
-  docker rmi $(docker image ls -aq)
-  df -h
   - uses: actions/setup-python@v1
 with:
-  python-version: ${{ env.PYTHON_VERSION }}
-  - name: "Before install"
-run: ./scripts/ci/ci_before_install.sh
-  - name: "Static checks"
-if: success()
+  python-version: '3.x'
+  - name: Cache pre-commit env
+uses: actions/cache@v1
 env:
-  PYTHON_VERSION: 3.6
-run: ./scripts/ci/ci_run_all_static_checks.sh
+  cache-name: cache-pre-commit
+with:
+  path: ~/.cache/pre-commit
 
 Review comment:
   Can you explain the purpose of this cache?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] turbaszek commented on a change in pull request #8393: Bring back CI optimisations

[GitBox]

turbaszek commented on a change in pull request #8393: Bring back CI 
optimisations
URL: https://github.com/apache/airflow/pull/8393#discussion_r410708271
 
 

 ##
 File path: .github/workflows/ci.yml
 ##
 @@ -17,272 +17,275 @@
 #
 
 ---
-name: Airflow
+name: CI Build
 on:
+  schedule:
+- cron: '10 2 * * *'
   push:
-branches: ['*']
+branches: ['master', 'v1-10-test', 'v1-10-stable']
   pull_request:
-branches: [master]
+branches: ['master', 'v1-10-test', 'v1-10-stable']
 env:
-  BUILD_ID: ${{github.sha }}
   MOUNT_LOCAL_SOURCES: "false"
-  MOUNT_HOST_AIRFLOW_VOLUME: "true"
   FORCE_ANSWER_TO_QUESTIONS: "yes"
   SKIP_CHECK_REMOTE_IMAGE: "true"
+  SKIP_CI_IMAGE_CHECK: "true"
   DB_RESET: "true"
   VERBOSE: "true"
-  CI: "true"
-  # Should be a target branch
-  TRAVIS_BRANCH: "master"
-  TRAVIS: "true"
+  UPGRADE_TO_LATEST_REQUIREMENTS: "false"
+  ENABLED_INTEGRATIONS: "cassandra kerberos mongo openldap presto rabbitmq 
redis"
+  PYTHON_MAJOR_MINOR_VERSION: 3.6
+  # TODO: Enable cache registry once a master build succeeds uploading the 
images to Github registry cache
+  DISABLE_CACHE_REGISTRY: "true"
+  CACHE_REGISTRY: "docker.pkg.github.com"
+  CACHE_IMAGE_PREFIX: ${{ github.repository }}
+  CACHE_REGISTRY_USERNAME: ${{ github.actor }}
+  CACHE_REGISTRY_PASSWORD: ${{ secrets.GITHUB_TOKEN }}
+
 jobs:
-  statics:
-name: Static checks
+
+  static-checks-1:
+name: "Checks: pylint, mypy"
 runs-on: ubuntu-latest
 env:
-  PYTHON_VERSION: 3.6
-  AIRFLOW_MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
-  TRAVIS_JOB_NAME: "Static"
+  MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
+  CI_JOB_TYPE: "Pylint checks for tests"
 steps:
   - uses: actions/checkout@master
-  - name: free disk space
-run: |
-  sudo swapoff -a
-  sudo rm -f /swapfile
-  sudo apt clean
-  docker rmi $(docker image ls -aq)
-  df -h
   - uses: actions/setup-python@v1
 with:
-  python-version: ${{ env.PYTHON_VERSION }}
-  - name: "Before install"
-run: ./scripts/ci/ci_before_install.sh
-  - name: "Static checks"
-if: success()
+  python-version: '3.x'
+  - name: Cache pre-commit env
+uses: actions/cache@v1
 env:
-  PYTHON_VERSION: 3.6
-run: ./scripts/ci/ci_run_all_static_checks.sh
+  cache-name: cache-pre-commit
+with:
+  path: ~/.cache/pre-commit
+  key: ${{ env.cache-name }}-${{ github.job }}-${{ 
hashFiles('.pre-commit-config.yaml') }}
+  - name: "Free space"
+run: ./scripts/ci/ci_free_space_on_ci.sh
+  - name: "Build CI image"
+run: ./scripts/ci/ci_prepare_image_on_ci.sh
+  - name: "Pylint checks for tests"
+run: |
+  python -m pip install pre-commit \
+  --constraint 
requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt
+  ./scripts/ci/ci_run_static_checks.sh pylint-tests mypy
 
-  statics-tests:
-name: Pylint for tests
+  static-checks-2:
+name: "Checks: All other"
 runs-on: ubuntu-latest
 env:
-  PYTHON_VERSION: 3.6
-  AIRFLOW_MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
-  TRAVIS_JOB_NAME: "Static"
+  SKIP: pylint-tests,mypy
+  MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
+  CI_JOB_TYPE: "Static checks"
 steps:
   - uses: actions/checkout@master
   - uses: actions/setup-python@v1
 with:
-  python-version: ${{ env.PYTHON_VERSION }}
-  - name: "Before install"
-run: ./scripts/ci/ci_before_install.sh
-  - name: "Static checks tests"
-if: success()
+  python-version: '3.x'
+  - name: Cache pre-commit env
+uses: actions/cache@v1
 env:
-  PYTHON_VERSION: 3.6
-run: ./scripts/ci/ci_run_static_checks_pylint_tests.sh
+  cache-name: cache-pre-commit
+with:
+  path: ~/.cache/pre-commit
+  key: ${{ env.cache-name }}-${{ github.job }}-${{ 
hashFiles('.pre-commit-config.yaml') }}
+  - name: "Free space"
+run: ./scripts/ci/ci_free_space_on_ci.sh
+  - name: "Build CI image"
+run: ./scripts/ci/ci_prepare_image_on_ci.sh
+  - name: "Static checks"
+run: |
+  python -m pip install pre-commit \
+  --constraint 
requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt
+  ./scripts/ci/ci_run_static_checks.sh
 
   docs:
 name: Build docs
 runs-on: ubuntu-latest
 env:
-  TRAVIS_JOB_NAME: "Build documentation"
-  PYTHON_VERSION: 3.6
+  CI_JOB_TYPE: "Documentation"
 steps:
   - uses: actions/checkout@master
-  - name: "Build documentation"
+  - name: "Build CI image ${{ matrix.python-version }}"
+run: ./scripts/ci/ci_prepare_image_on_ci.sh
+  - name: "Build docs"
 run: ./scripts/ci/ci_docs.sh
 
-  tests-p36-postgres-integrations:
-name: "Tests [Postgres9.6][

[GitHub] [airflow] kaxil commented on issue #8432: Provide GCP credentials in Bash/Python operators

[GitBox]

kaxil commented on issue #8432: Provide GCP credentials in Bash/Python operators
URL: https://github.com/apache/airflow/pull/8432#issuecomment-615884586
 
 
   Yeah I am fine if we can create something "generic"


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] turbaszek commented on issue #8432: Provide GCP credentials in Bash/Python operators

[GitBox]

turbaszek commented on issue #8432: Provide GCP credentials in Bash/Python 
operators
URL: https://github.com/apache/airflow/pull/8432#issuecomment-615884076
 
 
   I agree that having provider-specific code in Bash / Python ops doesn't 
sound good. However, I think that it would be nice to help users somehow to 
authorize in those ops.
   
   Other idea I have is to pass an authorization context manager to operators:
   ```python
   class BashOperator:
   def __init__(..., authctx=None):
   self.authctx = authctx
   
   def pre_execute(self):
   self.authctx.enter()
   
def post_execute(self):
   self.authctx.close()
   ```
   and then users can do something like this:
   ```
   bop = BashOperator(task_id="id", ..., authctx=gcp_auth(KEY_NAME))
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] stale[bot] closed pull request #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result

[GitBox]

stale[bot] closed pull request #7508: [AIRFLOW-6886] Cleanup the 
SageMakerTrainingOperator result
URL: https://github.com/apache/airflow/pull/7508
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6886) Cleanup the SageMakerTrainingOperator result

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086452#comment-17086452
 ] 

ASF GitHub Bot commented on AIRFLOW-6886:
-

stale[bot] commented on pull request #7508: [AIRFLOW-6886] Cleanup the 
SageMakerTrainingOperator result
URL: https://github.com/apache/airflow/pull/7508
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Cleanup the SageMakerTrainingOperator result
> 
>
> Key: AIRFLOW-6886
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6886
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: aws
>Affects Versions: 2.0.0
>Reporter: Bas Harenslak
>Assignee: Bas Harenslak
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] kaxil edited a comment on issue #8432: Provide GCP credentials in Bash/Python operators

[GitBox]

kaxil edited a comment on issue #8432: Provide GCP credentials in Bash/Python 
operators
URL: https://github.com/apache/airflow/pull/8432#issuecomment-615865246
 
 
   I agree with Jarek here, any cloud specific code in Core Operators would 
hurt my eyes.
   
   Each cloud provider also provides a way to authenticate using an Environment 
Variable too. In case of GCP it is GOOGLE_APPLICATION_CREDENTIALS which can be 
used for Bash and Python Operators.
   
   And if the Virtual Machine is on that Cloud Provider itself you don't need 
to authenticate as you can VM's credentials too.
   
   We already allow injecting env vars to BashOperator.
   
   Users have enough flexibility in Bash and Python, is my main point. I would 
let them take care of this, rather than having us to maintain this code.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil edited a comment on issue #8432: Provide GCP credentials in Bash/Python operators

[GitBox]

kaxil edited a comment on issue #8432: Provide GCP credentials in Bash/Python 
operators
URL: https://github.com/apache/airflow/pull/8432#issuecomment-615865246
 
 
   I agree with Jarek here, any cloud specific code in Core Operators would 
hurt my eyes.
   
   Each cloud provider also provides a way to authenticate using an Environment 
Variable too. In case of GCP it is GOOGLE_APPLICATION_CREDENTIALS which can be 
used for Bash and Python Operators.
   
   And if the Virtual Machine is on that Cloud Provider itself you don't need 
to authenticate as you can VM's credentials too.
   
   We already allow injecting env vars to BashOperator.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] kaxil commented on issue #8432: Provide GCP credentials in Bash/Python operators

[GitBox]

kaxil commented on issue #8432: Provide GCP credentials in Bash/Python operators
URL: https://github.com/apache/airflow/pull/8432#issuecomment-615865246
 
 
   I agree with Jarek here, any cloud specific code in Core Operators would 
hurt my eyes.
   
   Each cloud provider also provides a way to authenticate using an Environment 
Variable too. In case of GCP it is GOOGLE_APPLICATION_CREDENTIALS which can be 
used for Bash and Python Operators.
   
   And if the Virtual Machine is on that Cloud Provider itself you don't need 
to authenticate as you can VM's credentials tooo


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] eladkal commented on issue #7858: ShortCircuitOperator Ignoring Trigger Rules for Downstream Task and Skipping all Tasks

[GitBox]

eladkal commented on issue #7858: ShortCircuitOperator Ignoring Trigger Rules 
for Downstream Task and Skipping all Tasks
URL: https://github.com/apache/airflow/issues/7858#issuecomment-615863490
 
 
   When the callable of `ShortCircuitOperator` is evaluated to `False` it 
short-circuits the entire downstream branch. This means that all downstream 
tasks are set to `SKIP` immediately. This is done by calling `skip` method of 
`SkipMixin`.
   The `ShortCircuitOperator` is more for the usage of "Stop here. don't 
continue no matter what."
   
   The docs seems to be written by the spirit of `BranchPythonOperator` where 
it uses `skip_all_except` method of `SkipMixin`. It's not quite the same 
because it checks for dependency between tasks rather than trigger rules but at 
it uses a less violent approach.
   
   I'm not sure if this this is a bug in the `ShortCircuitOperator`. I for once 
have workflows that need the functionality exactly as it is now so I think this 
is more case of wrong documentation issue rather than a bug.
   
   On the other hand maybe the `ShortCircuitOperator` can be enhanced to have 
two modes: `hard` & `soft`. Where `hard` is the same behavior as now while 
`soft` will check if SKIP should cascade further.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6609) Airflow upgradedb fails serialized_dag table add on revision id d38e04c12aa2

[Kaxil Naik (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086435#comment-17086435
 ] 

Kaxil Naik commented on AIRFLOW-6609:
-

That looks someone ran airflow initdb/upgradedb and stopped it midway

Go to Airflow Metadata DB, find the alembic table and update the value to 
d38e04c12aa2

> Airflow upgradedb fails serialized_dag table add on revision id d38e04c12aa2
> 
>
> Key: AIRFLOW-6609
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6609
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: database
>Affects Versions: 1.10.7
>Reporter: Chris Schmautz
>Priority: Major
>  Labels: database, postgres
>
> We're attempting an upgrade from 1.10.3 to 1.10.7 to use some of the great 
> features available in later revisions; however, the upgrade from 1.10.6 to 
> 1.10.7 is causing some heartburn.
> +Runtime environment:+
>  - Docker containers for each runtime segment (webserver, scheduler, flower, 
> postgres, redis, worker)
>  - Using CeleryExecutor queued with Redis
>  - Using Postgres backend
>  
> +Steps to reproduce:+
>  1. Author base images relating to each version of Airflow between 1.10.3 and 
> 1.10.7 (if you want the full regression we have done)
>  2. 'airflow initdb' on revision 1.10.3
>  3. Start up the containers, run some dags, produce metadata
>  4. Increment / swap out base image revision from 1.10.3 base to 1.10.4 base 
> image
>  5. Run 'airflow upgradedb'
>  6. Validate success
>  n. Eventually you will get to the 1.10.6 revision, stepping up to 1.10.7, 
> which produces the error below
>  
> {code:java}
> INFO  [alembic.runtime.migration] Running upgrade 6e96a59344a4 -> 
> d38e04c12aa2, add serialized_dag table
> Revision ID: d38e04c12aa2
> Revises: 6e96a59344a4
> Create Date: 2019-08-01 14:39:35.616417
> Traceback (most recent call last):
>   File 
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py",
>  line 1246, in _execute_context
> cursor, statement, parameters, context
>   File 
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/default.py",
>  line 581, in do_execute
> cursor.execute(statement, parameters)
> psycopg2.errors.DuplicateTable: relation "serialized_dag" already exists
> The above exception was the direct cause of the following exception:Traceback 
> (most recent call last):
>   File "/opt/anaconda/miniconda3/envs/airflow/bin/airflow", line 37, in 
> 
> args.func(args)
>   File 
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/airflow/utils/cli.py",
>  line 75, in wrapper
> return f(*args, **kwargs)
>   File 
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/airflow/bin/cli.py",
>  line 1193, in upgradedb
> db.upgradedb()
>   File 
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/airflow/utils/db.py",
>  line 376, in upgradedb
> command.upgrade(config, 'heads')
>   File 
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/alembic/command.py",
>  line 298, in upgrade
> script.run_env()
>   File 
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/alembic/script/base.py",
>  line 489, in run_env
> util.load_python_file(self.dir, "env.py")
>   File 
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/alembic/util/pyfiles.py",
>  line 98, in load_python_file
> module = load_module_py(module_id, path)
>   File 
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/alembic/util/compat.py",
>  line 173, in load_module_py
> spec.loader.exec_module(module)
>   File "", line 678, in exec_module
>   File "", line 219, in _call_with_frames_removed
>   File 
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/airflow/migrations/env.py",
>  line 96, in 
> run_migrations_online()
>   File 
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/airflow/migrations/env.py",
>  line 90, in run_migrations_online
> context.run_migrations()
>   File "", line 8, in run_migrations
>   File 
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/alembic/runtime/environment.py",
>  line 846, in run_migrations
> self.get_context().run_migrations(**kw)
>   File 
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/alembic/runtime/migration.py",
>  line 518, in run_migrations
> step.migration_fn(**kw)
>   File 
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/airflow/migrations/versions/d38e04c12aa2_add_serialized_dag_table.py",
>  line 54, in upgrade
> sa.PrimaryKeyConstraint('dag_id'))
>   File "", line 8, in create_table
>   File "", line 3, in create

[jira] [Comment Edited] (AIRFLOW-6609) Airflow upgradedb fails serialized_dag table add on revision id d38e04c12aa2

[Kaxil Naik (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086435#comment-17086435
 ] 

Kaxil Naik edited comment on AIRFLOW-6609 at 4/18/20, 12:20 PM:


That looks someone ran airflow initdb/upgradedb and stopped it midway.

Causing alembic to not update identifier in its table

Go to Airflow Metadata DB, find the alembic table and update the value to 
d38e04c12aa2


was (Author: kaxilnaik):
That looks someone ran airflow initdb/upgradedb and stopped it midway

Go to Airflow Metadata DB, find the alembic table and update the value to 
d38e04c12aa2

> Airflow upgradedb fails serialized_dag table add on revision id d38e04c12aa2
> 
>
> Key: AIRFLOW-6609
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6609
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: database
>Affects Versions: 1.10.7
>Reporter: Chris Schmautz
>Priority: Major
>  Labels: database, postgres
>
> We're attempting an upgrade from 1.10.3 to 1.10.7 to use some of the great 
> features available in later revisions; however, the upgrade from 1.10.6 to 
> 1.10.7 is causing some heartburn.
> +Runtime environment:+
>  - Docker containers for each runtime segment (webserver, scheduler, flower, 
> postgres, redis, worker)
>  - Using CeleryExecutor queued with Redis
>  - Using Postgres backend
>  
> +Steps to reproduce:+
>  1. Author base images relating to each version of Airflow between 1.10.3 and 
> 1.10.7 (if you want the full regression we have done)
>  2. 'airflow initdb' on revision 1.10.3
>  3. Start up the containers, run some dags, produce metadata
>  4. Increment / swap out base image revision from 1.10.3 base to 1.10.4 base 
> image
>  5. Run 'airflow upgradedb'
>  6. Validate success
>  n. Eventually you will get to the 1.10.6 revision, stepping up to 1.10.7, 
> which produces the error below
>  
> {code:java}
> INFO  [alembic.runtime.migration] Running upgrade 6e96a59344a4 -> 
> d38e04c12aa2, add serialized_dag table
> Revision ID: d38e04c12aa2
> Revises: 6e96a59344a4
> Create Date: 2019-08-01 14:39:35.616417
> Traceback (most recent call last):
>   File 
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py",
>  line 1246, in _execute_context
> cursor, statement, parameters, context
>   File 
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/default.py",
>  line 581, in do_execute
> cursor.execute(statement, parameters)
> psycopg2.errors.DuplicateTable: relation "serialized_dag" already exists
> The above exception was the direct cause of the following exception:Traceback 
> (most recent call last):
>   File "/opt/anaconda/miniconda3/envs/airflow/bin/airflow", line 37, in 
> 
> args.func(args)
>   File 
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/airflow/utils/cli.py",
>  line 75, in wrapper
> return f(*args, **kwargs)
>   File 
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/airflow/bin/cli.py",
>  line 1193, in upgradedb
> db.upgradedb()
>   File 
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/airflow/utils/db.py",
>  line 376, in upgradedb
> command.upgrade(config, 'heads')
>   File 
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/alembic/command.py",
>  line 298, in upgrade
> script.run_env()
>   File 
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/alembic/script/base.py",
>  line 489, in run_env
> util.load_python_file(self.dir, "env.py")
>   File 
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/alembic/util/pyfiles.py",
>  line 98, in load_python_file
> module = load_module_py(module_id, path)
>   File 
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/alembic/util/compat.py",
>  line 173, in load_module_py
> spec.loader.exec_module(module)
>   File "", line 678, in exec_module
>   File "", line 219, in _call_with_frames_removed
>   File 
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/airflow/migrations/env.py",
>  line 96, in 
> run_migrations_online()
>   File 
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/airflow/migrations/env.py",
>  line 90, in run_migrations_online
> context.run_migrations()
>   File "", line 8, in run_migrations
>   File 
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/alembic/runtime/environment.py",
>  line 846, in run_migrations
> self.get_context().run_migrations(**kw)
>   File 
> "/opt/anaconda/miniconda3/envs/airflow/lib/python3.6/site-packages/alembic/runtime/migration.py",
>  line 518, in run_migrations
> step.migra

[jira] [Commented] (AIRFLOW-5156) Add other authentication mechanisms to HttpHook

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086415#comment-17086415
 ] 

ASF GitHub Bot commented on AIRFLOW-5156:
-

randr97 commented on pull request #8434: [AIRFLOW-5156] Fixed doc strigns for 
HttpHook
URL: https://github.com/apache/airflow/pull/8434
 
 
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add other authentication mechanisms to HttpHook
> ---
>
> Key: AIRFLOW-5156
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5156
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: hooks
>Affects Versions: 1.10.4
>Reporter: Joshua Kornblum
>Assignee: Rohit S S
>Priority: Minor
>
> It looks like the only supported authentication for HttpHooks is basic auth.
> The hook code shows 
> {quote}_if conn.login:_
>   _session.auth = (conn.login, conn.password)_
> {quote}
> requests library supports any auth that inherits AuthBase – in my scenario we 
> need ntlmauth for API on IIS server. 
> [https://2.python-requests.org/en/master/user/advanced/#custom-authentication]
> I would suggest option to pass auth object in constructor then add to if/else 
> control flow like
> {quote}_if self.auth is not None:_
>   _session.auth = self.auth_
> _elif conn.login:_
>   _session.auth = (conn.login, conn.password)_
> {quote}
> One would have to fetch the connection themselves and then fill out auth and 
> then pass that to hook which is flexible although a little awkard.
> {quote}api_conn = BaseHook().get_connection('my_api')
> auth = HttpNtlmAuth(api_conn.login, api_conn.password)
> HttpSensor(task_id='sensing', auth=auth, )
> {quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] randr97 opened a new pull request #8434: [AIRFLOW-5156] Fixed doc strigns for HttpHook

[GitBox]

randr97 opened a new pull request #8434: [AIRFLOW-5156] Fixed doc strigns for 
HttpHook
URL: https://github.com/apache/airflow/pull/8434
 
 
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on issue #7735: [AIRFLOW-4549] Allow skipped tasks to satisfy wait_for_downstream

[GitBox]

ashb commented on issue #7735: [AIRFLOW-4549] Allow skipped tasks to satisfy 
wait_for_downstream
URL: https://github.com/apache/airflow/pull/7735#issuecomment-615841790
 
 
   Code looks fine, just need other people to weigh in on if this makes sense 
as a change - I just don't have head space to work that out right now, sorry


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] boring-cyborg[bot] commented on issue #8433: Support slackclient v2 with 2.0

[GitBox]

boring-cyborg[bot] commented on issue #8433: Support slackclient v2 with 2.0
URL: https://github.com/apache/airflow/issues/8433#issuecomment-615841556
 
 
   Thanks for opening your first issue here! Be sure to follow the issue 
template!
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] michaelosthege opened a new issue #8433: Support slackclient v2 with 2.0

[GitBox]

michaelosthege opened a new issue #8433: Support slackclient v2 with 2.0
URL: https://github.com/apache/airflow/issues/8433
 
 
   **Description**
   Slack migrated their Python client to v2. In v1 they ended support for 
Python 2.7 with its EOL.
   
   Airflow 2.0 will also end support for Python 2.7, so there's no need for the 
outdated dependency to stick around.
   
   **Use case / motivation**
   The [Python Slack SDK documentation](https://slack.dev/python-slackclient/) 
for v1 of the SDK is nowhere to be found, taking all the fun out of using it.
   
   Furthermore, [Slack does not add new features to the v1 
SDK](https://github.com/slackapi/python-slackclient/wiki/Migrating-to-2.x#minimum-python-versions).
   
   **Related Issues / PR**
   PR https://github.com/apache/airflow/pull/5519 already implemented the 
change for the Slack operators, but was not merged because Python 2 was still 
supported at the time.
   
   The corresponding 
[AIRFLOW-4543](https://issues.apache.org/jira/browse/AIRFLOW-4543) ticket is 
closed and cannot be reopened.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ansh08lehri commented on issue #8207: Create guide for Machine Learning Engine operators

[GitBox]

ansh08lehri commented on issue #8207: Create guide for Machine Learning Engine 
operators
URL: https://github.com/apache/airflow/issues/8207#issuecomment-615840911
 
 
   Hello, I am not able to understand what you are asking for. If you can just 
explain it more, it will be very useful.
   Best regards,
   Ansh


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services