[GitHub] [airflow] YiiTing commented on issue #14352: Airflow2.0 cannot read remote log from GCP GCS

[GitBox]

YiiTing commented on issue #14352:
URL: https://github.com/apache/airflow/issues/14352#issuecomment-831062244


   Hi all,
   i have some problems
   
   here is my UI log
   `*** Reading remote log from 
gs://airflow-log/inspect_apii/main.inspect_p/2021-05-03T06:40:00+00:00/1.log.
   b'*** Previous log discarded: 404 GET 
https://storage.googleapis.com/download/storage/v1/b/airflow-log/o/inspect_apii%2Fmain.inspect_p%2F2021-05-03T06%3A40%3A00%2B00%3A00%2F1.log?alt=media:
 No such obj`
   
   but i can find my log in gcs 
   `*** Previous log discarded: 404 GET 
https://storage.googleapis.com/download/storage/v1/b/airflow-log/o/inspect_apii%2Fmain.inspect_p%2F2021-05-03T06%3A40%3A00%2B00%3A00%2F1.log?alt=media:
 No such object: 
airflow-log/inspect_apii/main.inspect_p/2021-05-03T06:40:00+00:00/1.log: 
('Request failed with status code', 404, 'Expected one of', , )
   `
   
   what's going on?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow-ci-infra] xinbinhuang commented on a change in pull request #20: Sync committers daily

[GitBox]

xinbinhuang commented on a change in pull request #20:
URL: https://github.com/apache/airflow-ci-infra/pull/20#discussion_r624894663



##
File path: lambdas/scale_out_runner/app.py
##
@@ -101,12 +107,38 @@ def index():
 return payload
 
 
-def commiters(ssm_repo_name: str = os.getenv('SSM_REPO_NAME', 
'apache/airflow')):
+@app.schedule(Rate(1, unit=Rate.DAYS))
+def sync_committers(_):
+app.log.info("Parsing committers list from %s", GITHUB_CI_YML)
+resp = requests.get(GITHUB_CI_YML)
+ci_yml = yaml.load(resp.text, yaml.SafeLoader)
+runs_on = ci_yml['jobs']['build-info']['runs-on'].replace("\n", '')
+committers = re.findall(r"fromJSON\('(\[.*\])'\)", runs_on)[0]
+
+client = boto3.client('ssm')
+param_path = get_ssm_param_path(SSM_REPO_NAME)
+
+app.log.info("Updating config overlay for %s", param_path)
+ssm_value = {"pullRequestSecurity": {"allowedAuthors": committers}}

Review comment:
   I try to guess this part based on the func `commiters()`. Will refactor 
this with `committers()` if you can confirm this's the right SSM value. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow-ci-infra] xinbinhuang commented on a change in pull request #20: Sync committers daily

[GitBox]

xinbinhuang commented on a change in pull request #20:
URL: https://github.com/apache/airflow-ci-infra/pull/20#discussion_r624894663



##
File path: lambdas/scale_out_runner/app.py
##
@@ -101,12 +107,38 @@ def index():
 return payload
 
 
-def commiters(ssm_repo_name: str = os.getenv('SSM_REPO_NAME', 
'apache/airflow')):
+@app.schedule(Rate(1, unit=Rate.DAYS))
+def sync_committers(_):
+app.log.info("Parsing committers list from %s", GITHUB_CI_YML)
+resp = requests.get(GITHUB_CI_YML)
+ci_yml = yaml.load(resp.text, yaml.SafeLoader)
+runs_on = ci_yml['jobs']['build-info']['runs-on'].replace("\n", '')
+committers = re.findall(r"fromJSON\('(\[.*\])'\)", runs_on)[0]
+
+client = boto3.client('ssm')
+param_path = get_ssm_param_path(SSM_REPO_NAME)
+
+app.log.info("Updating config overlay for %s", param_path)
+ssm_value = {"pullRequestSecurity": {"allowedAuthors": committers}}

Review comment:
   I guess this part based on the func `commiters()`. Will refactor this 
with `committers()` if you can confirm this's the right SSM value. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow-ci-infra] branch sync-commiters updated (4544246 -> e9e21a5)

[binh]

This is an automated email from the ASF dual-hosted git repository.

binh pushed a change to branch sync-commiters
in repository https://gitbox.apache.org/repos/asf/airflow-ci-infra.git.


from 4544246  Sync committers daily
 add e9e21a5  fixup! Sync committers daily

No new revisions were added by this update.

Summary of changes:
 lambdas/scale_out_runner/app.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

[airflow-ci-infra] branch sync-commiters created (now 4544246)

[binh]

This is an automated email from the ASF dual-hosted git repository.

binh pushed a change to branch sync-commiters
in repository https://gitbox.apache.org/repos/asf/airflow-ci-infra.git.


  at 4544246  Sync committers daily

No new revisions were added by this update.

[GitHub] [airflow-ci-infra] xinbinhuang opened a new pull request #20: Sync committers daily

[GitBox]

xinbinhuang opened a new pull request #20:
URL: https://github.com/apache/airflow-ci-infra/pull/20


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #14521: Add Asana Provider

[GitBox]

github-actions[bot] commented on pull request #14521:
URL: https://github.com/apache/airflow/pull/14521#issuecomment-831033815


   [The Workflow run](https://github.com/apache/airflow/actions/runs/805696475) 
is cancelling this PR. Building images for the PR has failed. Follow the 
workflow link to check the reason.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated: Fix add quotes to extraConfigMaps/extraSecrets (#15633)

[binh]

This is an automated email from the ASF dual-hosted git repository.

binh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/master by this push:
 new 7318787  Fix add quotes to extraConfigMaps/extraSecrets (#15633)
7318787 is described below

commit 73187871703bce22783a42db3d3cec9045ee1de2
Author: YuDorogov <34017639+yudoro...@users.noreply.github.com>
AuthorDate: Mon May 3 12:04:09 2021 +0700

Fix add quotes to extraConfigMaps/extraSecrets (#15633)

* add quotes to extraConfigMaps

Add quotes to extraConfigMaps

* Update values.yaml

add quotes
---
 chart/values.yaml | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/chart/values.yaml b/chart/values.yaml
index 179da19..bc8cdd3 100644
--- a/chart/values.yaml
+++ b/chart/values.yaml
@@ -181,13 +181,13 @@ secret: []
 extraSecrets: {}
 # eg:
 # extraSecrets:
-#   {{ .Release.Name }}-airflow-connections:
+#   '{{ .Release.Name }}-airflow-connections':
 # data: |
 #   AIRFLOW_CONN_GCP: 'base64_encoded_gcp_conn_string'
 #   AIRFLOW_CONN_AWS: 'base64_encoded_aws_conn_string'
 # stringData: |
 #   AIRFLOW_CONN_OTHER: 'other_conn'
-#   {{ .Release.Name }}-other-secret-name-suffix: |
+#   '{{ .Release.Name }}-other-secret-name-suffix': |
 # data: |
 #...
 
@@ -200,7 +200,7 @@ extraSecrets: {}
 extraConfigMaps: {}
 # eg:
 # extraConfigMaps:
-#   {{ .Release.Name }}-airflow-variables:
+#   '{{ .Release.Name }}-airflow-variables':
 # data: |
 #   AIRFLOW_VAR_HELLO_MESSAGE: "Hi!"
 #   AIRFLOW_VAR_KUBERNETES_NAMESPACE: "{{ .Release.Namespace }}"

[GitHub] [airflow] xinbinhuang merged pull request #15633: Fix add quotes to extraConfigMaps/extraSecrets

[GitBox]

xinbinhuang merged pull request #15633:
URL: https://github.com/apache/airflow/pull/15633


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] YuDorogov opened a new pull request #15633: Fix add quotes to extraConfigMaps

[GitBox]

YuDorogov opened a new pull request #15633:
URL: https://github.com/apache/airflow/pull/15633


   Without quotes we gets helm error when **extraSecrets** and/or 
**extraConfigMaps** used with **{{ .Release.Name }}**:
   `Error: failed to parse ./values.yaml: error converting YAML to JSON: yaml: 
line 234: did not find expected key
   helm.go:94: [debug] error converting YAML to JSON: yaml: line 234: did not 
find expected key
   `
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] yuzeh commented on issue #13637: Scheduler takes 100% of CPU without task execution

[GitBox]

yuzeh commented on issue #13637:
URL: https://github.com/apache/airflow/issues/13637#issuecomment-831002551


   To add to @bobo333 's comment, here's the before + after on CPU utilization:
   
   
![image](https://user-images.githubusercontent.com/351023/116839049-a27c7e80-ab85-11eb-9c84-fd4150723456.png)
   
   Other info about our deployment:
   - Python 3.7
   - Everything runs in docker-compose with a custom built docker image 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] bobo333 edited a comment on issue #13637: Scheduler takes 100% of CPU without task execution

[GitBox]

bobo333 edited a comment on issue #13637:
URL: https://github.com/apache/airflow/issues/13637#issuecomment-831001974


   hi, we (@yuzeh and I) are also seeing this behavior on airflow `2.0.2` using 
`LocalExecutor`. With `min_file_process_interval = 60` and `parsing_processes = 
2` cpu usage for the scheduler is very similar to what @linxiaohui posted, both 
processes are using 90% or higher cpu, often up to 150%.
   
   There are no active workloads running during this time either, those only 
run on the hour, but this behavior happens constantly.
   
   Note: this was _not_ happening before we upgraded to airflow 2. On `1.10.14` 
we did not see this level of cpu usage from the scheduler.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] bobo333 commented on issue #13637: Scheduler takes 100% of CPU without task execution

[GitBox]

bobo333 commented on issue #13637:
URL: https://github.com/apache/airflow/issues/13637#issuecomment-831001974


   hi, we (@yuzeh and I) are also seeing this behavior on airflow `2.0.2` using 
`LocalExecutor`. With `min_file_process_interval = 60` and `parsing_processes = 
2` cpu usage for the scheduler is very similar to what @linxiaohui posted, both 
processes are using 90% or higher cpu, often up to 150%.
   
   There are no active workloads running during this time either, those only 
run on the hour, but this behavior happens constantly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] sunkickr opened a new pull request #15632: Add link to Oracle Connection Docs

[GitBox]

sunkickr opened a new pull request #15632:
URL: https://github.com/apache/airflow/pull/15632


   This PR adds a link to the Oracle Connection docs to the docstrings of 
modules that use the oracle conn id. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] iantbutler01 commented on issue #10270: KubernetesPodOperator annotations kwarg should support templating.

[GitBox]

iantbutler01 commented on issue #10270:
URL: https://github.com/apache/airflow/issues/10270#issuecomment-830991921


   I am not, my work has carried me far away from Airflow and I do not have the 
time these days @eladkal 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] closed issue #14538: Task can not exit normally when I use map in billiard

[GitBox]

github-actions[bot] closed issue #14538:
URL: https://github.com/apache/airflow/issues/14538


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on issue #14538: Task can not exit normally when I use map in billiard

[GitBox]

github-actions[bot] commented on issue #14538:
URL: https://github.com/apache/airflow/issues/14538#issuecomment-830933646


   This issue has been closed because it has not received response from the 
issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on issue #14596: Max Active Run Feature is not Running

[GitBox]

github-actions[bot] commented on issue #14596:
URL: https://github.com/apache/airflow/issues/14596#issuecomment-830933634


   This issue has been closed because it has not received response from the 
issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] closed pull request #14666: chore: remove redundant character escape from regex

[GitBox]

github-actions[bot] closed pull request #14666:
URL: https://github.com/apache/airflow/pull/14666


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] closed issue #14851: Airflow workers causing 100% CPU on PostgreSQL Database

[GitBox]

github-actions[bot] closed issue #14851:
URL: https://github.com/apache/airflow/issues/14851


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on issue #14851: Airflow workers causing 100% CPU on PostgreSQL Database

[GitBox]

github-actions[bot] commented on issue #14851:
URL: https://github.com/apache/airflow/issues/14851#issuecomment-830933611


   This issue has been closed because it has not received response from the 
issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] closed issue #14596: Max Active Run Feature is not Running

[GitBox]

github-actions[bot] closed issue #14596:
URL: https://github.com/apache/airflow/issues/14596


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] closed issue #14980: Login

[GitBox]

github-actions[bot] closed issue #14980:
URL: https://github.com/apache/airflow/issues/14980


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #14882: Add Azure Data Factory example DAG

[GitBox]

github-actions[bot] commented on pull request #14882:
URL: https://github.com/apache/airflow/pull/14882#issuecomment-830933603


   This pull request has been automatically marked as stale because it has not 
had recent activity. It will be closed in 5 days if no further activity occurs. 
Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on issue #14980: Login

[GitBox]

github-actions[bot] commented on issue #14980:
URL: https://github.com/apache/airflow/issues/14980#issuecomment-830933585


   This issue has been closed because it has not received response from the 
issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] xinbinhuang commented on issue #12985: PythonVirtualenvOperator with provide_context=True does not have 'ti' keyword

[GitBox]

xinbinhuang commented on issue #12985:
URL: https://github.com/apache/airflow/issues/12985#issuecomment-830928107


   > Also found this Jira story 
(https://issues.apache.org/jira/browse/AIRFLOW-2738) that's still open that 
seems to be addressing the same problem.
   
   Just a quick note: the community has migrated issues tracking system from 
Jira to GitHub issues, so whatever left in Jira right now is likely outdated or 
inaccurate.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] xinbinhuang edited a comment on issue #12985: PythonVirtualenvOperator with provide_context=True does not have 'ti' keyword

[GitBox]

xinbinhuang edited a comment on issue #12985:
URL: https://github.com/apache/airflow/issues/12985#issuecomment-830928107


   > Also found this Jira story 
(https://issues.apache.org/jira/browse/AIRFLOW-2738) that's still open that 
seems to be addressing the same problem.
   
   Just a quick note: the community has migrated the issues tracking system 
from Jira to GitHub issues, so whatever left in Jira right now is likely 
outdated or inaccurate.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] BenoitHanotte edited a comment on pull request #15423: UI: DAG Calendar View

[GitBox]

BenoitHanotte edited a comment on pull request #15423:
URL: https://github.com/apache/airflow/pull/15423#issuecomment-830876792


   > Way better! Good call on `interpolateHsl` We should update the legend to 
explain the range of colors. That should help mitigate your first concern.
   
   @bbovenzi I have updated the legend, it adds a bit of complexity as to 
display the "gradient" in the legend it required moving it to the svg graph 
itself:
   
![image](https://user-images.githubusercontent.com/6229788/116828480-f6dc2a00-ab96-11eb-9d13-3c2046f0be5a.png)
   
   > Would it be more useful to reverse the year ordering? Seems like the 
current year is likely to be the most relevant to the user.
   
   @ryanahamilton  I experimented with it but I fear it might be a bit 
confusing as consecutive days are not juxtaposed from left-to-right and 
top-to-bottom anymore:
   
![image](https://user-images.githubusercontent.com/6229788/116828555-52a6b300-ab97-11eb-8fed-0a0ba58ec0ec.png)
   
   Let me know which option you prefer.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] BenoitHanotte edited a comment on pull request #15423: UI: DAG Calendar View

[GitBox]

BenoitHanotte edited a comment on pull request #15423:
URL: https://github.com/apache/airflow/pull/15423#issuecomment-830876792


   > Way better! Good call on `interpolateHsl` We should update the legend to 
explain the range of colors. That should help mitigate your first concern.
   
   I have updated the legend, it adds a bit of complexity as to display the 
"gradient" in the legend it required moving it to the svg graph itself:
   
![image](https://user-images.githubusercontent.com/6229788/116828480-f6dc2a00-ab96-11eb-9d13-3c2046f0be5a.png)
   
   > Would it be more useful to reverse the year ordering? Seems like the 
current year is likely to be the most relevant to the user.
   
   I experimented with it might be a bit confusing as consecutive days are not 
juxtaposed from left-to-right and top-to-bottom anymore:
   
![image](https://user-images.githubusercontent.com/6229788/116828555-52a6b300-ab97-11eb-8fed-0a0ba58ec0ec.png)
   
   Let me know which option you prefer.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] BenoitHanotte commented on pull request #15423: UI: DAG Calendar View

[GitBox]

BenoitHanotte commented on pull request #15423:
URL: https://github.com/apache/airflow/pull/15423#issuecomment-830876792


   > Way better! Good call on `interpolateHsl` We should update the legend to 
explain the range of colors. That should help mitigate your first concern.
   
   I have updated the legend, it adds a bit of complexity as to display the 
"gradient" in the legend it required moving the legend in the svg object:
   
![image](https://user-images.githubusercontent.com/6229788/116828480-f6dc2a00-ab96-11eb-9d13-3c2046f0be5a.png)
   
   > Would it be more useful to reverse the year ordering? Seems like the 
current year is likely to be the most relevant to the user.
   
   I experimented with it might be a bit confusing as consecutive days are not 
juxtaposed from left-to-right and top-to-bottom anymore:
   
![image](https://user-images.githubusercontent.com/6229788/116828555-52a6b300-ab97-11eb-8fed-0a0ba58ec0ec.png)
   
   Let me know which option you prefer.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Dr-Denzy commented on a change in pull request #15617: Improve Test Coverage for Kubernetes Executor

[GitBox]

Dr-Denzy commented on a change in pull request #15617:
URL: https://github.com/apache/airflow/pull/15617#discussion_r624754291



##
File path: tests/executors/test_kubernetes_executor.py
##
@@ -120,6 +119,71 @@ def test_execution_date_serialize_deserialize(self):
 
 assert datetime_obj == new_datetime_obj
 
+@unittest.skipIf(AirflowKubernetesScheduler is None, 'kubernetes python 
package is not installed')
+@mock.patch('airflow.executors.kubernetes_executor.get_kube_client')
+@mock.patch('airflow.executors.kubernetes_executor.client')
+@mock.patch('airflow.executors.kubernetes_executor.KubernetesJobWatcher')
+def test_delete_pod_successfully(
+self, mock_watcher, mock_client, mock_kube_client
+):  # pylint: disable=unused-argument
+pod_id = "my-pod-1"
+namespace = "my-namespace"
+
+kube_client = mock.MagicMock()
+mock_kube_client.return_value.delete_namespaced_pod = kube_client

Review comment:
   > This feels weird, why would 
`mock_kube_client.return_value.delete_namespaced_pod` be equal to `kube_client` 
?
   
   Hmm... I will change the mock object naming.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Dr-Denzy commented on a change in pull request #15617: Improve Test Coverage for Kubernetes Executor

[GitBox]

Dr-Denzy commented on a change in pull request #15617:
URL: https://github.com/apache/airflow/pull/15617#discussion_r624754291



##
File path: tests/executors/test_kubernetes_executor.py
##
@@ -120,6 +119,71 @@ def test_execution_date_serialize_deserialize(self):
 
 assert datetime_obj == new_datetime_obj
 
+@unittest.skipIf(AirflowKubernetesScheduler is None, 'kubernetes python 
package is not installed')
+@mock.patch('airflow.executors.kubernetes_executor.get_kube_client')
+@mock.patch('airflow.executors.kubernetes_executor.client')
+@mock.patch('airflow.executors.kubernetes_executor.KubernetesJobWatcher')
+def test_delete_pod_successfully(
+self, mock_watcher, mock_client, mock_kube_client
+):  # pylint: disable=unused-argument
+pod_id = "my-pod-1"
+namespace = "my-namespace"
+
+kube_client = mock.MagicMock()
+mock_kube_client.return_value.delete_namespaced_pod = kube_client

Review comment:
   > This feels weird, why would 
`mock_kube_client.return_value.delete_namespaced_pod` be equal to `kube_client` 
?
   
   Hmm... I will change mock object naming.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ByPort opened a new pull request #15631: Add setting up using VS Code

[GitBox]

ByPort opened a new pull request #15631:
URL: https://github.com/apache/airflow/pull/15631


   
   
   Added a few screenshots and extra instructions for VS Code users.
   
   Sadly, currently there are some [issues with using pytest from VS 
Code](https://github.com/microsoft/vscode-python/issues/14570), so I am unable 
to fill TESTING.rst with screenshots and instructions for VS Code users.
   
   Also i would like to add a letter numbering for a second level here:
   
   
![image](https://user-images.githubusercontent.com/7266338/116827187-6f93b400-aba0-11eb-90dd-bf69f686e877.png)
   
   I would be glad if someone could help me with it.
   
   closes: #15205 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] xinbinhuang edited a comment on issue #15547: Jinja templating throws bug when using a PostgresOperator within the Taskflow API

[GitBox]

xinbinhuang edited a comment on issue #15547:
URL: https://github.com/apache/airflow/issues/15547#issuecomment-830841726


   Though the approach suggested by @eladkal works, I would recommend using 
hooks directly as suggested by @uranusjr. Operators represent a single task in 
the airflow DAG graph (with other things/metadata specific to airflow), and 
it's using hooks to do the actual work. If you are finding something that can 
help "do the work", you should use hooks directly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on a change in pull request #15528: Adds script to automate renaming docker images from master to main

[GitBox]

potiuk commented on a change in pull request #15528:
URL: https://github.com/apache/airflow/pull/15528#discussion_r624748717



##
File path: IMAGES.rst
##
@@ -320,10 +320,10 @@ Images built as "Run ID snapshot":
 
 .. code-block:: bash
 
-  docker.pkg.github.com.io/apache-airflow/-pythonX.Y-ci-v2:
- for CI images
-  docker.pkg.github.com/apache-airflow/-pythonX.Y-v2:   - 
for production images
-  docker.pkg.github.com/apache-airflow/-pythonX.Y-build-v2: - 
for production build stage
-  
docker.pkg.github.com/apache-airflow/pythonX.Y--v2:X.Y-slim-buster-
  - for base Python images
+  docker.pkg.github.com.io/apache-airflow/-pythonX.Y-ci-v2:
- for CI images
+  docker.pkg.github.com/apache-airflow/-pythonX.Y-v2:   - 
for production images
+  docker.pkg.github.com/apache-airflow/-pythonX.Y-build-v2: - 
for production build stage
+  docker.pkg.github.com/apache-airflow/python-v2:X.Y-slim-buster-  - 
for base Python images

Review comment:
   And added it as comment in the script.
   
   Indeed. Assumptions (which I have and others don't) are the root of evil :)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on a change in pull request #15528: Adds script to automate renaming docker images from master to main

[GitBox]

potiuk commented on a change in pull request #15528:
URL: https://github.com/apache/airflow/pull/15528#discussion_r624747760



##
File path: IMAGES.rst
##
@@ -320,10 +320,10 @@ Images built as "Run ID snapshot":
 
 .. code-block:: bash
 
-  docker.pkg.github.com.io/apache-airflow/-pythonX.Y-ci-v2:
- for CI images
-  docker.pkg.github.com/apache-airflow/-pythonX.Y-v2:   - 
for production images
-  docker.pkg.github.com/apache-airflow/-pythonX.Y-build-v2: - 
for production build stage
-  
docker.pkg.github.com/apache-airflow/pythonX.Y--v2:X.Y-slim-buster-
  - for base Python images
+  docker.pkg.github.com.io/apache-airflow/-pythonX.Y-ci-v2:
- for CI images
+  docker.pkg.github.com/apache-airflow/-pythonX.Y-v2:   - 
for production images
+  docker.pkg.github.com/apache-airflow/-pythonX.Y-build-v2: - 
for production build stage
+  docker.pkg.github.com/apache-airflow/python-v2:X.Y-slim-buster-  - 
for base Python images

Review comment:
   Absolutely - you are perfectly right @XD-DENG. I've added all the 
explanation as the commit message




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] XD-DENG commented on a change in pull request #15528: Adds script to automate renaming docker images from master to main

[GitBox]

XD-DENG commented on a change in pull request #15528:
URL: https://github.com/apache/airflow/pull/15528#discussion_r624746800



##
File path: IMAGES.rst
##
@@ -320,10 +320,10 @@ Images built as "Run ID snapshot":
 
 .. code-block:: bash
 
-  docker.pkg.github.com.io/apache-airflow/-pythonX.Y-ci-v2:
- for CI images
-  docker.pkg.github.com/apache-airflow/-pythonX.Y-v2:   - 
for production images
-  docker.pkg.github.com/apache-airflow/-pythonX.Y-build-v2: - 
for production build stage
-  
docker.pkg.github.com/apache-airflow/pythonX.Y--v2:X.Y-slim-buster-
  - for base Python images
+  docker.pkg.github.com.io/apache-airflow/-pythonX.Y-ci-v2:
- for CI images
+  docker.pkg.github.com/apache-airflow/-pythonX.Y-v2:   - 
for production images
+  docker.pkg.github.com/apache-airflow/-pythonX.Y-build-v2: - 
for production build stage
+  docker.pkg.github.com/apache-airflow/python-v2:X.Y-slim-buster-  - 
for base Python images

Review comment:
   Personally I'm ok to keep them here, but good to update the commit msg 
and PR title/description as well. Do you agree?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on a change in pull request #15528: Adds script to automate renaming docker images from master to main

[GitBox]

potiuk commented on a change in pull request #15528:
URL: https://github.com/apache/airflow/pull/15528#discussion_r624746604



##
File path: dev/rename_docker_images_to_main.py
##
@@ -0,0 +1,70 @@
+#!/usr/bin/python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import subprocess
+from typing import List
+
+PYTHON_VERSIONS = ["3.6", "3.7", "3.8"]
+
+DOCKERHUB_PREFIX = "apache/airflow"
+
+DOCKERHUB_IMAGES = [
+"{prefix}:python{python_version}-{branch}",

Review comment:
   The problem with DOCKERHUB_IMAGES is that they are all "apache/airflow" 
- because that's the only image we have access to in DockerHub. So in DockerHub 
the image is always "apache/airflow" but has a different tag. 
   
   In GitHub registries (depends on the type of registry) we have more 
flexibility":
   * in the old GitHub docker registry - docker.pkg.github.com -  (current but 
already deprecated) we can use "apache/airflow/IMAGE:tag"
   * in the new package registry (ghcr.io) - we can submit anything under 
apache/airflow-* but then we link it to the project via docker image label.
   
   So naming conventions for all three types of images are different :(




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on a change in pull request #15528: Adds script to automate renaming docker images from master to main

[GitBox]

potiuk commented on a change in pull request #15528:
URL: https://github.com/apache/airflow/pull/15528#discussion_r624745678



##
File path: IMAGES.rst
##
@@ -320,10 +320,10 @@ Images built as "Run ID snapshot":
 
 .. code-block:: bash
 
-  docker.pkg.github.com.io/apache-airflow/-pythonX.Y-ci-v2:
- for CI images
-  docker.pkg.github.com/apache-airflow/-pythonX.Y-v2:   - 
for production images
-  docker.pkg.github.com/apache-airflow/-pythonX.Y-build-v2: - 
for production build stage
-  
docker.pkg.github.com/apache-airflow/pythonX.Y--v2:X.Y-slim-buster-
  - for base Python images
+  docker.pkg.github.com.io/apache-airflow/-pythonX.Y-ci-v2:
- for CI images
+  docker.pkg.github.com/apache-airflow/-pythonX.Y-v2:   - 
for production images
+  docker.pkg.github.com/apache-airflow/-pythonX.Y-build-v2: - 
for production build stage
+  docker.pkg.github.com/apache-airflow/python-v2:X.Y-slim-buster-  - 
for base Python images

Review comment:
   They are correcting. The names were wrong (I found it while preparing 
the script). Happy to split it off if needed 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #15528: Adds script to automate renaming docker images from master to main

[GitBox]

potiuk commented on pull request #15528:
URL: https://github.com/apache/airflow/pull/15528#issuecomment-830861670


   Those are scripts to move the GitHub and DockerHub "master" images to "main" 
 (basically re-tagging them). Since we are going to implement the "master" -> 
"main" rename eventually 
(https://lists.apache.org/thread.html/r97f52ffa37868be325c22c48cef2e28639538c35f99eff4807931556%40%3Cdev.airflow.apache.org%3E)
 - having script to do it semi-automatically will be helpful. 
   
   Those images are used in Github Actions and Breeze (as cache of the latest 
built images) - thank to those building an image in GitHub CI takes < 1 m in 
most cases and refreshing Breeze image is also < 2 minutes (instead of ~ 10-15 
minutes).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] XD-DENG commented on a change in pull request #15528: Adds script to automate renaming docker images from master to main

[GitBox]

XD-DENG commented on a change in pull request #15528:
URL: https://github.com/apache/airflow/pull/15528#discussion_r624743014



##
File path: dev/rename_docker_images_to_main.py
##
@@ -0,0 +1,70 @@
+#!/usr/bin/python3

Review comment:
   Do we have a documentation change to record usage/purpose of this script?

##
File path: dev/rename_docker_images_to_main.py
##
@@ -0,0 +1,70 @@
+#!/usr/bin/python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import subprocess
+from typing import List
+
+PYTHON_VERSIONS = ["3.6", "3.7", "3.8"]
+
+DOCKERHUB_PREFIX = "apache/airflow"
+
+DOCKERHUB_IMAGES = [
+"{prefix}:python{python_version}-{branch}",

Review comment:
   Is image tag purposely ignored for `DOCKERHUB_IMAGES`?
   
   I notice they are all `latest` for `GITHUB_REGISTRY_IMAGES` and 
`GHCR_IO_IMAGES`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] CoburnJoe commented on issue #15570: Gunicorn dependency request smuggling vulnerability

[GitBox]

CoburnJoe commented on issue #15570:
URL: https://github.com/apache/airflow/issues/15570#issuecomment-830858847


   > @CoburnJoe -> Airflow master is at Gunicorn 20.1 now (#15611). Next 
release should use it.
   
   Fantastic - thank you! 🎉 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] XD-DENG commented on a change in pull request #15528: Adds script to automate renaming docker images from master to main

[GitBox]

XD-DENG commented on a change in pull request #15528:
URL: https://github.com/apache/airflow/pull/15528#discussion_r624742560



##
File path: IMAGES.rst
##
@@ -320,10 +320,10 @@ Images built as "Run ID snapshot":
 
 .. code-block:: bash
 
-  docker.pkg.github.com.io/apache-airflow/-pythonX.Y-ci-v2:
- for CI images
-  docker.pkg.github.com/apache-airflow/-pythonX.Y-v2:   - 
for production images
-  docker.pkg.github.com/apache-airflow/-pythonX.Y-build-v2: - 
for production build stage
-  
docker.pkg.github.com/apache-airflow/pythonX.Y--v2:X.Y-slim-buster-
  - for base Python images
+  docker.pkg.github.com.io/apache-airflow/-pythonX.Y-ci-v2:
- for CI images
+  docker.pkg.github.com/apache-airflow/-pythonX.Y-v2:   - 
for production images
+  docker.pkg.github.com/apache-airflow/-pythonX.Y-build-v2: - 
for production build stage
+  docker.pkg.github.com/apache-airflow/python-v2:X.Y-slim-buster-  - 
for base Python images

Review comment:
   Are these changes simply cleaning/correcting? Or they are related to the 
main change in this PR?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated: Better description of UID/GID behaviour in image and quickstart (#15592)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/master by this push:
 new 4226f64  Better description of UID/GID behaviour in image and 
quickstart (#15592)
4226f64 is described below

commit 4226f640864b0996dc156da244047a34c58be63c
Author: Jarek Potiuk 
AuthorDate: Sun May 2 20:06:57 2021 +0200

Better description of UID/GID behaviour in image and quickstart (#15592)

* Better description of UID/GID behaviour in image and quickstart

Following the discussion in
https://github.com/apache/airflow/discussions/15579
seems that the AIRFLOW_UID/GID parameters were not clearly
explained in the Docker Quick-start guide and some users could
find it confusing.

This PR attempts to clarify it.

* fixup! Better description of UID/GID behaviour in image and quickstart
---
 docs/apache-airflow/start/docker.rst | 52 +++-
 docs/docker-stack/entrypoint.rst | 22 +++
 2 files changed, 73 insertions(+), 1 deletion(-)

diff --git a/docs/apache-airflow/start/docker.rst 
b/docs/apache-airflow/start/docker.rst
index abd63dc..26a9132 100644
--- a/docs/apache-airflow/start/docker.rst
+++ b/docs/apache-airflow/start/docker.rst
@@ -62,6 +62,9 @@ Some directories in the container are mounted, which means 
that their contents a
 
 This file uses the latest Airflow image (`apache/airflow 
`__). If you need install a new Python 
library or system library, you can :doc:`customize and extend it 
`.
 
+.. _initializing_docker_compose_environment:
+
+
 Initializing Environment
 
 
@@ -74,6 +77,8 @@ On **Linux**, the mounted volumes in container use the native 
Linux filesystem u
 mkdir ./dags ./logs ./plugins
 echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env
 
+See:ref:`Docker Compose environment variables `
+
 On **all operating systems**, you need to run database migrations and create 
the first user account. To do it, run.
 
 .. code-block:: bash
@@ -185,7 +190,6 @@ Here is a sample ``curl`` command, which sends a request to 
retrieve a pool list
 --user "airflow:airflow" \
 "${ENDPOINT_URL}/api/v1/pools"
 
-
 Cleaning up
 ===
 
@@ -207,3 +211,49 @@ What's Next?
 
 
 From this point, you can head to the :doc:`/tutorial` section for further 
examples or the :doc:`/howto/index` section if you're ready to get your hands 
dirty.
+
+.. _docker-compose-env-variables:
+
+Environment variables supported by Docker Compose
+=
+
+Do not confuse the variable names here with the build arguments set when image 
is built. The
+``AIRFLOW_UID`` and ``AIRFLOW_GID`` build args default to ``5`` when the 
image is built, so they are
+"baked" into the image. On the other hand, the environment variables below can 
be set when the container
+is running, using - for example - result of ``id -u`` command, which allows to 
use the dynamic host
+runtime user id which is unknown at the time of building the image.
+
+++-+--+
+|   Variable | Description 
| Default  |
+++=+==+
+| ``AIRFLOW_IMAGE_NAME`` | Airflow Image to use.   
| apache/airflow:|version| |
+++-+--+
+| ``AIRFLOW_UID``| UID of the user to run Airflow containers 
as.   | ``5``|
+|| Override if you want to use use non-default 
Airflow |  |
+|| UID (for example when you map folders from 
host,|  |
+|| it should be set to result of ``id -u`` 
call. If|  |
+|| you change it from default 5, you must 
set  |  |
+|| ``AIRFLOW_GID`` to ``0``. When it is 
changed,   |  |
+|| a 2nd user with the UID specified is 
dynamically|  |
+|| created with ``default`` name inside the 
container  |  |
+|| and home of the use is set to 
``/airflow/home/``|  |
+|| in order to share Python libraries 
installed there. |   

[GitHub] [airflow] potiuk merged pull request #15592: Better description of UID/GID behaviour in image and quickstart

[GitBox]

potiuk merged pull request #15592:
URL: https://github.com/apache/airflow/pull/15592


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #15592: Better description of UID/GID behaviour in image and quickstart

[GitBox]

github-actions[bot] commented on pull request #15592:
URL: https://github.com/apache/airflow/pull/15592#issuecomment-830847125


   The PR is likely ready to be merged. No tests are needed as no important 
environment files, nor python files were modified by it. However, committers 
might decide that full test matrix is needed and add the 'full tests needed' 
label. Then you should rebase it to the latest master or amend the last commit 
of the PR, and push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #10426: Multiple prefixes in GoogleCloudStorageListOperator and GoogleCloudStorageDeleteOperator

[GitBox]

potiuk commented on issue #10426:
URL: https://github.com/apache/airflow/issues/10426#issuecomment-830846445


   @eladkal 's proposal sounds good.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #15623: Testing status of the Providers released 2021.05.01

[GitBox]

potiuk commented on issue #15623:
URL: https://github.com/apache/airflow/issues/15623#issuecomment-830846213


   Unless this is really serious, I think I'd prefer to leave it for a 
follow-up ad-hoc release (1.0.3)? The mechanics of releasing providers works 
best if we release them in batches. From the bug description, I do not think 
the bug was critical (correct me if I am wrong?) so 1.0.2 will still be mostly 
working OK, so we can easily make a (very fast) 1.0.3 release right after?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #15547: Jinja templating throws bug when using a PostgresOperator within the Taskflow API

[GitBox]

potiuk commented on issue #15547:
URL: https://github.com/apache/airflow/issues/15547#issuecomment-830843697


   Yep. Using hook is the way to go.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] zhzhang commented on issue #15623: Testing status of the Providers released 2021.05.01

[GitBox]

zhzhang commented on issue #15623:
URL: https://github.com/apache/airflow/issues/15623#issuecomment-830843581


   I had another [PR](https://github.com/apache/airflow/pull/15412) on the 
postgres provider get merged just now, any chance it can be included in the RC?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #15592: Better description of UID/GID behaviour in image and quickstart

[GitBox]

potiuk commented on pull request #15592:
URL: https://github.com/apache/airflow/pull/15592#issuecomment-830843054


   @kaxil :)?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] xinbinhuang edited a comment on pull request #15412: PostgresHook: deepcopy connection to avoid mutating connection obj

[GitBox]

xinbinhuang edited a comment on pull request #15412:
URL: https://github.com/apache/airflow/pull/15412#issuecomment-830843018


   @zhzhang @kurtqq  Nice catch! thanks for the PR


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] xinbinhuang commented on pull request #15412: PostgresHook: deepcopy connection to avoid mutating connection obj

[GitBox]

xinbinhuang commented on pull request #15412:
URL: https://github.com/apache/airflow/pull/15412#issuecomment-830843018


   @zhzhang Nice catch! thanks for the PR


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated (bf81d2e -> fc845ca)

[binh]

This is an automated email from the ASF dual-hosted git repository.

binh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from bf81d2e  Add docker-context-files detection and cleanup flag. (#15593)
 add fc845ca  PostgresHook: deepcopy connection to avoid mutating 
connection obj (#15412)

No new revisions were added by this update.

Summary of changes:
 airflow/providers/postgres/hooks/postgres.py|  3 ++-
 tests/providers/postgres/hooks/test_postgres.py | 12 
 2 files changed, 14 insertions(+), 1 deletion(-)

[GitHub] [airflow] xinbinhuang merged pull request #15412: PostgresHook: deepcopy connection to avoid mutating connection obj

[GitBox]

xinbinhuang merged pull request #15412:
URL: https://github.com/apache/airflow/pull/15412


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] xinbinhuang commented on issue #15547: Jinja templating throws bug when using a PostgresOperator within the Taskflow API

[GitBox]

xinbinhuang commented on issue #15547:
URL: https://github.com/apache/airflow/issues/15547#issuecomment-830841726


   Though the approach suggested by @eladkal works, I would recommend using 
hooks directly as suggested by @uranusjr. Operators represent a single task in 
the airflow DAG graph (with other things/metadata specific to airflow), and 
it's using hooks to do the actual work. If you are finding something that can 
help do the "work", you should use hooks directly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow-ci-infra] branch add-gcp-cloud-init updated (e20c7b0 -> 2df17fd)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch add-gcp-cloud-init
in repository https://gitbox.apache.org/repos/asf/airflow-ci-infra.git.


 discard e20c7b0  Cloud init is adapted to run GCE instance
 discard 11956c7  Updated requirements.
omit ccb5a79  Do not pre-bake images in the instance
 add 1328951  Do not pre-bake images in the instance (#13)
 add 4ff4feb  User Packer to build a pre-built AMI with everything we need 
(#15)
 add 86bb048  Fix runner AMI so it (#16)
 add e036ae9  Don't encrypt the AMI's root snapshot (#17)
 add 2df17fd  Cloud init is adapted to run GCE instance

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (e20c7b0)
\
 N -- N -- N   refs/heads/add-gcp-cloud-init (2df17fd)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .gitignore |  35 +
 .../packer/files/actions-runner-ec2-reporting.sh   |   8 +-
 .../packer/files/actions.runner-supervisor.service |  29 ++--
 .../packer/files/actions.runner.service|  35 +++--
 .../packer/files/cloudwatch-metrics-github-runners |   4 +-
 .../packer/files/docker-compose.sh |  24 +--
 .../packer/files/install-dependencies.sh   |  37 +++--
 .../packer/files/install-files.sh  |  26 ++--
 .../packer/files/mounts_setup.sh   |   6 +-
 .../packer/files/rules.v4  |  21 +--
 .../packer/files/runner|   4 +-
 .../packer/files/runner-cleanup-workdir.sh |  32 ++--
 .../packer/files}/runner-supervisor.py |   0
 github-runner-ami/packer/files/runner_bootstrap.sh |  45 ++
 .../packer/files/source-list-additions.sh  |   9 +-
 .../packer/files/stop-runner-if-no-job.sh  |  36 +++--
 github-runner-ami/packer/files/timber.key  |  36 +
 github-runner-ami/packer/files/vector.toml |  98 
 github-runner-ami/packer/ubuntu2004.pkr.hcl| 148 ++
 .../packer/vars/variables.pkrvars.hcl  |  10 +-
 lambdas/scale_out_runner/requirements.txt  |   1 -
 terraform/packer_roles/.terraform.lock.hcl |  38 +
 terraform/packer_roles/main.tf | 171 +
 .../.gitignore => terraform/packer_roles/output.tf |   6 +-
 24 files changed, 716 insertions(+), 143 deletions(-)
 copy lambdas/scale_out_runner/.gitignore => 
github-runner-ami/packer/files/actions-runner-ec2-reporting.sh (74%)
 copy tests/lambdas/scale_out_runner/conftest.py => 
github-runner-ami/packer/files/actions.runner-supervisor.service (65%)
 copy tests/lambdas/scale_out_runner/conftest.py => 
github-runner-ami/packer/files/actions.runner.service (55%)
 copy lambdas/scale_out_runner/.gitignore => 
github-runner-ami/packer/files/cloudwatch-metrics-github-runners (92%)
 copy tests/lambdas/scale_out_runner/conftest.py => 
github-runner-ami/packer/files/docker-compose.sh (63%)
 copy tests/lambdas/scale_out_runner/conftest.py => 
github-runner-ami/packer/files/install-dependencies.sh (53%)
 copy tests/lambdas/scale_out_runner/conftest.py => 
github-runner-ami/packer/files/install-files.sh (52%)
 copy lambdas/scale_out_runner/.gitignore => 
github-runner-ami/packer/files/mounts_setup.sh (83%)
 copy requirements.txt => github-runner-ami/packer/files/rules.v4 (67%)
 copy lambdas/scale_out_runner/.gitignore => 
github-runner-ami/packer/files/runner (89%)
 copy tests/lambdas/scale_out_runner/conftest.py => 
github-runner-ami/packer/files/runner-cleanup-workdir.sh (61%)
 rename {scripts => github-runner-ami/packer/files}/runner-supervisor.py (100%)
 create mode 100644 github-runner-ami/packer/files/runner_bootstrap.sh
 copy lambdas/scale_out_runner/.gitignore => 
github-runner-ami/packer/files/source-list-additions.sh (71%)
 copy tests/lambdas/scale_out_runner/conftest.py => 
github-runner-ami/packer/files/stop-runner-if-no-job.sh (54%)
 create mode 100644 github-runner-ami/packer/files/timber.key
 create mode 100644 github-runner-ami/packer/files/vector.toml
 create mode 100644 github-runner-ami/packer/ubuntu2004.pkr.hcl
 copy lambdas/scale_out_runner/.gitignore => 
github-runner-ami/packer/vars/variables.pkrvars.hcl (73%)
 create mode 100644 terraform/packer_roles/.terraform.lock.hcl
 create mode 100644 terraform/packer_roles/main.tf
 copy lambdas/

[airflow-ci-infra] branch update-requirements updated (11956c7 -> d047252)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch update-requirements
in repository https://gitbox.apache.org/repos/asf/airflow-ci-infra.git.


omit 11956c7  Updated requirements.
omit ccb5a79  Do not pre-bake images in the instance
 add 1328951  Do not pre-bake images in the instance (#13)
 add 4ff4feb  User Packer to build a pre-built AMI with everything we need 
(#15)
 add 86bb048  Fix runner AMI so it (#16)
 add e036ae9  Don't encrypt the AMI's root snapshot (#17)
 add d047252  Updated requirements.

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (11956c7)
\
 N -- N -- N   refs/heads/update-requirements (d047252)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .gitignore |  35 +
 .../packer/files/actions-runner-ec2-reporting.sh   |  13 +-
 .../packer/files/actions.runner-supervisor.service |  29 ++--
 .../packer/files/actions.runner.service|  35 +++--
 .../packer/files/cloudwatch-metrics-github-runners |   4 +-
 .../packer/files/docker-compose.sh |  24 +--
 .../packer/files/install-dependencies.sh   |  37 +++--
 .../packer/files/install-files.sh  |  26 ++--
 .../packer/files/mounts_setup.sh   |  11 +-
 .../packer/files/rules.v4  |  19 ++-
 .../packer/files/runner|   9 +-
 .../packer/files/runner-cleanup-workdir.sh |  32 ++--
 .../packer/files}/runner-supervisor.py |   0
 github-runner-ami/packer/files/runner_bootstrap.sh |  45 ++
 .../packer/files/source-list-additions.sh  |  14 +-
 .../packer/files/stop-runner-if-no-job.sh  |  36 +++--
 github-runner-ami/packer/files/timber.key  |  36 +
 github-runner-ami/packer/files/vector.toml |  98 
 github-runner-ami/packer/ubuntu2004.pkr.hcl| 148 ++
 .../packer/vars/variables.pkrvars.hcl  |  15 +-
 terraform/packer_roles/.terraform.lock.hcl |  38 +
 terraform/packer_roles/main.tf | 171 +
 .../.gitignore => terraform/packer_roles/output.tf |   6 +-
 23 files changed, 716 insertions(+), 165 deletions(-)
 copy requirements.txt => 
github-runner-ami/packer/files/actions-runner-ec2-reporting.sh (74%)
 copy tests/lambdas/scale_out_runner/conftest.py => 
github-runner-ami/packer/files/actions.runner-supervisor.service (65%)
 copy tests/lambdas/scale_out_runner/conftest.py => 
github-runner-ami/packer/files/actions.runner.service (55%)
 copy lambdas/scale_out_runner/.gitignore => 
github-runner-ami/packer/files/cloudwatch-metrics-github-runners (92%)
 copy tests/lambdas/scale_out_runner/conftest.py => 
github-runner-ami/packer/files/docker-compose.sh (63%)
 copy tests/lambdas/scale_out_runner/conftest.py => 
github-runner-ami/packer/files/install-dependencies.sh (53%)
 copy tests/lambdas/scale_out_runner/conftest.py => 
github-runner-ami/packer/files/install-files.sh (52%)
 copy requirements.txt => github-runner-ami/packer/files/mounts_setup.sh (83%)
 copy requirements.txt => github-runner-ami/packer/files/rules.v4 (67%)
 copy requirements.txt => github-runner-ami/packer/files/runner (89%)
 copy tests/lambdas/scale_out_runner/conftest.py => 
github-runner-ami/packer/files/runner-cleanup-workdir.sh (61%)
 rename {scripts => github-runner-ami/packer/files}/runner-supervisor.py (100%)
 create mode 100644 github-runner-ami/packer/files/runner_bootstrap.sh
 copy requirements.txt => 
github-runner-ami/packer/files/source-list-additions.sh (71%)
 copy tests/lambdas/scale_out_runner/conftest.py => 
github-runner-ami/packer/files/stop-runner-if-no-job.sh (54%)
 create mode 100644 github-runner-ami/packer/files/timber.key
 create mode 100644 github-runner-ami/packer/files/vector.toml
 create mode 100644 github-runner-ami/packer/ubuntu2004.pkr.hcl
 copy requirements.txt => github-runner-ami/packer/vars/variables.pkrvars.hcl 
(73%)
 create mode 100644 terraform/packer_roles/.terraform.lock.hcl
 create mode 100644 terraform/packer_roles/main.tf
 copy lambdas/scale_out_runner/.gitignore => terraform/packer_roles/output.tf 
(92%)

[GitHub] [airflow-ci-infra] potiuk opened a new pull request #19: Add gcp cloud init

[GitBox]

potiuk opened a new pull request #19:
URL: https://github.com/apache/airflow-ci-infra/pull/19


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow-ci-infra] 01/01: Cloud init is adapted to run GCE instance

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch add-gcp-cloud-init
in repository https://gitbox.apache.org/repos/asf/airflow-ci-infra.git

commit e20c7b07d10d012f12ea974e3c5656fabf85005a
Author: Jarek Potiuk 
AuthorDate: Sun May 2 18:54:44 2021 +0200

Cloud init is adapted to run GCE instance
---
 cloud-init.yml | 66 +-
 gcp/README.md  | 45 +++
 gcp/metrics/gcp_create_metrics_descriptor.py   | 56 ++
 .../metrics/gcp_delete_metrics_descriptor.py   | 23 +++-
 gcp/metrics/gcp_write_metrics_data.py  | 57 +++
 lambdas/scale_out_runner/app.py|  7 ++-
 lambdas/scale_out_runner/requirements.txt  |  1 +
 requirements.txt   |  2 +
 scripts/runner-supervisor.py   |  2 +-
 9 files changed, 240 insertions(+), 19 deletions(-)

diff --git a/cloud-init.yml b/cloud-init.yml
index 05d71d5..af14d2c 100644
--- a/cloud-init.yml
+++ b/cloud-init.yml
@@ -39,14 +39,21 @@ runcmd:
 - -c
 - |
   set -eu -o pipefail
-  echo "AWS_DEFAULT_REGION=$(cloud-init query region)" >> /etc/environment
+  if [[ $(cloud-init query cloud_name) == "aws" ]]; then
+  echo "AWS_DEFAULT_REGION=$(cloud-init query region)" >> 
/etc/environment
+  fi
+  if [[ $(cloud-init query cloud_name) == "gce" ]]; then
+  echo "GCP_DEFAULT_REGION=$(cloud-init query region)" >> 
/etc/environment
+  fi
   # Set an env var (that is visible in runners) that will let us know we 
are on a self-hosted runner
   echo 'AIRFLOW_SELF_HOSTED_RUNNER="[\"self-hosted\"]"' >> /etc/environment
   set -a
   . /etc/environment
   set +a
-  echo "ASG_GROUP_NAME=$(aws ec2 describe-tags --filter 
Name=resource-id,Values=$(cloud-init query instance_id) 
Name=key,Values=aws:autoscaling:groupName \
-| jq -r '@sh "\(.Tags[0].Value)"')" >> /etc/environment
+  if [[ $(cloud-init query cloud_name) == "aws" ]]; then
+  echo "ASG_GROUP_NAME=$(aws ec2 describe-tags --filter 
Name=resource-id,Values=$(cloud-init query instance_id) 
Name=key,Values=aws:autoscaling:groupName \
+  | jq -r '@sh "\(.Tags[0].Value)"')" >> /etc/environment
+  fi
   - [systemctl, daemon-reload]
   -
 - bash
@@ -75,10 +82,27 @@ runcmd:
   . /etc/environment
   set +a
 
-  aws s3 cp s3://airflow-ci-assets/runner-supervisor.py 
/opt/runner-supervisor/bin/runner-supervisor
+  if [[ $(cloud-init query cloud_name) == "aws" ]]; then
+  aws s3 cp s3://airflow-ci-assets/runner-supervisor.py 
/opt/runner-supervisor/bin/runner-supervisor
+  fi
+  if [[ $(cloud-init query cloud_name) == "gce" ]]; then
+  gsutil cp gs://airflow-ci-assets/runner-supervisor.py 
/opt/runner-supervisor/bin/runner-supervisor
+  fi
   chmod 755 /opt/runner-supervisor/bin/runner-supervisor
-
 - 2.277.1-airflow3
+  -
+- bash
+- -c
+- |
+  set -eu -o pipefail
+  if [[ $(cloud-init query cloud_name) == "gce" ]]; then
+  gsutil cp gs://airflow-ci-assets/requirements.txt 
/opt/requirements.txt
+  python3 -mvenv /opt/gcp-metrics-writer
+  /opt/gcp-metrics-writer/bin/pip install -r /opt/requirements.txt
+  gsutil cp gs://airflow-ci-assets/gcp_write_metrics_data.py 
/opt/gcp-metrics-writer/bin/gcp_write_metrics_data
+  chmod a+x /opt/gcp-metrics-writer/bin/gcp_write_metrics_data
+  fi
+
   - [systemctl, enable, --now, iptables.service]
   # Restart docker after applying the user firewall -- else some rules/chains 
might be list!
   - [systemctl, restart, docker.service]
@@ -120,9 +144,15 @@ write_files:
   docker ps -qa | xargs --verbose --no-run-if-empty docker rm -fv
 
   echo "Log in to a paid docker user to get unlimited docker pulls"
-  aws ssm get-parameter --with-decryption --name 
/runners/apache/airflow/dockerPassword | \
-jq .Parameter.Value -r | \
-sudo -u runner docker login --username airflowcirunners 
--password-stdin
+  if [[ $(cloud-init query cloud_name) == "aws" ]]; then
+  aws ssm get-parameter --with-decryption --name 
/runners/apache/airflow/dockerPassword | \
+jq .Parameter.Value -r | \
+sudo -u runner docker login --username airflowcirunners 
--password-stdin
+  fi
+  if [[ $(cloud-init query cloud_name) == "gce" ]]; then
+  gcloud secrets versions access latest --secret 
"runners-apache-airflow-dockerPassword" | \
+sudo -u runner docker login --username airflowcirunners 
--password-stdin
+  fi
 
   if [[ -d ~runner/actions-runner/_work/airflow/airflow ]]; then
 cd ~runner/actions-runner/_work/airflow/airflow
@@ -134,7 +164,8 @@ write_files:
 git submodule deinit --all -f && \
 git submodule fore

[airflow-ci-infra] branch add-gcp-cloud-init created (now e20c7b0)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch add-gcp-cloud-init
in repository https://gitbox.apache.org/repos/asf/airflow-ci-infra.git.


  at e20c7b0  Cloud init is adapted to run GCE instance

This branch includes the following new commits:

 new e20c7b0  Cloud init is adapted to run GCE instance

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[GitHub] [airflow] kurtqq commented on pull request #15412: PostgresHook: deepcopy connection to avoid mutating connection obj

[GitBox]

kurtqq commented on pull request #15412:
URL: https://github.com/apache/airflow/pull/15412#issuecomment-830832357


   you need one of the maintainers to approve
   @kaxil @dstandish @XD-DENG @xinbinhuang 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] malthe commented on a change in pull request #15581: Add bindvars Xcom result to Oracle operator

[GitBox]

malthe commented on a change in pull request #15581:
URL: https://github.com/apache/airflow/pull/15581#discussion_r624716665



##
File path: airflow/providers/oracle/operators/oracle.py
##
@@ -63,4 +68,25 @@ def __init__(
 def execute(self, context) -> None:
 self.log.info('Executing: %s', self.sql)
 hook = OracleHook(oracle_conn_id=self.oracle_conn_id)
-hook.run(self.sql, autocommit=self.autocommit, 
parameters=self.parameters)
+
+def handler(cur):
+bindvars = cur.bindvars
+
+if isinstance(bindvars, list):
+bindvars = [v.getvalue() for v in bindvars]
+elif isinstance(bindvars, dict):
+bindvars = {n: v.getvalue() for (n, v) in bindvars.items()}
+else:
+raise TypeError(bindvars)
+
+return {
+"bindvars": bindvars,
+"rows": cur.fetchall(),
+}
+
+return hook.run(

Review comment:
   @potiuk good point – fixed now in 
b81bf4d6de52d4528e58324722a2118235570da0.
   
   Thanks!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] davido912 commented on issue #15630: testing that this still works

[GitBox]

davido912 commented on issue #15630:
URL: https://github.com/apache/airflow/issues/15630#issuecomment-830822251


   This was an accident :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] davido912 closed issue #15630: testing that this still works

[GitBox]

davido912 closed issue #15630:
URL: https://github.com/apache/airflow/issues/15630


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] davido912 opened a new issue #15630: testing that this still works

[GitBox]

davido912 opened a new issue #15630:
URL: https://github.com/apache/airflow/issues/15630


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ArtemisDicoTiar closed issue #15629: wtform

[GitBox]

ArtemisDicoTiar closed issue #15629:
URL: https://github.com/apache/airflow/issues/15629


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ArtemisDicoTiar opened a new issue #15629: wtform

[GitBox]

ArtemisDicoTiar opened a new issue #15629:
URL: https://github.com/apache/airflow/issues/15629


   
   
   
   
   **Apache Airflow version**:
   
   
   **Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
   
   **Environment**:
   
   - **Cloud provider or hardware configuration**:
   - **OS** (e.g. from /etc/os-release):
   - **Kernel** (e.g. `uname -a`):
   - **Install tools**:
   - **Others**:
   
   **What happened**:
   
   
   
   **What you expected to happen**:
   
   
   
   **How to reproduce it**:
   
   
   
   **Anything else we need to know**:
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #15629: wtform

[GitBox]

boring-cyborg[bot] commented on issue #15629:
URL: https://github.com/apache/airflow/issues/15629#issuecomment-830819751


   Thanks for opening your first issue here! Be sure to follow the issue 
template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #9046: TimeDeltaSensor and schedule_interval="@once" leads to crash

[GitBox]

eladkal commented on issue #9046:
URL: https://github.com/apache/airflow/issues/9046#issuecomment-830801831


   As explained it doesn't make sense to use `@once` with TimeDeltaSensor. Note 
that the sensor calculates `execution_date + schedule_interval` when you set 
`schedule_interval="@once"` there is no interval (so it's None) which is 
exactly what the error message says: 
   `TypeError: unsupported operand type(s) for +=: 'NoneType' and 
'datetime.timedelta'`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] eladkal closed issue #9046: TimeDeltaSensor and schedule_interval="@once" leads to crash

[GitBox]

eladkal closed issue #9046:
URL: https://github.com/apache/airflow/issues/9046


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Robstaa commented on issue #15547: Jinja templating throws bug when using a PostgresOperator within the Taskflow API

[GitBox]

Robstaa commented on issue #15547:
URL: https://github.com/apache/airflow/issues/15547#issuecomment-830796404


   That makes sense, thank you so much! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #10426: Multiple prefixes in GoogleCloudStorageListOperator and GoogleCloudStorageDeleteOperator

[GitBox]

eladkal commented on issue #10426:
URL: https://github.com/apache/airflow/issues/10426#issuecomment-830793326


   > What I'm planning to do is to modify the `GCSHook.list()` method to accept 
`prefixes` instead of `prefix`. I need to know how we can do that with backward 
compatibility? Some old code will assume this hook is accepting one prefix and 
we need to raise a deprecation warning. Or maybe it is only used internally and 
I need to refactor the operators who use it?
   
   `prefix` is a parameter of `list_blobs` 
https://googleapis.dev/python/storage/latest/client.html
   even if you modify the parameter on the hook at the end you will still be 
able to utalize only single prefix each time.
   You can modify prefix to accept `Optional[str,List[str]]` that way the 
modification is also backward compatible. This has some similarities to 
approach suggested on https://github.com/apache/airflow/issues/15001


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #10270: KubernetesPodOperator annotations kwarg should support templating.

[GitBox]

eladkal commented on issue #10270:
URL: https://github.com/apache/airflow/issues/10270#issuecomment-830791256


   @iantbutler01 are you still working on it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #8713: Spark SQL Hook not using connections

[GitBox]

eladkal commented on issue #8713:
URL: https://github.com/apache/airflow/issues/8713#issuecomment-830790606


   @danielenricocahall I'm unasinging you as you didn't complete the PR if you 
wish to finish it let us know.
   
   This issue is open to anyone who wants to work on it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated (db557a8 -> bf81d2e)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from db557a8  Docs: Replace 'airflow' to 'apache-airflow' to install extra 
(#15628)
 add bf81d2e  Add docker-context-files detection and cleanup flag. (#15593)

No new revisions were added by this update.

Summary of changes:
 BREEZE.rst | 18 +++
 breeze | 17 ++
 breeze-complete|  1 +
 .../ci/images/ci_wait_for_and_verify_ci_image.sh   |  2 +-
 .../ci/images/ci_wait_for_and_verify_prod_image.sh |  2 +-
 scripts/ci/libraries/_build_images.sh  | 60 --
 scripts/ci/libraries/_initialization.sh|  3 ++
 7 files changed, 75 insertions(+), 28 deletions(-)

[GitHub] [airflow] potiuk merged pull request #15593: Add docker-context-files detection and cleanup flag.

[GitBox]

potiuk merged pull request #15593:
URL: https://github.com/apache/airflow/pull/15593


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #15593: Add docker-context-files detection and cleanup flag.

[GitBox]

github-actions[bot] commented on pull request #15593:
URL: https://github.com/apache/airflow/pull/15593#issuecomment-830775282


   The PR most likely needs to run full matrix of tests because it modifies 
parts of the core of Airflow. However, committers might decide to merge it 
quickly and take the risk. If they don't merge it quickly - please rebase it to 
the latest master at your convenience, or amend the last commit of the PR, and 
push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on a change in pull request #15581: Add bindvars Xcom result to Oracle operator

[GitBox]

potiuk commented on a change in pull request #15581:
URL: https://github.com/apache/airflow/pull/15581#discussion_r624657163



##
File path: airflow/providers/oracle/operators/oracle.py
##
@@ -63,4 +68,25 @@ def __init__(
 def execute(self, context) -> None:
 self.log.info('Executing: %s', self.sql)
 hook = OracleHook(oracle_conn_id=self.oracle_conn_id)
-hook.run(self.sql, autocommit=self.autocommit, 
parameters=self.parameters)
+
+def handler(cur):
+bindvars = cur.bindvars
+
+if isinstance(bindvars, list):
+bindvars = [v.getvalue() for v in bindvars]
+elif isinstance(bindvars, dict):
+bindvars = {n: v.getvalue() for (n, v) in bindvars.items()}
+else:
+raise TypeError(bindvars)
+
+return {
+"bindvars": bindvars,
+"rows": cur.fetchall(),
+}
+
+return hook.run(

Review comment:
   The problem with that is that we are releasing providers from "master"  
so if we make a "2.1" change, it will have to wait until  2.1 release to be 
merged (and will become available only in the next release of Oracle provider). 
   
   IMHO much better solution is  implementing it in a backwards-compatible way, 
 merging it now and releasing new provider that will simply use the 
functionality when Airflow 2.1 is released.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on a change in pull request #15581: Add bindvars Xcom result to Oracle operator

[GitBox]

potiuk commented on a change in pull request #15581:
URL: https://github.com/apache/airflow/pull/15581#discussion_r624657163



##
File path: airflow/providers/oracle/operators/oracle.py
##
@@ -63,4 +68,25 @@ def __init__(
 def execute(self, context) -> None:
 self.log.info('Executing: %s', self.sql)
 hook = OracleHook(oracle_conn_id=self.oracle_conn_id)
-hook.run(self.sql, autocommit=self.autocommit, 
parameters=self.parameters)
+
+def handler(cur):
+bindvars = cur.bindvars
+
+if isinstance(bindvars, list):
+bindvars = [v.getvalue() for v in bindvars]
+elif isinstance(bindvars, dict):
+bindvars = {n: v.getvalue() for (n, v) in bindvars.items()}
+else:
+raise TypeError(bindvars)
+
+return {
+"bindvars": bindvars,
+"rows": cur.fetchall(),
+}
+
+return hook.run(

Review comment:
   The problem with that is that we are releasing providers from "master"  
so if we make a "2.1" change, it will have to wait until  2.1 release to be 
merged. 
   
   IMHO much better solution is  implementing it in a backwards-compatible way, 
 merging it now and releasing new provider that will simply use the 
functionality when Airflow 2.1 is released.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on a change in pull request #15581: Add bindvars Xcom result to Oracle operator

[GitBox]

potiuk commented on a change in pull request #15581:
URL: https://github.com/apache/airflow/pull/15581#discussion_r624657163



##
File path: airflow/providers/oracle/operators/oracle.py
##
@@ -63,4 +68,25 @@ def __init__(
 def execute(self, context) -> None:
 self.log.info('Executing: %s', self.sql)
 hook = OracleHook(oracle_conn_id=self.oracle_conn_id)
-hook.run(self.sql, autocommit=self.autocommit, 
parameters=self.parameters)
+
+def handler(cur):
+bindvars = cur.bindvars
+
+if isinstance(bindvars, list):
+bindvars = [v.getvalue() for v in bindvars]
+elif isinstance(bindvars, dict):
+bindvars = {n: v.getvalue() for (n, v) in bindvars.items()}
+else:
+raise TypeError(bindvars)
+
+return {
+"bindvars": bindvars,
+"rows": cur.fetchall(),
+}
+
+return hook.run(

Review comment:
   The problem with that is that we are releasing providers from "master"  
so if we make a "2.1" change, it will have to wait until to 2.1 release to be 
merged. 
   
   IMHO much better solution is  implementing it in a backwards-compatible 
version,  merging it now and releasing new provider that will simply use the 
functionality when Airflow 2.1 is released.

##
File path: airflow/providers/oracle/operators/oracle.py
##
@@ -63,4 +68,25 @@ def __init__(
 def execute(self, context) -> None:
 self.log.info('Executing: %s', self.sql)
 hook = OracleHook(oracle_conn_id=self.oracle_conn_id)
-hook.run(self.sql, autocommit=self.autocommit, 
parameters=self.parameters)
+
+def handler(cur):
+bindvars = cur.bindvars
+
+if isinstance(bindvars, list):
+bindvars = [v.getvalue() for v in bindvars]
+elif isinstance(bindvars, dict):
+bindvars = {n: v.getvalue() for (n, v) in bindvars.items()}
+else:
+raise TypeError(bindvars)
+
+return {
+"bindvars": bindvars,
+"rows": cur.fetchall(),
+}
+
+return hook.run(

Review comment:
   The problem with that is that we are releasing providers from "master"  
so if we make a "2.1" change, it will have to wait until to 2.1 release to be 
merged. 
   
   IMHO much better solution is  implementing it in a backwards-compatible way, 
 merging it now and releasing new provider that will simply use the 
functionality when Airflow 2.1 is released.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on a change in pull request #15581: Add bindvars Xcom result to Oracle operator

[GitBox]

potiuk commented on a change in pull request #15581:
URL: https://github.com/apache/airflow/pull/15581#discussion_r624657163



##
File path: airflow/providers/oracle/operators/oracle.py
##
@@ -63,4 +68,25 @@ def __init__(
 def execute(self, context) -> None:
 self.log.info('Executing: %s', self.sql)
 hook = OracleHook(oracle_conn_id=self.oracle_conn_id)
-hook.run(self.sql, autocommit=self.autocommit, 
parameters=self.parameters)
+
+def handler(cur):
+bindvars = cur.bindvars
+
+if isinstance(bindvars, list):
+bindvars = [v.getvalue() for v in bindvars]
+elif isinstance(bindvars, dict):
+bindvars = {n: v.getvalue() for (n, v) in bindvars.items()}
+else:
+raise TypeError(bindvars)
+
+return {
+"bindvars": bindvars,
+"rows": cur.fetchall(),
+}
+
+return hook.run(

Review comment:
   The problem with that is that we are releasing providers from "master"  
so if we make a "2.1" change, it will have to wait until to 2.1 release to be 
merged. 
   
   IMHO much better solution i  implementing it in a backwards-compatible 
version,  merging it now and releasing new provider that will simply use the 
functionality when Airflow 2.1 is released.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #15036: Scheduler is skipping a day sometimes

[GitBox]

eladkal commented on issue #15036:
URL: https://github.com/apache/airflow/issues/15036#issuecomment-830765361


   @iameugenejo can you share the DAG code? we need more information here.
   If we can't reproduce it's almost impossible to find a fix.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org