[GitHub] [airflow] mik-laj commented on a change in pull request #9170: [WIP] Read only endpoint for XCom #8134

2020-06-09 Thread GitBox
mik-laj commented on a change in pull request #9170: URL: https://github.com/apache/airflow/pull/9170#discussion_r437084427 ## File path: airflow/api_connexion/schemas/xcom_schema.py ## @@ -0,0 +1,67 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more co

[GitHub] [airflow] SamWheating commented on a change in pull request #9159: Adding Google Deployment Manager Hook

2020-06-09 Thread GitBox
SamWheating commented on a change in pull request #9159: URL: https://github.com/apache/airflow/pull/9159#discussion_r436680469 ## File path: airflow/providers/google/cloud/hooks/gdm.py ## @@ -0,0 +1,103 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or

[airflow] branch v1-10-test updated (4b8fbf2 -> f21618b)

2020-06-09 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git. omit 4b8fbf2 Improved cloud tool available in the trimmed down CI container (#9167) new 5661dc9 Improved clou

[GitHub] [airflow] potiuk commented on pull request #9029: Remove Hive/Hadoop/Java dependency from unit tests

2020-06-09 Thread GitBox
potiuk commented on pull request #9029: URL: https://github.com/apache/airflow/pull/9029#issuecomment-641191536 Hey @jhtimmins -> I cherry-picked your changes to https://github.com/apache/airflow/commit/d4a00b7c69d5d1c4514bc70b2a0022f36b85a06b I'd love if you can take a look and se

[GitHub] [airflow] leahecole commented on pull request #9174: Don't use the term "whitelist" - language matters

2020-06-09 Thread GitBox
leahecole commented on pull request #9174: URL: https://github.com/apache/airflow/pull/9174#issuecomment-640865479 I was just going to make this PR - so glad you beat me to it!! This is an automated message from the Apache G

[GitHub] [airflow] crhyatt opened a new pull request #8500: [Airflow-3391] Upgrade pendulum to latest major version

2020-06-09 Thread GitBox
crhyatt opened a new pull request #8500: URL: https://github.com/apache/airflow/pull/8500 --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for changes (not needed for documentation chan

[GitHub] [airflow] driquelme opened a new issue #9186: Add SRV DNS record support to http connections

2020-06-09 Thread GitBox
driquelme opened a new issue #9186: URL: https://github.com/apache/airflow/issues/9186 **Description** SRV records are DNS records that resolve to a hostname and a port. Being able to specify http connections with SRV resolution support is useful in contexts where external s

[GitHub] [airflow] mik-laj opened a new pull request #9194: Remove httplib2 from Google requirements

2020-06-09 Thread GitBox
mik-laj opened a new pull request #9194: URL: https://github.com/apache/airflow/pull/9194 We don't use this library directly, so we can rely on the requirements of other libraries. On the other hand, this requirement conflicts with Apache Beam v2.19 ('httplib2>=0.8,<=0.12.0',). It is l

[GitHub] [airflow] languitar commented on issue #8629: DockerOperator cannot pull image when docker client uses json-file log driver

2020-06-09 Thread GitBox
languitar commented on issue #8629: URL: https://github.com/apache/airflow/issues/8629#issuecomment-641282826 Does anyone know when this will end up in a stable release? This is an automated message from the Apache Git Servic

[jira] [Commented] (AIRFLOW-3973) `airflow initdb` logs errors when `Variable` is used in DAGs and Postgres is used for the internal database

2020-06-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17129581#comment-17129581 ] ASF GitHub Bot commented on AIRFLOW-3973: - Fokko commented on pull request #479

[GitHub] [airflow] Fokko commented on pull request #4797: [AIRFLOW-3973] Run each Alembic migration in separate transaction

2020-06-09 Thread GitBox
Fokko commented on pull request #4797: URL: https://github.com/apache/airflow/pull/4797#issuecomment-640588665 @eeshugerman please make a PR to the `v1-10-test` branch and I'll merge it. This is an automated message from the

[GitHub] [airflow] ashb merged pull request #9180: Don't use the `|safe` filter in code, it's risky

2020-06-09 Thread GitBox
ashb merged pull request #9180: URL: https://github.com/apache/airflow/pull/9180 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] jimmajure opened a new issue #9181: Make KubernetesExecutor PersistentVolumeClaim mounts more flexible.

2020-06-09 Thread GitBox
jimmajure opened a new issue #9181: URL: https://github.com/apache/airflow/issues/9181 **Description** In the KubernetesExecutor configuration, allow the DAGs, logs, plugins folders to be supported by a single PersistentVolumeClaim. Currently the configuration forces a separate

[jira] [Commented] (AIRFLOW-6994) SparkSubmitOperator re launches spark driver even when original driver still running

2020-06-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17129576#comment-17129576 ] ASF GitHub Bot commented on AIRFLOW-6994: - stale[bot] commented on pull request

[GitHub] [airflow] stale[bot] commented on pull request #7637: [AIRFLOW-6994] SparkSubmitOperator re-launches spark driver even when original driver still running

2020-06-09 Thread GitBox
stale[bot] commented on pull request #7637: URL: https://github.com/apache/airflow/pull/7637#issuecomment-64410 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[GitHub] [airflow] dimon222 commented on issue #9148: Retries are causing constraint violation (Airflow 1.10.10)

2020-06-09 Thread GitBox
dimon222 commented on issue #9148: URL: https://github.com/apache/airflow/issues/9148#issuecomment-641335335 > @dimon222 Can you try setting `max_num_rendered_ti_fields_per_task` under `[core]` section to 1 and see it that helps for now ? Not helping, receiving same error. I'm afraid

[GitHub] [airflow] potiuk merged pull request #9188: Fixes failure of the build scripts when remote repo does not exist

2020-06-09 Thread GitBox
potiuk merged pull request #9188: URL: https://github.com/apache/airflow/pull/9188 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[GitHub] [airflow] mik-laj merged pull request #9177: Add PR/issue note in Contribution Workflow Example

2020-06-09 Thread GitBox
mik-laj merged pull request #9177: URL: https://github.com/apache/airflow/pull/9177 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] mik-laj commented on a change in pull request #8575: [AIRFLOW-6981] Move Google Cloud Build from Discovery API to Python Library

2020-06-09 Thread GitBox
mik-laj commented on a change in pull request #8575: URL: https://github.com/apache/airflow/pull/8575#discussion_r437198960 ## File path: airflow/providers/google/cloud/hooks/cloud_build.py ## @@ -47,90 +47,528 @@ class CloudBuildHook(GoogleBaseHook): :type delegate_to: st

[GitHub] [airflow] turbaszek commented on a change in pull request #8805: Resolve upstream tasks when template field is XComArg

2020-06-09 Thread GitBox
turbaszek commented on a change in pull request #8805: URL: https://github.com/apache/airflow/pull/8805#discussion_r437473652 ## File path: airflow/serialization/serialized_objects.py ## @@ -297,7 +297,7 @@ class SerializedBaseOperator(BaseOperator, BaseSerialization): _d

[jira] [Commented] (AIRFLOW-6981) Move Google Cloud Build from Discovery API to Python Library

2020-06-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17129572#comment-17129572 ] ASF GitHub Bot commented on AIRFLOW-6981: - mik-laj commented on a change in pul

[GitHub] [airflow] OmairK commented on a change in pull request #9097: [WIP] Schema and CRUD endpoints Pool

2020-06-09 Thread GitBox
OmairK commented on a change in pull request #9097: URL: https://github.com/apache/airflow/pull/9097#discussion_r437471258 ## File path: airflow/api_connexion/endpoints/pool_endpoint.py ## @@ -26,18 +30,33 @@ def delete_pool(): raise NotImplementedError("Not implemented ye

[GitHub] [airflow] mik-laj commented on a change in pull request #9097: [WIP] Schema and CRUD endpoints Pool

2020-06-09 Thread GitBox
mik-laj commented on a change in pull request #9097: URL: https://github.com/apache/airflow/pull/9097#discussion_r437017632 ## File path: airflow/api_connexion/endpoints/pool_endpoint.py ## @@ -26,18 +30,33 @@ def delete_pool(): raise NotImplementedError("Not implemented y

[GitHub] [airflow] JeffryMAC commented on a change in pull request #9179: [Feature] Add @minutely to schedule_interval

2020-06-09 Thread GitBox
JeffryMAC commented on a change in pull request #9179: URL: https://github.com/apache/airflow/pull/9179#discussion_r436784765 ## File path: airflow/utils/dates.py ## @@ -24,6 +24,7 @@ from airflow.utils import timezone cron_presets = { +'@minutely': '* * * * *', Review

[jira] [Commented] (AIRFLOW-7014) Operator for Apache Kylin

2020-06-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-7014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17129559#comment-17129559 ] ASF GitHub Bot commented on AIRFLOW-7014: - liuyonghengheng commented on a chang

[GitHub] [airflow] liuyonghengheng commented on a change in pull request #9149: [AIRFLOW-7014] Add Apache Kylin operator

2020-06-09 Thread GitBox
liuyonghengheng commented on a change in pull request #9149: URL: https://github.com/apache/airflow/pull/9149#discussion_r437343515 ## File path: airflow/providers/apache/kylin/operators/kylin.py ## @@ -0,0 +1,166 @@ +# +# Licensed to the Apache Software Foundation (ASF) under

[GitHub] [airflow] turbaszek merged pull request #9087: Query TaskReschedule only if task is UP_FOR_RESCHEDULE

2020-06-09 Thread GitBox
turbaszek merged pull request #9087: URL: https://github.com/apache/airflow/pull/9087 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[GitHub] [airflow] the-horhe opened a new issue #9190: DockerOperator loosing some initial container output

2020-06-09 Thread GitBox
the-horhe opened a new issue #9190: URL: https://github.com/apache/airflow/issues/9190 **Apache Airflow version**: 1.10.10 **Environment**: official docker image apache/airflow:1.10.10, docker api version: 1.40 **What happened**: Don't see any container output in task logs

[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9153: [WIP] add readonly endpoints for dagruns

2020-06-09 Thread GitBox
ephraimbuddy commented on a change in pull request #9153: URL: https://github.com/apache/airflow/pull/9153#discussion_r436643921 ## File path: tests/api_connexion/endpoints/test_dag_run_endpoint.py ## @@ -29,26 +35,471 @@ def setUpClass(cls) -> None: def setUp(self) -> N

[GitHub] [airflow] ashb opened a new pull request #9191: Update AWS connection example to show how to set from env var

2020-06-09 Thread GitBox
ashb opened a new pull request #9191: URL: https://github.com/apache/airflow/pull/9191 The trailing `@` wasn't obvious/documented anywhere (and took me some trial and error to work out) so to save time for the next person let's add it to the docs --- Make sure to mark the

[GitHub] [airflow] turbaszek commented on a change in pull request #9054: Add MySqlToS3Operator

2020-06-09 Thread GitBox
turbaszek commented on a change in pull request #9054: URL: https://github.com/apache/airflow/pull/9054#discussion_r437208286 ## File path: airflow/providers/amazon/aws/operators/mysql_to_s3.py ## @@ -0,0 +1,126 @@ +# +# Licensed to the Apache Software Foundation (ASF) under on

[GitHub] [airflow] mik-laj merged pull request #9169: Allows using private endpoints in GKEStartPodOperator

2020-06-09 Thread GitBox
mik-laj merged pull request #9169: URL: https://github.com/apache/airflow/pull/9169 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] ashb commented on pull request #9162: Improve production image iteration speed

2020-06-09 Thread GitBox
ashb commented on pull request #9162: URL: https://github.com/apache/airflow/pull/9162#issuecomment-640741641 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [airflow] potiuk commented on pull request #4797: [AIRFLOW-3973] Run each Alembic migration in separate transaction

2020-06-09 Thread GitBox
potiuk commented on pull request #4797: URL: https://github.com/apache/airflow/pull/4797#issuecomment-640687352 Just a small update: I think merge PR should go to v1-10-stable - in test we are usually cherry-pick stuff (I have 23 cherry-picked commits to push soon) so better to keep it thi

[GitHub] [airflow] feluelle commented on a change in pull request #9195: Update pre-commit-hooks repo version

2020-06-09 Thread GitBox
feluelle commented on a change in pull request #9195: URL: https://github.com/apache/airflow/pull/9195#discussion_r437508400 ## File path: .pre-commit-config.yaml ## @@ -159,15 +159,23 @@ metastore_browser/templates/.*\\.html$|.*\\.jinja2" hooks: - id: rst-backticks

[GitHub] [airflow] ashb opened a new pull request #9180: Don't use the `|safe` filter in code, it's risky

2020-06-09 Thread GitBox
ashb opened a new pull request #9180: URL: https://github.com/apache/airflow/pull/9180 Most things already use the `Markup` class to correctly escape problem areas, this commit just fixes the last instances so that we can assert that `|safe` is never used. --- Make sure t

[jira] [Commented] (AIRFLOW-3973) `airflow initdb` logs errors when `Variable` is used in DAGs and Postgres is used for the internal database

2020-06-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17129539#comment-17129539 ] ASF GitHub Bot commented on AIRFLOW-3973: - potiuk commented on pull request #47

[GitHub] [airflow] potiuk opened a new pull request #9188: Fixes failure of the build scripts when remote repo does not exist

2020-06-09 Thread GitBox
potiuk opened a new pull request #9188: URL: https://github.com/apache/airflow/pull/9188 --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for changes (not needed for documentation chang

[GitHub] [airflow] ashb commented on pull request #9187: Set conn_type as not-nullable

2020-06-09 Thread GitBox
ashb commented on pull request #9187: URL: https://github.com/apache/airflow/pull/9187#issuecomment-641273689 > I think we should also set conn_id as non-nullable Yes, agreed. Let's do that in one migration. This is an

[GitHub] [airflow] mik-laj commented on a change in pull request #9180: Don't use the `|safe` filter in code, it's risky

2020-06-09 Thread GitBox
mik-laj commented on a change in pull request #9180: URL: https://github.com/apache/airflow/pull/9180#discussion_r436887040 ## File path: airflow/www/views.py ## @@ -631,11 +630,9 @@ def rendered(self): for template_field in task.template_fields: content =

[GitHub] [airflow] edejong commented on issue #8903: BigQueryHook refactor + deterministic BQ Job ID

2020-06-09 Thread GitBox
edejong commented on issue #8903: URL: https://github.com/apache/airflow/issues/8903#issuecomment-641387021 I should really check GitHub more often, I only saw the notification now. Let me know if I understand the question correctly: should the BigQueryHook's interface rely on classe

[GitHub] [airflow] mjpieters opened a new issue #9185: Include plugin information in the version page

2020-06-09 Thread GitBox
mjpieters opened a new issue #9185: URL: https://github.com/apache/airflow/issues/9185 **Description** Please include plug-in information (see below) in the Airflow "About -> Version" information page in the web interface. **Use case / motivation** Plug-ins have a lot o

[GitHub] [airflow] randr97 commented on a change in pull request #9170: [WIP] Read only endpoint for XCom #8134

2020-06-09 Thread GitBox
randr97 commented on a change in pull request #9170: URL: https://github.com/apache/airflow/pull/9170#discussion_r437123042 ## File path: airflow/api_connexion/schemas/xcom_schema.py ## @@ -0,0 +1,67 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more co

[GitHub] [airflow] crhyatt opened a new pull request #9184: Upgrade pendulum to latest major version ~2.0

2020-06-09 Thread GitBox
crhyatt opened a new pull request #9184: URL: https://github.com/apache/airflow/pull/9184 --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for changes (not needed for documentation chan

[GitHub] [airflow] potiuk commented on pull request #9162: Improve production image iteration speed

2020-06-09 Thread GitBox
potiuk commented on pull request #9162: URL: https://github.com/apache/airflow/pull/9162#issuecomment-640799787 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

[GitHub] [airflow] turbaszek commented on a change in pull request #8962: [AIRFLOW-8057] [AIP-31] Add @task decorator

2020-06-09 Thread GitBox
turbaszek commented on a change in pull request #8962: URL: https://github.com/apache/airflow/pull/8962#discussion_r437204483 ## File path: airflow/operators/python.py ## @@ -145,6 +149,131 @@ def execute_callable(self): return self.python_callable(*self.op_args, **sel

[GitHub] [airflow] chamcca commented on pull request #8749: add AWS StepFunctions integrations to the aws provider

2020-06-09 Thread GitBox
chamcca commented on pull request #8749: URL: https://github.com/apache/airflow/pull/8749#issuecomment-640603607 @feluelle any additional comments/recommendations for this PR? This is an automated message from the Apache Git

[GitHub] [airflow] mik-laj commented on pull request #9067: [AIRFLOW-8608] Make conn_id unique in Connections table

2020-06-09 Thread GitBox
mik-laj commented on pull request #9067: URL: https://github.com/apache/airflow/pull/9067#issuecomment-640912099 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[airflow] branch master updated (b762763 -> 7ad827f)

2020-06-09 Thread kamilbregula
This is an automated email from the ASF dual-hosted git repository. kamilbregula pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from b762763 Query TaskReschedule only if task is UP_FOR_RESCHEDULE (#9087) add 7ad827f Set conn_type as not

[GitHub] [airflow] turbaszek commented on pull request #8805: Resolve upstream tasks when template field is XComArg

2020-06-09 Thread GitBox
turbaszek commented on pull request #8805: URL: https://github.com/apache/airflow/pull/8805#issuecomment-641253096 Meh. The serialization doesn't work because `__call__` has other signature than `BaseOperator.__init__`. I think this could be easily solved by doing `super().__call__()` inst

[GitHub] [airflow] ashb merged pull request #9178: Validate only task commands are run by executors

2020-06-09 Thread GitBox
ashb merged pull request #9178: URL: https://github.com/apache/airflow/pull/9178 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] JavierLopezT commented on a change in pull request #9054: Add MySqlToS3Operator

2020-06-09 Thread GitBox
JavierLopezT commented on a change in pull request #9054: URL: https://github.com/apache/airflow/pull/9054#discussion_r437244791 ## File path: airflow/providers/amazon/aws/operators/mysql_to_s3.py ## @@ -0,0 +1,126 @@ +# +# Licensed to the Apache Software Foundation (ASF) under

[GitHub] [airflow] ChristianYeah commented on issue #9131: max_active_runs failed in production env with multiple workers

2020-06-09 Thread GitBox
ChristianYeah commented on issue #9131: URL: https://github.com/apache/airflow/issues/9131#issuecomment-641222683 I’ll duplicate the production env to the test later later , see if it will be reproduced.  发自我的iPhone -- Original --

[GitHub] [airflow] mik-laj commented on pull request #9124: [AIRFLOW-9099] Merging multiple sql operators

2020-06-09 Thread GitBox
mik-laj commented on pull request #9124: URL: https://github.com/apache/airflow/pull/9124#issuecomment-641106425 Can you add these cases to the list of changed classes? The file name suggests something different, but that's a different problem. Here we have all the name changes that we the

[GitHub] [airflow] mik-laj commented on a change in pull request #9187: Set conn_type as not-nullable

2020-06-09 Thread GitBox
mik-laj commented on a change in pull request #9187: URL: https://github.com/apache/airflow/pull/9187#discussion_r437254077 ## File path: airflow/migrations/versions/8f966b9c467a_set_conn_type_as_non_nullable.py ## @@ -0,0 +1,47 @@ +# +# Licensed to the Apache Software Foundat

[GitHub] [airflow] ashb opened a new pull request #9178: Validate only task commands are run by executors

2020-06-09 Thread GitBox
ashb opened a new pull request #9178: URL: https://github.com/apache/airflow/pull/9178 --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for changes (not needed for documentation changes

[GitHub] [airflow] turbaszek commented on pull request #8954: Wait for pipeline state in Data Fusion operators

2020-06-09 Thread GitBox
turbaszek commented on pull request #8954: URL: https://github.com/apache/airflow/pull/8954#issuecomment-641147473 Hi @jaketf @sreevatsanraman what should we do to move this forward? This is an automated message from the Apac

[GitHub] [airflow] boring-cyborg[bot] commented on issue #9185: Include plugin information in the version page

2020-06-09 Thread GitBox
boring-cyborg[bot] commented on issue #9185: URL: https://github.com/apache/airflow/issues/9185#issuecomment-640820162 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated

[GitHub] [airflow] ashb merged pull request #9191: Update AWS connection example to show how to set from env var

2020-06-09 Thread GitBox
ashb merged pull request #9191: URL: https://github.com/apache/airflow/pull/9191 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] nullhack opened a new pull request #9179: Feature - add @minutely to schedule_interval

2020-06-09 Thread GitBox
nullhack opened a new pull request #9179: URL: https://github.com/apache/airflow/pull/9179 **Description** The only reason I can think to not include `minutely` on `schedule_interval`, is to keep consistency against `cron`. As we can see, all other relevant shortcuts are already [t

[GitHub] [airflow] mik-laj commented on a change in pull request #9179: [Feature] Add @minutely to schedule_interval

2020-06-09 Thread GitBox
mik-laj commented on a change in pull request #9179: URL: https://github.com/apache/airflow/pull/9179#discussion_r436674133 ## File path: airflow/utils/dates.py ## @@ -24,6 +24,7 @@ from airflow.utils import timezone cron_presets = { +'@minutely': '* * * * *', Review c

[GitHub] [airflow] mik-laj opened a new pull request #9187: Set conn_type as not-nullable

2020-06-09 Thread GitBox
mik-laj opened a new pull request #9187: URL: https://github.com/apache/airflow/pull/9187 The `conn_type` column in the `connection` table must contain content. Previously, this rule was enforced by application logic, but was not enforced by the database schema. Before: ```

[GitHub] [airflow] nullhack commented on a change in pull request #9179: [Feature] Add @minutely to schedule_interval

2020-06-09 Thread GitBox
nullhack commented on a change in pull request #9179: URL: https://github.com/apache/airflow/pull/9179#discussion_r436736580 ## File path: airflow/utils/dates.py ## @@ -24,6 +24,7 @@ from airflow.utils import timezone cron_presets = { +'@minutely': '* * * * *', Review

[GitHub] [airflow] ashb commented on a change in pull request #9187: Set conn_type as not-nullable

2020-06-09 Thread GitBox
ashb commented on a change in pull request #9187: URL: https://github.com/apache/airflow/pull/9187#discussion_r437393065 ## File path: airflow/migrations/versions/8f966b9c467a_set_conn_type_as_non_nullable.py ## @@ -0,0 +1,47 @@ +# +# Licensed to the Apache Software Foundation

[GitHub] [airflow] nullhack commented on pull request #9179: Feature - add @minutely to schedule_interval

2020-06-09 Thread GitBox
nullhack commented on pull request #9179: URL: https://github.com/apache/airflow/pull/9179#issuecomment-640585047 @mik-laj sorry, don't know any other maintainers yet to ask for review XD may I ask you to review (or tag someone to)? It's a small change, but my colleagues liked the id

[jira] [Commented] (AIRFLOW-3973) `airflow initdb` logs errors when `Variable` is used in DAGs and Postgres is used for the internal database

2020-06-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17129502#comment-17129502 ] ASF GitHub Bot commented on AIRFLOW-3973: - eeshugerman commented on pull reques

[jira] [Commented] (AIRFLOW-6994) SparkSubmitOperator re launches spark driver even when original driver still running

2020-06-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17129500#comment-17129500 ] ASF GitHub Bot commented on AIRFLOW-6994: - tooptoop4 commented on pull request

[GitHub] [airflow] eeshugerman edited a comment on pull request #4797: [AIRFLOW-3973] Run each Alembic migration in separate transaction

2020-06-09 Thread GitBox
eeshugerman edited a comment on pull request #4797: URL: https://github.com/apache/airflow/pull/4797#issuecomment-640745946 A cherry-pick/rebase was required to get a clean PR to `v1-10-stable` or `v1-10-test`, so wasn't sure which to do? I opened a PR for each, feel free to close one or t

[GitHub] [airflow] tooptoop4 commented on pull request #7637: [AIRFLOW-6994] SparkSubmitOperator re-launches spark driver even when original driver still running

2020-06-09 Thread GitBox
tooptoop4 commented on pull request #7637: URL: https://github.com/apache/airflow/pull/7637#issuecomment-641116742 crispy This is an automated message from the Apache Git Service. To respond to the message, please log on to G

[jira] [Commented] (AIRFLOW-3973) `airflow initdb` logs errors when `Variable` is used in DAGs and Postgres is used for the internal database

2020-06-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17129498#comment-17129498 ] ASF GitHub Bot commented on AIRFLOW-3973: - eeshugerman edited a comment on pull

[GitHub] [airflow] eeshugerman commented on pull request #4797: [AIRFLOW-3973] Run each Alembic migration in separate transaction

2020-06-09 Thread GitBox
eeshugerman commented on pull request #4797: URL: https://github.com/apache/airflow/pull/4797#issuecomment-640745946 A cherry-pick/rebase was required to get a clean PR to `v1-10-stable` or `v1-10-test`, so wasn't sure which to do? I opened a PR for each, feel free to close one or the othe

[GitHub] [airflow] nullhack edited a comment on pull request #9165: Fix DockerOperator xcom

2020-06-09 Thread GitBox
nullhack edited a comment on pull request #9165: URL: https://github.com/apache/airflow/pull/9165#issuecomment-640226947 @mik-laj Thank you for the suggestions. I wrote some example on the issue Itself (https://github.com/apache/airflow/issues/9164) if you need It to test now. I'll

[GitHub] [airflow] feluelle opened a new pull request #9195: Update pre-commit-hooks repo version

2020-06-09 Thread GitBox
feluelle opened a new pull request #9195: URL: https://github.com/apache/airflow/pull/9195 - use official isort pre-commit-hook - use official yamllint pre-commit-hook --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context o

[GitHub] [airflow] turbaszek removed a comment on pull request #8805: Resolve upstream tasks when template field is XComArg

2020-06-09 Thread GitBox
turbaszek removed a comment on pull request #8805: URL: https://github.com/apache/airflow/pull/8805#issuecomment-641253096 Meh. The serialization doesn't work because `__call__` has other signature than `BaseOperator.__init__`. I think this could be easily solved by doing `super().__call__

[GitHub] [airflow] ash commented on pull request #9162: Improve production image iteration speed

2020-06-09 Thread GitBox
ash commented on pull request #9162: URL: https://github.com/apache/airflow/pull/9162#issuecomment-641062231 Guys. This is very annoying to get email notifications when you mistype the name of one of you. This is an automate

[GitHub] [airflow] boring-cyborg[bot] commented on issue #9190: DockerOperator loosing some initial container output

2020-06-09 Thread GitBox
boring-cyborg[bot] commented on issue #9190: URL: https://github.com/apache/airflow/issues/9190#issuecomment-641100635 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated

[GitHub] [airflow] potiuk merged pull request #9167: Improved cloud tool available in the trimmed down CI container

2020-06-09 Thread GitBox
potiuk merged pull request #9167: URL: https://github.com/apache/airflow/pull/9167 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[GitHub] [airflow] ashb commented on pull request #9170: [WIP] Read only endpoint for XCom #8134

2020-06-09 Thread GitBox
ashb commented on pull request #9170: URL: https://github.com/apache/airflow/pull/9170#issuecomment-641274716 @mik-laj Are you happy with https://github.com/apache/airflow/pull/9170/commits/c973a0710a6785b043d1d20ee9e09eb96cecbc86? -

[GitHub] [airflow] crhyatt closed pull request #8500: [Airflow-3391] Upgrade pendulum to latest major version

2020-06-09 Thread GitBox
crhyatt closed pull request #8500: URL: https://github.com/apache/airflow/pull/8500 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

[GitHub] [airflow] madison-ookla commented on pull request #9174: Don't use the term "whitelist" - language matters

2020-06-09 Thread GitBox
madison-ookla commented on pull request #9174: URL: https://github.com/apache/airflow/pull/9174#issuecomment-640719783 Thanks Ash :) This is great to see! This is an automated message from the Apache Git Service. To respond t

[GitHub] [airflow] Acehaidrey commented on pull request #8598: make hive macros return string type vs bytes

2020-06-09 Thread GitBox
Acehaidrey commented on pull request #8598: URL: https://github.com/apache/airflow/pull/8598#issuecomment-640870324 This change is actually done using branch v1.10-stable. So it works as is here @ashb . Good question. So no concern there either

[jira] [Commented] (AIRFLOW-3973) `airflow initdb` logs errors when `Variable` is used in DAGs and Postgres is used for the internal database

2020-06-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17129483#comment-17129483 ] ASF GitHub Bot commented on AIRFLOW-3973: - eeshugerman opened a new pull reques

[GitHub] [airflow] turbaszek commented on a change in pull request #9187: Set conn_type as not-nullable

2020-06-09 Thread GitBox
turbaszek commented on a change in pull request #9187: URL: https://github.com/apache/airflow/pull/9187#discussion_r437240866 ## File path: airflow/migrations/versions/8f966b9c467a_set_conn_type_as_non_nullable.py ## @@ -0,0 +1,47 @@ +# +# Licensed to the Apache Software Found

[GitHub] [airflow] eeshugerman opened a new pull request #9183: [AIRFLOW-3973] Commit after each alembic migration

2020-06-09 Thread GitBox
eeshugerman opened a new pull request #9183: URL: https://github.com/apache/airflow/pull/9183 If `Variable`s are used in DAGs, and Postgres is used for the internal database, a fresh `$ airflow initdb` (or `$ airflow resetdb`) spams the logs with error messages (but does not fail).

[GitHub] [airflow] zikun commented on pull request #9189: Support additional apt dependencies

2020-06-09 Thread GitBox
zikun commented on pull request #9189: URL: https://github.com/apache/airflow/pull/9189#issuecomment-641077907 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

[GitHub] [airflow] feluelle commented on a change in pull request #8877: Add S3ToRedshift example dag and system test

2020-06-09 Thread GitBox
feluelle commented on a change in pull request #8877: URL: https://github.com/apache/airflow/pull/8877#discussion_r437309397 ## File path: tests/providers/amazon/aws/operators/test_s3_to_redshift_system.py ## @@ -0,0 +1,44 @@ +# +# Licensed to the Apache Software Foundation (AS

[GitHub] [airflow] potiuk commented on pull request #8974: Add VaultHook to hashicorp provider

2020-06-09 Thread GitBox
potiuk commented on pull request #8974: URL: https://github.com/apache/airflow/pull/8974#issuecomment-641179952 @kaxil :) ? This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [airflow] dinigo opened a new issue #9193: Celery 4.4.4 (known bug) lacks "future" module thus breaks airflow

2020-06-09 Thread GitBox
dinigo opened a new issue #9193: URL: https://github.com/apache/airflow/issues/9193 Hello. We are experiencing some issues in local development due to a celery using a module that it doesn't install anymore. This happened when upping dependency version of celery 4.4.3 to 4.4.4

[GitHub] [airflow] mik-laj commented on pull request #8575: [AIRFLOW-6981] Move Google Cloud Build from Discovery API to Python Library

2020-06-09 Thread GitBox
mik-laj commented on pull request #8575: URL: https://github.com/apache/airflow/pull/8575#issuecomment-641094098 I still have the following error: ``` /airflow/logs/previous_runs/2020-06-09_07_26_35/example_gcp_cloud_build/create_build_from_repo/2020-06-08T00:00:00+00:00/1.log ```

[GitHub] [airflow] boring-cyborg[bot] commented on issue #9186: Add SRV DNS record support to http connections

2020-06-09 Thread GitBox
boring-cyborg[bot] commented on issue #9186: URL: https://github.com/apache/airflow/issues/9186#issuecomment-640844130 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated

[jira] [Commented] (AIRFLOW-6981) Move Google Cloud Build from Discovery API to Python Library

2020-06-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17129466#comment-17129466 ] ASF GitHub Bot commented on AIRFLOW-6981: - mik-laj commented on pull request #8

[GitHub] [airflow] ashb commented on a change in pull request #9162: Improve production image iteration speed

2020-06-09 Thread GitBox
ashb commented on a change in pull request #9162: URL: https://github.com/apache/airflow/pull/9162#discussion_r436843199 ## File path: Dockerfile ## @@ -153,6 +153,23 @@ ENV PIP_VERSION=${PIP_VERSION} RUN pip install --upgrade pip==${PIP_VERSION} +ARG AIRFLOW_REPO=apache/a

[GitHub] [airflow] svetli-n commented on issue #9131: max_active_runs failed in production env with multiple workers

2020-06-09 Thread GitBox
svetli-n commented on issue #9131: URL: https://github.com/apache/airflow/issues/9131#issuecomment-641221074 @ChristianYeah Thanks. I can't reproduce it though. Feel free to share full dag run file including deafult settings and tasks, as well as how you trigger the dag run.

[GitHub] [airflow] nullhack edited a comment on issue #9190: DockerOperator loosing some initial container output

2020-06-09 Thread GitBox
nullhack edited a comment on issue #9190: URL: https://github.com/apache/airflow/issues/9190#issuecomment-641245736 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

[GitHub] [airflow] turbaszek commented on a change in pull request #8877: Add S3ToRedshift example dag and system test

2020-06-09 Thread GitBox
turbaszek commented on a change in pull request #8877: URL: https://github.com/apache/airflow/pull/8877#discussion_r437211600 ## File path: airflow/providers/amazon/aws/example_dags/example_s3_to_redshift.py ## @@ -41,13 +42,26 @@ schedule_interval=None, tags=['exampl

[GitHub] [airflow] nullhack commented on issue #9190: DockerOperator loosing some initial container output

2020-06-09 Thread GitBox
nullhack commented on issue #9190: URL: https://github.com/apache/airflow/issues/9190#issuecomment-641245736 This can be related to a recently merged [PR](https://github.com/apache/airflow/pull/9173). I tried to run on the docker image `apache/airflow:master-ci` and the output was a

[GitHub] [airflow] crhyatt commented on pull request #8500: [Airflow-3391] Upgrade pendulum to latest major version

2020-06-09 Thread GitBox
crhyatt commented on pull request #8500: URL: https://github.com/apache/airflow/pull/8500#issuecomment-640751169 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [airflow] nehiljain commented on issue #9015: Update to Pendulum 2.1.0

2020-06-09 Thread GitBox
nehiljain commented on issue #9015: URL: https://github.com/apache/airflow/issues/9015#issuecomment-640791485 +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[jira] [Commented] (AIRFLOW-3973) `airflow initdb` logs errors when `Variable` is used in DAGs and Postgres is used for the internal database

2020-06-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17129452#comment-17129452 ] ASF GitHub Bot commented on AIRFLOW-3973: - eeshugerman opened a new pull reques

[GitHub] [airflow] tooptoop4 commented on issue #9166: DAG run fails when task instance is still running

2020-06-09 Thread GitBox
tooptoop4 commented on issue #9166: URL: https://github.com/apache/airflow/issues/9166#issuecomment-640915884 https://github.com/apache/airflow/pull/8082 might help This is an automated message from the Apache Git Service. To

<    1   2   3   >