[GitHub] [airflow] turbaszek commented on a change in pull request #9037: Create guide for Dataproc Operators

2020-05-27 Thread GitBox
turbaszek commented on a change in pull request #9037: URL: https://github.com/apache/airflow/pull/9037#discussion_r431308938 ## File path: docs/howto/operator/gcp/dataproc.rst ## @@ -0,0 +1,186 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more con

[GitHub] [airflow] turbaszek commented on a change in pull request #9037: Create guide for Dataproc Operators

2020-05-27 Thread GitBox
turbaszek commented on a change in pull request #9037: URL: https://github.com/apache/airflow/pull/9037#discussion_r431309657 ## File path: docs/howto/operator/gcp/dataproc.rst ## @@ -0,0 +1,186 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more con

[GitHub] [airflow] turbaszek commented on a change in pull request #9037: Create guide for Dataproc Operators

2020-05-27 Thread GitBox
turbaszek commented on a change in pull request #9037: URL: https://github.com/apache/airflow/pull/9037#discussion_r431310411 ## File path: docs/howto/operator/gcp/dataproc.rst ## @@ -0,0 +1,186 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more con

[GitHub] [airflow] turbaszek commented on a change in pull request #8227: Add run_type to DagRun

2020-05-27 Thread GitBox
turbaszek commented on a change in pull request #8227: URL: https://github.com/apache/airflow/pull/8227#discussion_r431311590 ## File path: airflow/models/dag.py ## @@ -1468,14 +1472,26 @@ def create_dagrun(self, :param session: database session :type session:

[GitHub] [airflow] turbaszek commented on issue #8903: BigQueryHook refactor + deterministic BQ Job ID

2020-05-27 Thread GitBox
turbaszek commented on issue #8903: URL: https://github.com/apache/airflow/issues/8903#issuecomment-634816078 Summoning @edejong to hear his opinion :) This is an automated message from the Apache Git Service. To respond to t

[GitHub] [airflow] kaxil commented on a change in pull request #8227: Add run_type to DagRun

2020-05-27 Thread GitBox
kaxil commented on a change in pull request #8227: URL: https://github.com/apache/airflow/pull/8227#discussion_r431313499 ## File path: airflow/models/dag.py ## @@ -1468,14 +1472,26 @@ def create_dagrun(self, :param session: database session :type session: sql

[GitHub] [airflow] turbaszek commented on a change in pull request #8962: [AIRFLOW-8057] [AIP-31] Add @task decorator

2020-05-27 Thread GitBox
turbaszek commented on a change in pull request #8962: URL: https://github.com/apache/airflow/pull/8962#discussion_r431318258 ## File path: docs/concepts.rst ## @@ -116,6 +116,47 @@ DAGs can be used as context managers to automatically assign new operators to th op.dag

[GitHub] [airflow] potiuk commented on pull request #8974: Vault has now VaultHook not only SecretBackend

2020-05-27 Thread GitBox
potiuk commented on pull request #8974: URL: https://github.com/apache/airflow/pull/8974#issuecomment-634822700 Hey @kaxil -> I split the tests into separate client_vault (more detailed) and hook_vault tests (focused on "green-path" + reading parameters from connections). I think this way

[GitHub] [airflow] potiuk opened a new pull request #9039: Updated missing parameters for docker image building

2020-05-27 Thread GitBox
potiuk opened a new pull request #9039: URL: https://github.com/apache/airflow/pull/9039 --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for changes (not needed for documentation chang

[GitHub] [airflow] potiuk commented on pull request #9039: Updated missing parameters for docker image building

2020-05-27 Thread GitBox
potiuk commented on pull request #9039: URL: https://github.com/apache/airflow/pull/9039#issuecomment-634834608 cc @wittfabian This is an automated message from the Apache Git Service. To respond to the message, please log o

[GitHub] [airflow] alexbegg commented on issue #8712: docker image does not include 'pymssql'

2020-05-27 Thread GitBox
alexbegg commented on issue #8712: URL: https://github.com/apache/airflow/issues/8712#issuecomment-634841172 pymssql is installed with the airflow extra "mssql". It is not pre-installed with the docker image apache/airflow:1.10.10. I am currently installing it as a separate pip install of

[GitHub] [airflow] YingboWang commented on pull request #5499: [AIRFLOW-3964][AIP-17] Build smart sensor

2020-05-27 Thread GitBox
YingboWang commented on pull request #5499: URL: https://github.com/apache/airflow/pull/5499#issuecomment-634854013 Thanks @KevinYang21 for the comment. This PR description was updated with a high level architecture diagram and some charts for smart sensor impact after being deployed.

[jira] [Commented] (AIRFLOW-3964) Consolidate and de-duplicate sensor tasks

2020-05-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117987#comment-17117987 ] ASF GitHub Bot commented on AIRFLOW-3964: - YingboWang commented on pull request

[GitHub] [airflow] potiuk commented on pull request #8976: You can push with Breeze as separate command and to cache

2020-05-27 Thread GitBox
potiuk commented on pull request #8976: URL: https://github.com/apache/airflow/pull/8976#issuecomment-634859654 @ashb @dimberman @kaxil @mik-laj -> that one wil make it easier to iterate on CI/Prod images - as I will be switching back to it, It would be useful to merge it. --

[GitHub] [airflow] potiuk closed issue #8712: docker image does not include 'pymssql'

2020-05-27 Thread GitBox
potiuk closed issue #8712: URL: https://github.com/apache/airflow/issues/8712 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

[GitHub] [airflow] potiuk commented on issue #8712: docker image does not include 'pymssql'

2020-05-27 Thread GitBox
potiuk commented on issue #8712: URL: https://github.com/apache/airflow/issues/8712#issuecomment-634864391 Yep. We want to minimise the prod image size to get the most frequently used packages (and core ones). We've added a new feature today to the prod image - you can add additional extra

[GitHub] [airflow] feluelle commented on a change in pull request #8962: [AIRFLOW-8057] [AIP-31] Add @task decorator

2020-05-27 Thread GitBox
feluelle commented on a change in pull request #8962: URL: https://github.com/apache/airflow/pull/8962#discussion_r431366969 ## File path: docs/concepts.rst ## @@ -116,6 +116,47 @@ DAGs can be used as context managers to automatically assign new operators to th op.dag i

[GitHub] [airflow] potiuk commented on a change in pull request #9038: Use production image for k8s tests

2020-05-27 Thread GitBox
potiuk commented on a change in pull request #9038: URL: https://github.com/apache/airflow/pull/9038#discussion_r431367442 ## File path: scripts/ci/in_container/kubernetes/docker/rebuild_airflow_image.sh ## @@ -38,46 +38,25 @@ cp /entrypoint.sh scripts/docker/ echo echo "Buil

[GitHub] [airflow] dimberman commented on a change in pull request #9038: Use production image for k8s tests

2020-05-27 Thread GitBox
dimberman commented on a change in pull request #9038: URL: https://github.com/apache/airflow/pull/9038#discussion_r431398123 ## File path: scripts/ci/in_container/kubernetes/docker/rebuild_airflow_image.sh ## @@ -38,46 +38,25 @@ cp /entrypoint.sh scripts/docker/ echo echo "B

[GitHub] [airflow] kylegao91 commented on pull request #8081: [AIRFLOW-5500] Fix the trigger_dag api in the case of nested subdags

2020-05-27 Thread GitBox
kylegao91 commented on pull request #8081: URL: https://github.com/apache/airflow/pull/8081#issuecomment-634905164 Really hope this can be merged into 1.10.11, very important for our tasks. This is an automated message from t

[jira] [Commented] (AIRFLOW-5500) Bug in trigger api endpoint

2020-05-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118034#comment-17118034 ] ASF GitHub Bot commented on AIRFLOW-5500: - kylegao91 commented on pull request

[GitHub] [airflow] casassg commented on a change in pull request #8962: [AIRFLOW-8057] [AIP-31] Add @task decorator

2020-05-27 Thread GitBox
casassg commented on a change in pull request #8962: URL: https://github.com/apache/airflow/pull/8962#discussion_r431450538 ## File path: airflow/operators/python.py ## @@ -145,6 +147,141 @@ def execute_callable(self): return self.python_callable(*self.op_args, **self.

[GitHub] [airflow] casassg commented on a change in pull request #8962: [AIRFLOW-8057] [AIP-31] Add @task decorator

2020-05-27 Thread GitBox
casassg commented on a change in pull request #8962: URL: https://github.com/apache/airflow/pull/8962#discussion_r431450538 ## File path: airflow/operators/python.py ## @@ -145,6 +147,141 @@ def execute_callable(self): return self.python_callable(*self.op_args, **self.

[GitHub] [airflow] ephraimbuddy opened a new issue #9040: Correctly describe the dst parameter in LocalFilesystemToGCS operator

2020-05-27 Thread GitBox
ephraimbuddy opened a new issue #9040: URL: https://github.com/apache/airflow/issues/9040 **Apache Airflow version**: 2.0.0 **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): **Environment**: - **Cloud provider or hardware configuration

[GitHub] [airflow] boring-cyborg[bot] commented on issue #9040: Correctly describe the dst parameter in LocalFilesystemToGCS operator

2020-05-27 Thread GitBox
boring-cyborg[bot] commented on issue #9040: URL: https://github.com/apache/airflow/issues/9040#issuecomment-634950607 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated

[GitHub] [airflow] casassg commented on a change in pull request #8962: [AIRFLOW-8057] [AIP-31] Add @task decorator

2020-05-27 Thread GitBox
casassg commented on a change in pull request #8962: URL: https://github.com/apache/airflow/pull/8962#discussion_r431452493 ## File path: airflow/operators/python.py ## @@ -145,6 +147,141 @@ def execute_callable(self): return self.python_callable(*self.op_args, **self.

[GitHub] [airflow] casassg commented on a change in pull request #8962: [AIRFLOW-8057] [AIP-31] Add @task decorator

2020-05-27 Thread GitBox
casassg commented on a change in pull request #8962: URL: https://github.com/apache/airflow/pull/8962#discussion_r431452493 ## File path: airflow/operators/python.py ## @@ -145,6 +147,141 @@ def execute_callable(self): return self.python_callable(*self.op_args, **self.

[GitHub] [airflow] casassg commented on a change in pull request #8962: [AIRFLOW-8057] [AIP-31] Add @task decorator

2020-05-27 Thread GitBox
casassg commented on a change in pull request #8962: URL: https://github.com/apache/airflow/pull/8962#discussion_r431464263 ## File path: tests/operators/test_python.py ## @@ -311,6 +315,350 @@ def func(**context): python_operator.run(start_date=DEFAULT_DATE, end_date=

[GitHub] [airflow] casassg opened a new issue #9041: [AIP-31] Update Airflow tutorial to use functional DAGs

2020-05-27 Thread GitBox
casassg opened a new issue #9041: URL: https://github.com/apache/airflow/issues/9041 **Description** Change Airflow tutorial (or create new tutorial) using the new functional layer. **Use case / motivation** Help users onboard Airflow by reducing the complexity of t

[GitHub] [airflow] casassg commented on a change in pull request #8962: [AIRFLOW-8057] [AIP-31] Add @task decorator

2020-05-27 Thread GitBox
casassg commented on a change in pull request #8962: URL: https://github.com/apache/airflow/pull/8962#discussion_r431476613 ## File path: docs/concepts.rst ## @@ -116,6 +116,47 @@ DAGs can be used as context managers to automatically assign new operators to th op.dag is

[GitHub] [airflow] casassg commented on a change in pull request #8805: Resolve upstream tasks when template field is XComArg

2020-05-27 Thread GitBox
casassg commented on a change in pull request #8805: URL: https://github.com/apache/airflow/pull/8805#discussion_r431484377 ## File path: tests/models/test_baseoperator.py ## @@ -347,3 +350,66 @@ def test_lineage_composition(self): task4 = DummyOperator(task_id="op4",

[GitHub] [airflow] houqp opened a new pull request #9042: detect incompatible docker server version in breeze

2020-05-27 Thread GitBox
houqp opened a new pull request #9042: URL: https://github.com/apache/airflow/pull/9042 Without this detection code, breeze just fails with the following error: ``` === Chec

[GitHub] [airflow] ephraimbuddy opened a new pull request #9043: Add example dag and system test for LocalFilesystemToGCSOperator

2020-05-27 Thread GitBox
ephraimbuddy opened a new pull request #9043: URL: https://github.com/apache/airflow/pull/9043 --- This PR closes one of the issues in #8280 Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests cove

[GitHub] [airflow] houqp commented on pull request #9042: detect incompatible docker server version in breeze

2020-05-27 Thread GitBox
houqp commented on pull request #9042: URL: https://github.com/apache/airflow/pull/9042#issuecomment-635027681 @potiuk I noticed travis build is already failing with this check: ``` Client: Docker Engine - Community Version: 19.03.9 API version: 1.40 Go

[GitHub] [airflow] dazza-codes commented on pull request #6984: [WIP][AIRFLOW-6395] Add an asyncio compatible dask executor [AIP-28]

2020-05-27 Thread GitBox
dazza-codes commented on pull request #6984: URL: https://github.com/apache/airflow/pull/6984#issuecomment-635054207 For various reasons beyond my control, the complexity of Airflow seemed to be a hurdle for various ops and other personnel on a team at work. In short, I don't have time to

[jira] [Commented] (AIRFLOW-6395) [AIP-28] Add AsyncExecutor option

2020-05-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118237#comment-17118237 ] ASF GitHub Bot commented on AIRFLOW-6395: - dazza-codes commented on pull reques

[airflow] tag nightly-master updated (0b0e4f7 -> 7386670)

2020-05-27 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to tag nightly-master in repository https://gitbox.apache.org/repos/asf/airflow.git. *** WARNING: tag nightly-master was modified! *** from 0b0e4f7 (commit) to 7386670 (commit) from 0b

[GitHub] [airflow] msumit commented on a change in pull request #8265: Kubernetes on host

2020-05-27 Thread GitBox
msumit commented on a change in pull request #8265: URL: https://github.com/apache/airflow/pull/8265#discussion_r431569992 ## File path: scripts/ci/kubernetes/app/templates/airflow.template.yaml ## @@ -42,33 +42,43 @@ spec: name: airflow spec: initContainer

[GitHub] [airflow] aneesh-joseph opened a new pull request #9044: do a kinit before yarn kill if keytab and principal is provided

2020-05-27 Thread GitBox
aneesh-joseph opened a new pull request #9044: URL: https://github.com/apache/airflow/pull/9044 --- When the spark submit hook's `on_kill` is called, it tries to kill the yarn app, but doesn't do a kinit even if the keytab and principal is supplied. This would mean that the `yarn ap

[GitHub] [airflow] turbaszek commented on a change in pull request #8962: [AIRFLOW-8057] [AIP-31] Add @task decorator

2020-05-27 Thread GitBox
turbaszek commented on a change in pull request #8962: URL: https://github.com/apache/airflow/pull/8962#discussion_r431578845 ## File path: airflow/operators/python.py ## @@ -145,6 +147,141 @@ def execute_callable(self): return self.python_callable(*self.op_args, **sel

[GitHub] [airflow] turbaszek commented on a change in pull request #8805: Resolve upstream tasks when template field is XComArg

2020-05-27 Thread GitBox
turbaszek commented on a change in pull request #8805: URL: https://github.com/apache/airflow/pull/8805#discussion_r431580206 ## File path: tests/models/test_baseoperator.py ## @@ -347,3 +350,66 @@ def test_lineage_composition(self): task4 = DummyOperator(task_id="op4"

[GitHub] [airflow] turbaszek commented on a change in pull request #8805: Resolve upstream tasks when template field is XComArg

2020-05-27 Thread GitBox
turbaszek commented on a change in pull request #8805: URL: https://github.com/apache/airflow/pull/8805#discussion_r431580206 ## File path: tests/models/test_baseoperator.py ## @@ -347,3 +350,66 @@ def test_lineage_composition(self): task4 = DummyOperator(task_id="op4"

[GitHub] [airflow] turbaszek merged pull request #8922: Add query count test for LocalTaskJob

2020-05-27 Thread GitBox
turbaszek merged pull request #8922: URL: https://github.com/apache/airflow/pull/8922 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[airflow] branch master updated: Add query count test for LocalTaskJob (#8922)

2020-05-27 Thread turbaszek
This is an automated email from the ASF dual-hosted git repository. turbaszek pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/master by this push: new 369e637 Add query count test for LocalTaskJ

[GitHub] [airflow] turbaszek commented on pull request #9038: Use production image for k8s tests

2020-05-27 Thread GitBox
turbaszek commented on pull request #9038: URL: https://github.com/apache/airflow/pull/9038#issuecomment-635115335 Still the same, old error `Couldn't resolve host 'github.com'` :< But it seems the persistent mode work. Would we be ok with having only one k8s test for a while? --

[GitHub] [airflow] codecov-commenter commented on pull request #6794: [AIRFLOW-6231] Display DAG run conf in the list view

2020-05-27 Thread GitBox
codecov-commenter commented on pull request #6794: URL: https://github.com/apache/airflow/pull/6794#issuecomment-635124552 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6794?src=pr&el=h1) Report > Merging [#6794](https://codecov.io/gh/apache/airflow/pull/6794?src=pr&el=desc) int

[jira] [Commented] (AIRFLOW-6231) Show DAG Run conf in graph view

2020-05-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118342#comment-17118342 ] ASF GitHub Bot commented on AIRFLOW-6231: - codecov-commenter commented on pull

[GitHub] [airflow] codecov-commenter commented on pull request #7626: [AIRFLOW-3391] Upgrade pendulum to latest major version

2020-05-27 Thread GitBox
codecov-commenter commented on pull request #7626: URL: https://github.com/apache/airflow/pull/7626#issuecomment-635135025 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7626?src=pr&el=h1) Report > Merging [#7626](https://codecov.io/gh/apache/airflow/pull/7626?src=pr&el=desc) int

[jira] [Commented] (AIRFLOW-3391) Upgrade pendulum to latest major version

2020-05-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118368#comment-17118368 ] ASF GitHub Bot commented on AIRFLOW-3391: - codecov-commenter commented on pull

[GitHub] [airflow] codecov-commenter commented on pull request #8227: Add run_type to DagRun

2020-05-27 Thread GitBox
codecov-commenter commented on pull request #8227: URL: https://github.com/apache/airflow/pull/8227#issuecomment-635135845 # [Codecov](https://codecov.io/gh/apache/airflow/pull/8227?src=pr&el=h1) Report > Merging [#8227](https://codecov.io/gh/apache/airflow/pull/8227?src=pr&el=desc) int

[GitHub] [airflow] turbaszek commented on issue #9034: Fix GoogleDiscoveryApiHook docs warnings

2020-05-27 Thread GitBox
turbaszek commented on issue #9034: URL: https://github.com/apache/airflow/issues/9034#issuecomment-635142616 Indeed, thanks @mik-laj ~ This is an automated message from the Apache Git Service. To respond to the message, plea

[GitHub] [airflow] turbaszek closed issue #9034: Fix GoogleDiscoveryApiHook docs warnings

2020-05-27 Thread GitBox
turbaszek closed issue #9034: URL: https://github.com/apache/airflow/issues/9034 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

<    1   2