[GitHub] [airflow] dstandish commented on issue #6104: [AIRFLOW-4574] allow providing private_key in SSHHook
dstandish commented on issue #6104: [AIRFLOW-4574] allow providing private_key in SSHHook URL: https://github.com/apache/airflow/pull/6104#issuecomment-531580727 @mik-laj thank you -- did not realize there was this doc. I have updated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk merged pull request #6049: [AIRFLOW-XXX] Add prerequisite tasks for all GCP operators guide
potiuk merged pull request #6049: [AIRFLOW-XXX] Add prerequisite tasks for all GCP operators guide URL: https://github.com/apache/airflow/pull/6049 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6086: [AIRFLOW-XXX] Display GCP integration in table
mik-laj commented on a change in pull request #6086: [AIRFLOW-XXX] Display GCP integration in table URL: https://github.com/apache/airflow/pull/6086#discussion_r324480637 ## File path: docs/integration.rst ## @@ -237,230 +235,290 @@ Airflow can be configured to read and write task logs in Google Cloud Storage. See :ref:`write-logs-gcp`. -GoogleCloudBaseHook +Operators and Hooks ''' All hooks are based on :class:`airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook`. - -BigQuery - - -The operators are defined in the following modules: - * :mod:`airflow.gcp.operators.bigquery` - * :mod:`airflow.gcp.sensors.bigquery` - * :mod:`airflow.operators.bigquery_to_bigquery` - * :mod:`airflow.operators.bigquery_to_gcs` - * :mod:`airflow.operators.bigquery_to_mysql` - -They also use :class:`airflow.gcp.hooks.bigquery.BigQueryHook` to communicate with Google Cloud Platform. - -BigQuery Data Transfer Service -'' -The operators are defined in the following module: - - * :mod:`airflow.gcp.operators.bigquery_dts` - * :mod:`airflow.gcp.sensors.bigquery_dts` - -The operator is defined in the :class:`airflow.gcp.operators.spanner` package. - -They also use :class:`airflow.gcp.hooks.bigquery_dts.BiqQueryDataTransferServiceHook` to communicate with Google Cloud Platform. - - -Cloud Spanner -' - -The operator is defined in the :class:`airflow.gcp.operators.spanner` package. - -They also use :class:`airflow.gcp.hooks.spanner.CloudSpannerHook` to communicate with Google Cloud Platform. - - -Cloud SQL -' - -The operator is defined in the :class:`airflow.gcp.operators.cloud_sql` package. - -They also use :class:`airflow.gcp.hooks.cloud_sql.CloudSqlDatabaseHook` and :class:`airflow.gcp.hooks.cloud_sql.CloudSqlHook` to communicate with Google Cloud Platform. - - -Cloud Bigtable -'' - -The operator is defined in the :class:`airflow.gcp.operators.bigtable` package. - -They also use :class:`airflow.gcp.hooks.bigtable.BigtableHook` to communicate with Google Cloud Platform. - -Cloud Build -''' - -The operator is defined in the :class:`airflow.gcp.operators.cloud_build` package. - -They also use :class:`airflow.gcp.hooks.cloud_build.CloudBuildHook` to communicate with Google Cloud Platform. - - -Compute Engine -'' - -The operators are defined in the :class:`airflow.gcp.operators.compute` package. - -They also use :class:`airflow.gcp.hooks.compute.GceHook` to communicate with Google Cloud Platform. - - -Cloud Functions -''' - -The operators are defined in the :class:`airflow.gcp.operators.functions` package. - -They also use :class:`airflow.gcp.hooks.functions.GcfHook` to communicate with Google Cloud Platform. - - -Cloud DataFlow -'' - -The operators are defined in the :class:`airflow.gcp.operators.dataflow` package. - -They also use :class:`airflow.gcp.hooks.dataflow.DataFlowHook` to communicate with Google Cloud Platform. - - -Cloud DataProc -'' - -The operators are defined in the :class:`airflow.gcp.operators.dataproc` package. - - -Cloud Datastore -''' - -The operators are defined in the :class:`airflow.gcp.operators.datastore` package. - -They also use :class:`airflow.gcp.hooks.datastore.DatastoreHook` to communicate with Google Cloud Platform. - - -Cloud ML Engine -''' - -The operators are defined in the :class:`airflow.gcp.operators.mlengine` package. - -They also use :class:`airflow.gcp.hooks.mlengine.MLEngineHook` to communicate with Google Cloud Platform. - -Cloud Storage -' - -The operators are defined in the following module: - - * :mod:`airflow.operators.local_to_gcs` - * :mod:`airflow.gcp.operators.gcs` - * :mod:`airflow.operators.gcs_to_bq` - * :mod:`airflow.operators.mysql_to_gcs` - * :mod:`airflow.gcp.sensors.gcs` - -They also use :class:`airflow.gcp.hooks.gcs.GoogleCloudStorageHook` to communicate with Google Cloud Platform. - - -Transfer Service - - -The operators are defined in the following module: - - * :mod:`airflow.gcp.operators.cloud_storage_transfer_service` - * :mod:`airflow.gcp.sensors.cloud_storage_transfer_service` - -They also use :class:`airflow.gcp.hooks.cloud_storage_transfer_service.GCPTransferServiceHook` to communicate with Google Cloud Platform. - - -Cloud Vision - - -The operator is defined in the :class:`airflow.gcp.operators.vision` package. - -They also use :class:`airflow.gcp.hooks.vision.CloudVisionHook` to communicate with Google Cloud Platform. - -Cloud Text to Speech - - -The operator is defined in the :class:`airflow.gcp.operators.text_to_speech` package. - -They also use :class:`airflow.gcp.hooks.text_to_speech.GCPTextToSpeechHook` to communicate with Google Cloud Platform. - -Cloud Speech to Text - - -The operator is defined in the :class:`airflow.gcp.operators.speech_to_text`
[jira] [Assigned] (AIRFLOW-5495) Remove unneeded parens in dataproc.py
[
https://issues.apache.org/jira/browse/AIRFLOW-5495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adan Christian Rosales Ornelas reassigned AIRFLOW-5495:
---
Assignee: Adan Christian Rosales Ornelas
> Remove unneeded parens in dataproc.py
> -
>
> Key: AIRFLOW-5495
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5495
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.10.6
>Reporter: Jakob Homan
>Assignee: Adan Christian Rosales Ornelas
>Priority: Minor
> Labels: ccoss2019, newbie
>
> Note: This ticket's being created to facilitate a new contributor's workshop
> for Airflow. After the workshop has completed, I'll mark these all available
> for anyone that might like to take them on.
> The parens around {{self.custom_image_project_id}} don't need to be there; we
> should remove them.
> airflow/gcp/operators/dataproc.py:409
> {code:java}
> elif self.custom_image:
> project_id = self.custom_image_project_id if
> (self.custom_image_project_id) else self.project_id
> custom_image_url = 'https://www.googleapis.com/compute/beta/projects/' \
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [airflow] mik-laj commented on a change in pull request #6105: [AIRFLOW-5495] Remove unneeded parens in dataproc.py
mik-laj commented on a change in pull request #6105: [AIRFLOW-5495] Remove unneeded parens in dataproc.py URL: https://github.com/apache/airflow/pull/6105#discussion_r324480808 ## File path: airflow/gcp/operators/dataproc.py ## @@ -407,7 +407,7 @@ def _build_cluster_data(self): cluster_data['config']['softwareConfig']['imageVersion'] = self.image_version elif self.custom_image: -project_id = self.custom_image_project_id if (self.custom_image_project_id) else self.project_id +project_id = self.custom_image_project_id if self.custom_image_project_id else self.project_id Review comment: ```suggestion project_id = self.custom_image_project_id or self.project_id ``` This is the equivalent code, but simpler. WDYT? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-5427) Add system tests for Dataflow
[ https://issues.apache.org/jira/browse/AIRFLOW-5427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5427. --- Fix Version/s: 2.0.0 Resolution: Fixed > Add system tests for Dataflow > - > > Key: AIRFLOW-5427 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5427 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5427) Add system tests for Dataflow
[ https://issues.apache.org/jira/browse/AIRFLOW-5427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930031#comment-16930031 ] ASF subversion and git services commented on AIRFLOW-5427: -- Commit 61d3f0c615064a573a089da77c19b276c0eb51f7 in airflow's branch refs/heads/master from Kamil Breguła [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=61d3f0c ] [AIRFLOW-5427] Add system tests for Dataflow > Add system tests for Dataflow > - > > Key: AIRFLOW-5427 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5427 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [airflow] potiuk merged pull request #6035: [AIRFLOW-5427][depends on AIRFLOW-5428/AIRFLOW-5453] Add system tests for Dataflow integration
potiuk merged pull request #6035: [AIRFLOW-5427][depends on AIRFLOW-5428/AIRFLOW-5453] Add system tests for Dataflow integration URL: https://github.com/apache/airflow/pull/6035 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-5428) Dataflow with one job is not done correctly
[ https://issues.apache.org/jira/browse/AIRFLOW-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5428. --- Fix Version/s: 2.0.0 Resolution: Fixed > Dataflow with one job is not done correctly > --- > > Key: AIRFLOW-5428 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5428 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5428) Dataflow with one job is not done correctly
[ https://issues.apache.org/jira/browse/AIRFLOW-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930027#comment-16930027 ] ASF GitHub Bot commented on AIRFLOW-5428: - potiuk commented on pull request #6036: [AIRFLOW-5428] Dataflow with one job is not done correctly URL: https://github.com/apache/airflow/pull/6036 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Dataflow with one job is not done correctly > --- > > Key: AIRFLOW-5428 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5428 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5428) Dataflow with one job is not done correctly
[ https://issues.apache.org/jira/browse/AIRFLOW-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930028#comment-16930028 ] ASF subversion and git services commented on AIRFLOW-5428: -- Commit 52d9e6a64b0308af2afb4cac3762a93e53108588 in airflow's branch refs/heads/master from Kamil Breguła [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=52d9e6a ] [AIRFLOW-5428] Dataflow with one job is not done correctly (#6036) > Dataflow with one job is not done correctly > --- > > Key: AIRFLOW-5428 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5428 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5427) Add system tests for Dataflow
[ https://issues.apache.org/jira/browse/AIRFLOW-5427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930029#comment-16930029 ] ASF GitHub Bot commented on AIRFLOW-5427: - potiuk commented on pull request #6035: [AIRFLOW-5427][depends on AIRFLOW-5428/AIRFLOW-5453] Add system tests for Dataflow integration URL: https://github.com/apache/airflow/pull/6035 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add system tests for Dataflow > - > > Key: AIRFLOW-5427 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5427 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5453) Improve reading inputs from Dataflow console
[ https://issues.apache.org/jira/browse/AIRFLOW-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930030#comment-16930030 ] ASF subversion and git services commented on AIRFLOW-5453: -- Commit 5b1d529a69edf93d74725be542da3522fad23b74 in airflow's branch refs/heads/master from Kamil Breguła [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=5b1d529 ] [AIRFLOW-5453] Improve reading inputs from Dataflow console > Improve reading inputs from Dataflow console > > > Key: AIRFLOW-5453 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5453 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [airflow] mik-laj commented on a change in pull request #6104: [AIRFLOW-4574] allow providing private_key in SSHHook
mik-laj commented on a change in pull request #6104: [AIRFLOW-4574] allow providing private_key in SSHHook URL: https://github.com/apache/airflow/pull/6104#discussion_r324480874 ## File path: docs/howto/connection/ssh.rst ## @@ -62,8 +63,14 @@ Extra (optional) following the standard syntax of connections, where extras are passed as parameters of the URI (note that all components of the URI should be URL-encoded). -For example: +Example connection string with ``key_file`` (path to key file provided in connection): .. code-block:: bash export AIRFLOW_CONN_MAIN_SERVER='ssh://user:pass@localhost:22?timeout=10=false_host_key_check=false_host_key_change=true_file=%2Fhome%2Fairflow%2F.ssh%2Fid_rsa' + +Example connection string with ``private_key`` (actual private key provided in connection): + +.. code-block:: bash + + AIRFLOW_CONN_SSH_SERVER='SSH://127.0.0.1?private_key=-BEGIN+RSA+PRIVATE+KEY-%0AMIIEpAIBAAKCAQEAvYUM9xouSUtCKMwm%2FkogT4r3Y%2Bh7H0IPnd7DF9sKCHt9FPJ%2B%0ALaQNX%2FRgnOoPf5ySN42A1nmqv4WX5AKdjEYMIJzN2g2whnol8RVjzP4s2Ao%2B%2BWJ9%0AKstey85CQUgjWFO57ye3TyhbfMZI3fBqDX5RjgkgAZmUpKmv6ttSiCfdgGxLweD7%0ADZexlAjuSfr7i0UZWBIbSKJdePMnWGvZZO%2BGerGlOIKs%2Bqx5agMbNJqDhWn0u8OV%0ACMANhc0yaUAbN08Pjac94%2FxmZPHASytrBmTGd6zYcuzOyxwK8KHMeLUagByT3u7l%0AvWcVyRx8FAXkl7nGF2SQZ0z3JLhmdWMSXuc1AQIDAQABAoIBAQC8%2Bp1REVQyVc8k%0A612%2Bl5%2FccU%2F62elb4%2F26iFS1xv8cMjcp2hwj2sBTfFWSYnsN3syWhI2CUFQJImex%0AP0Jmi7qwEmvaEWiCz%2B5hldisoo%2BI5b6h4qm5MI3YYFYEzrAf9W0kos%2FRKQcBRp%2BG%0AX6MAzYL5RPQbZE%2BqWmJGqGiFyGrBEISl%2FMdoaqSJewTRLHwDtbD9lt4WRPUO%2Font%0A%2FUKwOu3i9z5hMQm9HJJLuKr3hl5jmjJbJUg50a7fjVJzr52VfxH73Z%2Fst40fD3x4%0AH1DHGbX4ar9JOYvhzdXkuxyNXvoglJUIOiAk23Od8q9xOMQAITuwkc1QaVRXwiE7%0Aw41lMC8ZAoGBAOB9PEFyzGwYZgiReOQsAJrlwT7zsY053OGSXAeoLC2OzyLNb8v7%0AnKy2qoTMwxe9LHUDDAp6I8btprvLq35Y72iCbGg0ZK5fIYv%2Bt03NjvOOl1zEuUny%0A5xGe1IvP4YgMQuVMVw5dj11Jmna5eW3oFXlyOQrlth9hrexuI%2BG25qwvAoGBANgf%0AOhy%2FofyIgrIGwaRbgg55rlqViLNGFcJ6I3dVlsRqFxND4PvQZZWfCN3LhIGgI8cT%0AN6hFGPR9QrsmXe3eHM7%2FUpMk53oiPD9E0MemPtQh2AFPUb%2BznqxrXNGvtww6xYBM%0AKYLXcQVn%2FKELwwMYw3F0HGKgCFF0XthV34f%2Bt%2FXPAoGBALVLjqEQlBTsM2LSEP68%0AppRx3nn3lrmGNGMbryUj5OG6BoCFxrbG8gXt05JCR4Bhb4jkOBIyB7i87r2VQ19b%0AdaVCR0h0n6bO%2FymvQNwdmUgLLSRnX3hgKcpqKh7reKlFtbS2zUu1tXVSXuNo8K8Z%0AElatL3Ikh8uaODrLzECaVHpTAoGAXcReoC58h2Zq3faUeUzChqlAfki2gKF9u1zm%0AmlXmDd3BmTgwGtD14g6X%2BDLekKb8Htk1oqooA5t9IlmpExT1BtI7719pltHXtdOT%0AiauVQtBUOW1CmJvD0ibapJdKIeI14k4pDH2QqbnOH8lMmMFbupOX5SptsXl91Pqc%0A%2BxIGmn0CgYBOL2o0Sn%2F8d7uzAZKUBG1%2F0eFr4j6wYwWajVDFOfbJ7WdIf5j%2BL3nY%0A3440i%2Fb2NlEE8nLPDl6cwiOtwV0XFkoiF3ctHvutlhGBxAKHetIxIsnQk7vXqgfP%0AnhsgNypNAQXbxe3gjJEb4Fzw3Ufz3mq5PllYtXKhc%2Bmc4%2B3sN5uGow%3D%3D%0A-END+RSA+PRIVATE+KEY-%0A' Review comment: ```suggestion export AIRFLOW_CONN_SSH_SERVER='SSH://127.0.0.1?private_key=-BEGIN+RSA+PRIVATE+KEY-%0AMIIEpAIBAAKCAQEAvYUM9xouSUtCKMwm%2FkogT4r3Y%2Bh7H0IPnd7DF9sKCHt9FPJ%2B%0ALaQNX%2FRgnOoPf5ySN42A1nmqv4WX5AKdjEYMIJzN2g2whnol8RVjzP4s2Ao%2B%2BWJ9%0AKstey85CQUgjWFO57ye3TyhbfMZI3fBqDX5RjgkgAZmUpKmv6ttSiCfdgGxLweD7%0ADZexlAjuSfr7i0UZWBIbSKJdePMnWGvZZO%2BGerGlOIKs%2Bqx5agMbNJqDhWn0u8OV%0ACMANhc0yaUAbN08Pjac94%2FxmZPHASytrBmTGd6zYcuzOyxwK8KHMeLUagByT3u7l%0AvWcVyRx8FAXkl7nGF2SQZ0z3JLhmdWMSXuc1AQIDAQABAoIBAQC8%2Bp1REVQyVc8k%0A612%2Bl5%2FccU%2F62elb4%2F26iFS1xv8cMjcp2hwj2sBTfFWSYnsN3syWhI2CUFQJImex%0AP0Jmi7qwEmvaEWiCz%2B5hldisoo%2BI5b6h4qm5MI3YYFYEzrAf9W0kos%2FRKQcBRp%2BG%0AX6MAzYL5RPQbZE%2BqWmJGqGiFyGrBEISl%2FMdoaqSJewTRLHwDtbD9lt4WRPUO%2Font%0A%2FUKwOu3i9z5hMQm9HJJLuKr3hl5jmjJbJUg50a7fjVJzr52VfxH73Z%2Fst40fD3x4%0AH1DHGbX4ar9JOYvhzdXkuxyNXvoglJUIOiAk23Od8q9xOMQAITuwkc1QaVRXwiE7%0Aw41lMC8ZAoGBAOB9PEFyzGwYZgiReOQsAJrlwT7zsY053OGSXAeoLC2OzyLNb8v7%0AnKy2qoTMwxe9LHUDDAp6I8btprvLq35Y72iCbGg0ZK5fIYv%2Bt03NjvOOl1zEuUny%0A5xGe1IvP4YgMQuVMVw5dj11Jmna5eW3oFXlyOQrlth9hrexuI%2BG25qwvAoGBANgf%0AOhy%2FofyIgrIGwaRbgg55rlqViLNGFcJ6I3dVlsRqFxND4PvQZZWfCN3LhIGgI8cT%0AN6hFGPR9QrsmXe3eHM7%2FUpMk53oiPD9E0MemPtQh2AFPUb%2BznqxrXNGvtww6xYBM%0AKYLXcQVn%2FKELwwMYw3F0HGKgCFF0XthV34f%2Bt%2FXPAoGBALVLjqEQlBTsM2LSEP68%0AppRx3nn3lrmGNGMbryUj5OG6BoCFxrbG8gXt05JCR4Bhb4jkOBIyB7i87r2VQ19b%0AdaVCR0h0n6bO%2FymvQNwdmUgLLSRnX3hgKcpqKh7reKlFtbS2zUu1tXVSXuNo8K8Z%0AElatL3Ikh8uaODrLzECaVHpTAoGAXcReoC58h2Zq3faUeUzChqlAfki2gKF9u1zm%0AmlXmDd3BmTgwGtD14g6X%2BDLekKb8Htk1oqooA5t9IlmpExT1BtI7719pltHXtdOT%0AiauVQtBUOW1CmJvD0ibapJdKIeI14k4pDH2QqbnOH8lMmMFbupOX5SptsXl91Pqc%0A%2BxIGmn0CgYBOL2o0Sn%2F8d7uzAZKUBG1%2F0eFr4j6wYwWajVDFOfbJ7WdIf5j%2BL3nY%0A3440i%2Fb2NlEE8nLPDl6cwiOtwV0XFkoiF3ctHvutlhGBxAKHetIxIsnQk7vXqgfP%0AnhsgNypNAQXbxe3gjJEb4Fzw3Ufz3mq5PllYtXKhc%2Bmc4%2B3sN5uGow%3D%3D%0A-END+RSA+PRIVATE+KEY-%0A' ``` This will be consistent with the example above. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For
[GitHub] [airflow] potiuk opened a new pull request #6115: [AIRFLOW-5150] Implement POC for GitLab + Kubernetes tests
potiuk opened a new pull request #6115: [AIRFLOW-5150] Implement POC for GitLab
+ Kubernetes tests
URL: https://github.com/apache/airflow/pull/6115
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6104: [AIRFLOW-4574] allow providing private_key in SSHHook
codecov-io commented on issue #6104: [AIRFLOW-4574] allow providing private_key in SSHHook URL: https://github.com/apache/airflow/pull/6104#issuecomment-531589540 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6104?src=pr=h1) Report > Merging [#6104](https://codecov.io/gh/apache/airflow/pull/6104?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/7be55609fa50319b7806c9b03167ca10e97257d8?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6104?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6104 +/- ## == - Coverage 80.09% 79.81% -0.29% == Files 606 607 +1 Lines 3489035031 +141 == + Hits2794527959 +14 - Misses 6945 7072 +127 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6104?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/hooks/ssh\_hook.py](https://codecov.io/gh/apache/airflow/pull/6104/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3NzaF9ob29rLnB5) | `88.78% <100%> (+1.28%)` | :arrow_up: | | [airflow/gcp/hooks/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6104/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGF0YWZsb3cucHk=) | `48.76% <0%> (-26.84%)` | :arrow_down: | | [airflow/gcp/example\_dags/example\_dataflow.py](https://codecov.io/gh/apache/airflow/pull/6104/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfZGF0YWZsb3cucHk=) | `0% <0%> (ø)` | | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6104/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.98% <0%> (+0.18%)` | :arrow_up: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6104/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `74.58% <0%> (+0.3%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6104?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6104?src=pr=footer). Last update [7be5560...51b05b3](https://codecov.io/gh/apache/airflow/pull/6104?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5453) Improve reading inputs from Dataflow console
[ https://issues.apache.org/jira/browse/AIRFLOW-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930038#comment-16930038 ] ASF GitHub Bot commented on AIRFLOW-5453: - potiuk commented on pull request #6073: [AIRFLOW-5453] Improve reading inputs from Dataflow console URL: https://github.com/apache/airflow/pull/6073 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Improve reading inputs from Dataflow console > > > Key: AIRFLOW-5453 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5453 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [airflow] feluelle commented on a change in pull request #6086: [AIRFLOW-XXX] Display GCP integration in table
feluelle commented on a change in pull request #6086: [AIRFLOW-XXX] Display GCP integration in table URL: https://github.com/apache/airflow/pull/6086#discussion_r324477112 ## File path: docs/integration.rst ## @@ -237,230 +235,290 @@ Airflow can be configured to read and write task logs in Google Cloud Storage. See :ref:`write-logs-gcp`. -GoogleCloudBaseHook +Operators and Hooks ''' All hooks are based on :class:`airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook`. - -BigQuery - - -The operators are defined in the following modules: - * :mod:`airflow.gcp.operators.bigquery` - * :mod:`airflow.gcp.sensors.bigquery` - * :mod:`airflow.operators.bigquery_to_bigquery` - * :mod:`airflow.operators.bigquery_to_gcs` - * :mod:`airflow.operators.bigquery_to_mysql` - -They also use :class:`airflow.gcp.hooks.bigquery.BigQueryHook` to communicate with Google Cloud Platform. - -BigQuery Data Transfer Service -'' -The operators are defined in the following module: - - * :mod:`airflow.gcp.operators.bigquery_dts` - * :mod:`airflow.gcp.sensors.bigquery_dts` - -The operator is defined in the :class:`airflow.gcp.operators.spanner` package. - -They also use :class:`airflow.gcp.hooks.bigquery_dts.BiqQueryDataTransferServiceHook` to communicate with Google Cloud Platform. - - -Cloud Spanner -' - -The operator is defined in the :class:`airflow.gcp.operators.spanner` package. - -They also use :class:`airflow.gcp.hooks.spanner.CloudSpannerHook` to communicate with Google Cloud Platform. - - -Cloud SQL -' - -The operator is defined in the :class:`airflow.gcp.operators.cloud_sql` package. - -They also use :class:`airflow.gcp.hooks.cloud_sql.CloudSqlDatabaseHook` and :class:`airflow.gcp.hooks.cloud_sql.CloudSqlHook` to communicate with Google Cloud Platform. - - -Cloud Bigtable -'' - -The operator is defined in the :class:`airflow.gcp.operators.bigtable` package. - -They also use :class:`airflow.gcp.hooks.bigtable.BigtableHook` to communicate with Google Cloud Platform. - -Cloud Build -''' - -The operator is defined in the :class:`airflow.gcp.operators.cloud_build` package. - -They also use :class:`airflow.gcp.hooks.cloud_build.CloudBuildHook` to communicate with Google Cloud Platform. - - -Compute Engine -'' - -The operators are defined in the :class:`airflow.gcp.operators.compute` package. - -They also use :class:`airflow.gcp.hooks.compute.GceHook` to communicate with Google Cloud Platform. - - -Cloud Functions -''' - -The operators are defined in the :class:`airflow.gcp.operators.functions` package. - -They also use :class:`airflow.gcp.hooks.functions.GcfHook` to communicate with Google Cloud Platform. - - -Cloud DataFlow -'' - -The operators are defined in the :class:`airflow.gcp.operators.dataflow` package. - -They also use :class:`airflow.gcp.hooks.dataflow.DataFlowHook` to communicate with Google Cloud Platform. - - -Cloud DataProc -'' - -The operators are defined in the :class:`airflow.gcp.operators.dataproc` package. - - -Cloud Datastore -''' - -The operators are defined in the :class:`airflow.gcp.operators.datastore` package. - -They also use :class:`airflow.gcp.hooks.datastore.DatastoreHook` to communicate with Google Cloud Platform. - - -Cloud ML Engine -''' - -The operators are defined in the :class:`airflow.gcp.operators.mlengine` package. - -They also use :class:`airflow.gcp.hooks.mlengine.MLEngineHook` to communicate with Google Cloud Platform. - -Cloud Storage -' - -The operators are defined in the following module: - - * :mod:`airflow.operators.local_to_gcs` - * :mod:`airflow.gcp.operators.gcs` - * :mod:`airflow.operators.gcs_to_bq` - * :mod:`airflow.operators.mysql_to_gcs` - * :mod:`airflow.gcp.sensors.gcs` - -They also use :class:`airflow.gcp.hooks.gcs.GoogleCloudStorageHook` to communicate with Google Cloud Platform. - - -Transfer Service - - -The operators are defined in the following module: - - * :mod:`airflow.gcp.operators.cloud_storage_transfer_service` - * :mod:`airflow.gcp.sensors.cloud_storage_transfer_service` - -They also use :class:`airflow.gcp.hooks.cloud_storage_transfer_service.GCPTransferServiceHook` to communicate with Google Cloud Platform. - - -Cloud Vision - - -The operator is defined in the :class:`airflow.gcp.operators.vision` package. - -They also use :class:`airflow.gcp.hooks.vision.CloudVisionHook` to communicate with Google Cloud Platform. - -Cloud Text to Speech - - -The operator is defined in the :class:`airflow.gcp.operators.text_to_speech` package. - -They also use :class:`airflow.gcp.hooks.text_to_speech.GCPTextToSpeechHook` to communicate with Google Cloud Platform. - -Cloud Speech to Text - - -The operator is defined in the :class:`airflow.gcp.operators.speech_to_text`
[jira] [Commented] (AIRFLOW-5484) PigCliHook has incorrect named parameter
[
https://issues.apache.org/jira/browse/AIRFLOW-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930081#comment-16930081
]
Josue Esaud Gomez Gonzalez commented on AIRFLOW-5484:
-
PigCliHook has incorrect named parameter
> PigCliHook has incorrect named parameter
>
>
> Key: AIRFLOW-5484
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5484
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
>Affects Versions: 1.10.6
>Reporter: Jakob Homan
>Priority: Minor
> Labels: ccoss2019, newbie
>
> When building the connection hook, we try to assign a variable named
> `pig_conn_id`. However, this doesn't exist, the correct name is
> `pig_cli_conn_id`. This will cause the correct config to not be picked up.
> airflow/models/connection.py:212
> {code:java}
> elif self.conn_type == 'pig_cli':
> from airflow.hooks.pig_hook import PigCliHook
> return PigCliHook(pig_conn_id=self.conn_id) {code}
> airflow/hooks/pig_hook.py:38
> {code:java}
> def __init__(
> self,
> pig_cli_conn_id="pig_cli_default"):
> conn = self.get_connection(pig_cli_conn_id)
> self.pig_properties = conn.extra_dejson.get('pig_properties', '')
> self.conn = conn {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [airflow] potiuk merged pull request #6036: [AIRFLOW-5428] Dataflow with one job is not done correctly
potiuk merged pull request #6036: [AIRFLOW-5428] Dataflow with one job is not done correctly URL: https://github.com/apache/airflow/pull/6036 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6062: [AIRFLOW-5445] Reduce the required resources for the Kubernetes's sidecar
potiuk commented on issue #6062: [AIRFLOW-5445] Reduce the required resources
for the Kubernetes's sidecar
URL: https://github.com/apache/airflow/pull/6062#issuecomment-531589124
@mik-laj I assume you refer to:
```
resources: {}
```
I could not find how it works in this case. Will we have requests/limits set
to default values in this case? I believe so - from the documentation it looks
like. And this is something you tried to prevent.
But maybe you find some proof otherwise?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #6086: [AIRFLOW-XXX] Display GCP integration in table
potiuk commented on a change in pull request #6086: [AIRFLOW-XXX] Display GCP integration in table URL: https://github.com/apache/airflow/pull/6086#discussion_r324475914 ## File path: docs/integration.rst ## @@ -237,230 +235,290 @@ Airflow can be configured to read and write task logs in Google Cloud Storage. See :ref:`write-logs-gcp`. -GoogleCloudBaseHook +Operators and Hooks ''' All hooks are based on :class:`airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook`. - -BigQuery - - -The operators are defined in the following modules: - * :mod:`airflow.gcp.operators.bigquery` - * :mod:`airflow.gcp.sensors.bigquery` - * :mod:`airflow.operators.bigquery_to_bigquery` - * :mod:`airflow.operators.bigquery_to_gcs` - * :mod:`airflow.operators.bigquery_to_mysql` - -They also use :class:`airflow.gcp.hooks.bigquery.BigQueryHook` to communicate with Google Cloud Platform. - -BigQuery Data Transfer Service -'' -The operators are defined in the following module: - - * :mod:`airflow.gcp.operators.bigquery_dts` - * :mod:`airflow.gcp.sensors.bigquery_dts` - -The operator is defined in the :class:`airflow.gcp.operators.spanner` package. - -They also use :class:`airflow.gcp.hooks.bigquery_dts.BiqQueryDataTransferServiceHook` to communicate with Google Cloud Platform. - - -Cloud Spanner -' - -The operator is defined in the :class:`airflow.gcp.operators.spanner` package. - -They also use :class:`airflow.gcp.hooks.spanner.CloudSpannerHook` to communicate with Google Cloud Platform. - - -Cloud SQL -' - -The operator is defined in the :class:`airflow.gcp.operators.cloud_sql` package. - -They also use :class:`airflow.gcp.hooks.cloud_sql.CloudSqlDatabaseHook` and :class:`airflow.gcp.hooks.cloud_sql.CloudSqlHook` to communicate with Google Cloud Platform. - - -Cloud Bigtable -'' - -The operator is defined in the :class:`airflow.gcp.operators.bigtable` package. - -They also use :class:`airflow.gcp.hooks.bigtable.BigtableHook` to communicate with Google Cloud Platform. - -Cloud Build -''' - -The operator is defined in the :class:`airflow.gcp.operators.cloud_build` package. - -They also use :class:`airflow.gcp.hooks.cloud_build.CloudBuildHook` to communicate with Google Cloud Platform. - - -Compute Engine -'' - -The operators are defined in the :class:`airflow.gcp.operators.compute` package. - -They also use :class:`airflow.gcp.hooks.compute.GceHook` to communicate with Google Cloud Platform. - - -Cloud Functions -''' - -The operators are defined in the :class:`airflow.gcp.operators.functions` package. - -They also use :class:`airflow.gcp.hooks.functions.GcfHook` to communicate with Google Cloud Platform. - - -Cloud DataFlow -'' - -The operators are defined in the :class:`airflow.gcp.operators.dataflow` package. - -They also use :class:`airflow.gcp.hooks.dataflow.DataFlowHook` to communicate with Google Cloud Platform. - - -Cloud DataProc -'' - -The operators are defined in the :class:`airflow.gcp.operators.dataproc` package. - - -Cloud Datastore -''' - -The operators are defined in the :class:`airflow.gcp.operators.datastore` package. - -They also use :class:`airflow.gcp.hooks.datastore.DatastoreHook` to communicate with Google Cloud Platform. - - -Cloud ML Engine -''' - -The operators are defined in the :class:`airflow.gcp.operators.mlengine` package. - -They also use :class:`airflow.gcp.hooks.mlengine.MLEngineHook` to communicate with Google Cloud Platform. - -Cloud Storage -' - -The operators are defined in the following module: - - * :mod:`airflow.operators.local_to_gcs` - * :mod:`airflow.gcp.operators.gcs` - * :mod:`airflow.operators.gcs_to_bq` - * :mod:`airflow.operators.mysql_to_gcs` - * :mod:`airflow.gcp.sensors.gcs` - -They also use :class:`airflow.gcp.hooks.gcs.GoogleCloudStorageHook` to communicate with Google Cloud Platform. - - -Transfer Service - - -The operators are defined in the following module: - - * :mod:`airflow.gcp.operators.cloud_storage_transfer_service` - * :mod:`airflow.gcp.sensors.cloud_storage_transfer_service` - -They also use :class:`airflow.gcp.hooks.cloud_storage_transfer_service.GCPTransferServiceHook` to communicate with Google Cloud Platform. - - -Cloud Vision - - -The operator is defined in the :class:`airflow.gcp.operators.vision` package. - -They also use :class:`airflow.gcp.hooks.vision.CloudVisionHook` to communicate with Google Cloud Platform. - -Cloud Text to Speech - - -The operator is defined in the :class:`airflow.gcp.operators.text_to_speech` package. - -They also use :class:`airflow.gcp.hooks.text_to_speech.GCPTextToSpeechHook` to communicate with Google Cloud Platform. - -Cloud Speech to Text - - -The operator is defined in the :class:`airflow.gcp.operators.speech_to_text`
[GitHub] [airflow] feluelle commented on a change in pull request #6086: [AIRFLOW-XXX] Display GCP integration in table
feluelle commented on a change in pull request #6086: [AIRFLOW-XXX] Display GCP integration in table URL: https://github.com/apache/airflow/pull/6086#discussion_r324477258 ## File path: docs/integration.rst ## @@ -237,230 +235,290 @@ Airflow can be configured to read and write task logs in Google Cloud Storage. See :ref:`write-logs-gcp`. -GoogleCloudBaseHook +Operators and Hooks ''' All hooks are based on :class:`airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook`. - -BigQuery - - -The operators are defined in the following modules: - * :mod:`airflow.gcp.operators.bigquery` - * :mod:`airflow.gcp.sensors.bigquery` - * :mod:`airflow.operators.bigquery_to_bigquery` - * :mod:`airflow.operators.bigquery_to_gcs` - * :mod:`airflow.operators.bigquery_to_mysql` - -They also use :class:`airflow.gcp.hooks.bigquery.BigQueryHook` to communicate with Google Cloud Platform. - -BigQuery Data Transfer Service -'' -The operators are defined in the following module: - - * :mod:`airflow.gcp.operators.bigquery_dts` - * :mod:`airflow.gcp.sensors.bigquery_dts` - -The operator is defined in the :class:`airflow.gcp.operators.spanner` package. - -They also use :class:`airflow.gcp.hooks.bigquery_dts.BiqQueryDataTransferServiceHook` to communicate with Google Cloud Platform. - - -Cloud Spanner -' - -The operator is defined in the :class:`airflow.gcp.operators.spanner` package. - -They also use :class:`airflow.gcp.hooks.spanner.CloudSpannerHook` to communicate with Google Cloud Platform. - - -Cloud SQL -' - -The operator is defined in the :class:`airflow.gcp.operators.cloud_sql` package. - -They also use :class:`airflow.gcp.hooks.cloud_sql.CloudSqlDatabaseHook` and :class:`airflow.gcp.hooks.cloud_sql.CloudSqlHook` to communicate with Google Cloud Platform. - - -Cloud Bigtable -'' - -The operator is defined in the :class:`airflow.gcp.operators.bigtable` package. - -They also use :class:`airflow.gcp.hooks.bigtable.BigtableHook` to communicate with Google Cloud Platform. - -Cloud Build -''' - -The operator is defined in the :class:`airflow.gcp.operators.cloud_build` package. - -They also use :class:`airflow.gcp.hooks.cloud_build.CloudBuildHook` to communicate with Google Cloud Platform. - - -Compute Engine -'' - -The operators are defined in the :class:`airflow.gcp.operators.compute` package. - -They also use :class:`airflow.gcp.hooks.compute.GceHook` to communicate with Google Cloud Platform. - - -Cloud Functions -''' - -The operators are defined in the :class:`airflow.gcp.operators.functions` package. - -They also use :class:`airflow.gcp.hooks.functions.GcfHook` to communicate with Google Cloud Platform. - - -Cloud DataFlow -'' - -The operators are defined in the :class:`airflow.gcp.operators.dataflow` package. - -They also use :class:`airflow.gcp.hooks.dataflow.DataFlowHook` to communicate with Google Cloud Platform. - - -Cloud DataProc -'' - -The operators are defined in the :class:`airflow.gcp.operators.dataproc` package. - - -Cloud Datastore -''' - -The operators are defined in the :class:`airflow.gcp.operators.datastore` package. - -They also use :class:`airflow.gcp.hooks.datastore.DatastoreHook` to communicate with Google Cloud Platform. - - -Cloud ML Engine -''' - -The operators are defined in the :class:`airflow.gcp.operators.mlengine` package. - -They also use :class:`airflow.gcp.hooks.mlengine.MLEngineHook` to communicate with Google Cloud Platform. - -Cloud Storage -' - -The operators are defined in the following module: - - * :mod:`airflow.operators.local_to_gcs` - * :mod:`airflow.gcp.operators.gcs` - * :mod:`airflow.operators.gcs_to_bq` - * :mod:`airflow.operators.mysql_to_gcs` - * :mod:`airflow.gcp.sensors.gcs` - -They also use :class:`airflow.gcp.hooks.gcs.GoogleCloudStorageHook` to communicate with Google Cloud Platform. - - -Transfer Service - - -The operators are defined in the following module: - - * :mod:`airflow.gcp.operators.cloud_storage_transfer_service` - * :mod:`airflow.gcp.sensors.cloud_storage_transfer_service` - -They also use :class:`airflow.gcp.hooks.cloud_storage_transfer_service.GCPTransferServiceHook` to communicate with Google Cloud Platform. - - -Cloud Vision - - -The operator is defined in the :class:`airflow.gcp.operators.vision` package. - -They also use :class:`airflow.gcp.hooks.vision.CloudVisionHook` to communicate with Google Cloud Platform. - -Cloud Text to Speech - - -The operator is defined in the :class:`airflow.gcp.operators.text_to_speech` package. - -They also use :class:`airflow.gcp.hooks.text_to_speech.GCPTextToSpeechHook` to communicate with Google Cloud Platform. - -Cloud Speech to Text - - -The operator is defined in the :class:`airflow.gcp.operators.speech_to_text`
[jira] [Resolved] (AIRFLOW-5476) Typo in BREEZE.rst
[ https://issues.apache.org/jira/browse/AIRFLOW-5476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lee Dongjin resolved AIRFLOW-5476. -- Resolution: Fixed > Typo in BREEZE.rst > -- > > Key: AIRFLOW-5476 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5476 > Project: Apache Airflow > Issue Type: Improvement > Components: documentation >Affects Versions: 1.10.5 >Reporter: Lee Dongjin >Assignee: Lee Dongjin >Priority: Trivial > > You can choose the optional flags you need with breez+s+ → You can choose the > optional flags you need with breez+e+ -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [airflow] kaxil commented on a change in pull request #6101: [AIRFLOW-5481] Allow Deleting Renamed DAGs
kaxil commented on a change in pull request #6101: [AIRFLOW-5481] Allow Deleting Renamed DAGs URL: https://github.com/apache/airflow/pull/6101#discussion_r324463012 ## File path: tests/core.py ## @@ -1696,6 +1696,15 @@ def test_delete_dag(self): '--yes']) ) +# Test to check that the DAG should be deleted even if +# the file containing it is not deleted Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (AIRFLOW-5482) Deprecate Schedule Interval on task level
[ https://issues.apache.org/jira/browse/AIRFLOW-5482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik reassigned AIRFLOW-5482: --- Assignee: Kaxil Naik > Deprecate Schedule Interval on task level > - > > Key: AIRFLOW-5482 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5482 > Project: Apache Airflow > Issue Type: Task > Components: core >Affects Versions: 2.0.0, 1.10.5 >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Minor > Fix For: 2.0.0 > > > It has been 4 years since it was deprecated. > [https://github.com/apache/airflow/commit/3e8bb2abf18c3a130c52288e25f5f7d114e407ad] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5482) Deprecate Schedule Interval on task level
[ https://issues.apache.org/jira/browse/AIRFLOW-5482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929949#comment-16929949 ] ASF subversion and git services commented on AIRFLOW-5482: -- Commit 7be55609fa50319b7806c9b03167ca10e97257d8 in airflow's branch refs/heads/master from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=7be5560 ] [AIRFLOW-5482] Deprecate Schedule Interval on task level (#6103) * [AIRFLOW-5482] Deprecate Schedule Interval on task level * Update baseoperator.py > Deprecate Schedule Interval on task level > - > > Key: AIRFLOW-5482 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5482 > Project: Apache Airflow > Issue Type: Task > Components: core >Affects Versions: 2.0.0, 1.10.5 >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Minor > Fix For: 2.0.0 > > > It has been 4 years since it was deprecated. > [https://github.com/apache/airflow/commit/3e8bb2abf18c3a130c52288e25f5f7d114e407ad] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5482) Deprecate Schedule Interval on task level
[ https://issues.apache.org/jira/browse/AIRFLOW-5482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929950#comment-16929950 ] ASF subversion and git services commented on AIRFLOW-5482: -- Commit 7be55609fa50319b7806c9b03167ca10e97257d8 in airflow's branch refs/heads/master from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=7be5560 ] [AIRFLOW-5482] Deprecate Schedule Interval on task level (#6103) * [AIRFLOW-5482] Deprecate Schedule Interval on task level * Update baseoperator.py > Deprecate Schedule Interval on task level > - > > Key: AIRFLOW-5482 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5482 > Project: Apache Airflow > Issue Type: Task > Components: core >Affects Versions: 2.0.0, 1.10.5 >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Minor > Fix For: 2.0.0 > > > It has been 4 years since it was deprecated. > [https://github.com/apache/airflow/commit/3e8bb2abf18c3a130c52288e25f5f7d114e407ad] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (AIRFLOW-5482) Deprecate Schedule Interval on task level
[ https://issues.apache.org/jira/browse/AIRFLOW-5482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-5482. - Resolution: Fixed > Deprecate Schedule Interval on task level > - > > Key: AIRFLOW-5482 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5482 > Project: Apache Airflow > Issue Type: Task > Components: core >Affects Versions: 2.0.0, 1.10.5 >Reporter: Kaxil Naik >Priority: Minor > Fix For: 2.0.0 > > > It has been 4 years since it was deprecated. > [https://github.com/apache/airflow/commit/3e8bb2abf18c3a130c52288e25f5f7d114e407ad] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [airflow] kaxil merged pull request #6103: [AIRFLOW-5482] Deprecate Schedule Interval on task level
kaxil merged pull request #6103: [AIRFLOW-5482] Deprecate Schedule Interval on task level URL: https://github.com/apache/airflow/pull/6103 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5482) Deprecate Schedule Interval on task level
[ https://issues.apache.org/jira/browse/AIRFLOW-5482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929948#comment-16929948 ] ASF GitHub Bot commented on AIRFLOW-5482: - kaxil commented on pull request #6103: [AIRFLOW-5482] Deprecate Schedule Interval on task level URL: https://github.com/apache/airflow/pull/6103 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Deprecate Schedule Interval on task level > - > > Key: AIRFLOW-5482 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5482 > Project: Apache Airflow > Issue Type: Task > Components: core >Affects Versions: 2.0.0, 1.10.5 >Reporter: Kaxil Naik >Priority: Minor > Fix For: 2.0.0 > > > It has been 4 years since it was deprecated. > [https://github.com/apache/airflow/commit/3e8bb2abf18c3a130c52288e25f5f7d114e407ad] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [airflow] mik-laj commented on issue #6104: [AIRFLOW-4574] allow providing private_key in SSHHook
mik-laj commented on issue #6104: [AIRFLOW-4574] allow providing private_key in SSHHook URL: https://github.com/apache/airflow/pull/6104#issuecomment-531545093 Can you also update documentation? https://airflow.readthedocs.io/en/latest/howto/connection/ssh.html This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] clacke commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill
clacke commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill URL: https://github.com/apache/airflow/pull/5498#issuecomment-531572470 I'll construct a minimal example and try things out. The operator creating the dagrun does sound like what should sensibly happen, and matches the description of the PR, but it doesn't match my observations. I don't *think* we're doing anything weird, but a minimal example will provide better data. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6103: [AIRFLOW-5482] Deprecate Schedule Interval on task level
codecov-io commented on issue #6103: [AIRFLOW-5482] Deprecate Schedule Interval on task level URL: https://github.com/apache/airflow/pull/6103#issuecomment-531540124 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6103?src=pr=h1) Report > Merging [#6103](https://codecov.io/gh/apache/airflow/pull/6103?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/33c2a2877359289a3de62ac05f6749570a182628?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6103?src=pr=tree) ```diff @@Coverage Diff@@ ## master #6103 +/- ## = - Coverage80.1% 80.1% -0.01% = Files 606 606 Lines 34893 34890 -3 = - Hits27951 27947 -4 - Misses 69426943 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6103?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/6103/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `95.35% <ø> (+0.44%)` | :arrow_up: | | [airflow/gcp/hooks/kubernetes\_engine.py](https://codecov.io/gh/apache/airflow/pull/6103/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3Mva3ViZXJuZXRlc19lbmdpbmUucHk=) | `95.89% <0%> (-2.67%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6103/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.8% <0%> (-0.19%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6103?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6103?src=pr=footer). Last update [33c2a28...81bf57c](https://codecov.io/gh/apache/airflow/pull/6103?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6103: [AIRFLOW-5482] Deprecate Schedule Interval on task level
codecov-io edited a comment on issue #6103: [AIRFLOW-5482] Deprecate Schedule Interval on task level URL: https://github.com/apache/airflow/pull/6103#issuecomment-531540124 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6103?src=pr=h1) Report > Merging [#6103](https://codecov.io/gh/apache/airflow/pull/6103?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/33c2a2877359289a3de62ac05f6749570a182628?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6103?src=pr=tree) ```diff @@Coverage Diff@@ ## master #6103 +/- ## = - Coverage80.1% 80.1% -0.01% = Files 606 606 Lines 34893 34890 -3 = - Hits27951 27947 -4 - Misses 69426943 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6103?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/6103/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `95.35% <ø> (+0.44%)` | :arrow_up: | | [airflow/gcp/hooks/kubernetes\_engine.py](https://codecov.io/gh/apache/airflow/pull/6103/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3Mva3ViZXJuZXRlc19lbmdpbmUucHk=) | `95.89% <0%> (-2.67%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6103/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.8% <0%> (-0.19%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6103?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6103?src=pr=footer). Last update [33c2a28...81bf57c](https://codecov.io/gh/apache/airflow/pull/6103?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] tooptoop4 commented on issue #5639: [AIRFLOW-4883] Add missing code lost during rebase of original commit
tooptoop4 commented on issue #5639: [AIRFLOW-4883] Add missing code lost during rebase of original commit URL: https://github.com/apache/airflow/pull/5639#issuecomment-531542823 @aoen can u look at https://stackoverflow.com/questions/57713394/airflow-scheduler-stuck ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #6086: [AIRFLOW-XXX] Display GCP integration in table
feluelle commented on a change in pull request #6086: [AIRFLOW-XXX] Display GCP integration in table URL: https://github.com/apache/airflow/pull/6086#discussion_r324453800 ## File path: docs/integration.rst ## @@ -237,230 +235,290 @@ Airflow can be configured to read and write task logs in Google Cloud Storage. See :ref:`write-logs-gcp`. -GoogleCloudBaseHook +Operators and Hooks ''' All hooks are based on :class:`airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook`. - -BigQuery - - -The operators are defined in the following modules: - * :mod:`airflow.gcp.operators.bigquery` - * :mod:`airflow.gcp.sensors.bigquery` - * :mod:`airflow.operators.bigquery_to_bigquery` - * :mod:`airflow.operators.bigquery_to_gcs` - * :mod:`airflow.operators.bigquery_to_mysql` - -They also use :class:`airflow.gcp.hooks.bigquery.BigQueryHook` to communicate with Google Cloud Platform. - -BigQuery Data Transfer Service -'' -The operators are defined in the following module: - - * :mod:`airflow.gcp.operators.bigquery_dts` - * :mod:`airflow.gcp.sensors.bigquery_dts` - -The operator is defined in the :class:`airflow.gcp.operators.spanner` package. - -They also use :class:`airflow.gcp.hooks.bigquery_dts.BiqQueryDataTransferServiceHook` to communicate with Google Cloud Platform. - - -Cloud Spanner -' - -The operator is defined in the :class:`airflow.gcp.operators.spanner` package. - -They also use :class:`airflow.gcp.hooks.spanner.CloudSpannerHook` to communicate with Google Cloud Platform. - - -Cloud SQL -' - -The operator is defined in the :class:`airflow.gcp.operators.cloud_sql` package. - -They also use :class:`airflow.gcp.hooks.cloud_sql.CloudSqlDatabaseHook` and :class:`airflow.gcp.hooks.cloud_sql.CloudSqlHook` to communicate with Google Cloud Platform. - - -Cloud Bigtable -'' - -The operator is defined in the :class:`airflow.gcp.operators.bigtable` package. - -They also use :class:`airflow.gcp.hooks.bigtable.BigtableHook` to communicate with Google Cloud Platform. - -Cloud Build -''' - -The operator is defined in the :class:`airflow.gcp.operators.cloud_build` package. - -They also use :class:`airflow.gcp.hooks.cloud_build.CloudBuildHook` to communicate with Google Cloud Platform. - - -Compute Engine -'' - -The operators are defined in the :class:`airflow.gcp.operators.compute` package. - -They also use :class:`airflow.gcp.hooks.compute.GceHook` to communicate with Google Cloud Platform. - - -Cloud Functions -''' - -The operators are defined in the :class:`airflow.gcp.operators.functions` package. - -They also use :class:`airflow.gcp.hooks.functions.GcfHook` to communicate with Google Cloud Platform. - - -Cloud DataFlow -'' - -The operators are defined in the :class:`airflow.gcp.operators.dataflow` package. - -They also use :class:`airflow.gcp.hooks.dataflow.DataFlowHook` to communicate with Google Cloud Platform. - - -Cloud DataProc -'' - -The operators are defined in the :class:`airflow.gcp.operators.dataproc` package. - - -Cloud Datastore -''' - -The operators are defined in the :class:`airflow.gcp.operators.datastore` package. - -They also use :class:`airflow.gcp.hooks.datastore.DatastoreHook` to communicate with Google Cloud Platform. - - -Cloud ML Engine -''' - -The operators are defined in the :class:`airflow.gcp.operators.mlengine` package. - -They also use :class:`airflow.gcp.hooks.mlengine.MLEngineHook` to communicate with Google Cloud Platform. - -Cloud Storage -' - -The operators are defined in the following module: - - * :mod:`airflow.operators.local_to_gcs` - * :mod:`airflow.gcp.operators.gcs` - * :mod:`airflow.operators.gcs_to_bq` - * :mod:`airflow.operators.mysql_to_gcs` - * :mod:`airflow.gcp.sensors.gcs` - -They also use :class:`airflow.gcp.hooks.gcs.GoogleCloudStorageHook` to communicate with Google Cloud Platform. - - -Transfer Service - - -The operators are defined in the following module: - - * :mod:`airflow.gcp.operators.cloud_storage_transfer_service` - * :mod:`airflow.gcp.sensors.cloud_storage_transfer_service` - -They also use :class:`airflow.gcp.hooks.cloud_storage_transfer_service.GCPTransferServiceHook` to communicate with Google Cloud Platform. - - -Cloud Vision - - -The operator is defined in the :class:`airflow.gcp.operators.vision` package. - -They also use :class:`airflow.gcp.hooks.vision.CloudVisionHook` to communicate with Google Cloud Platform. - -Cloud Text to Speech - - -The operator is defined in the :class:`airflow.gcp.operators.text_to_speech` package. - -They also use :class:`airflow.gcp.hooks.text_to_speech.GCPTextToSpeechHook` to communicate with Google Cloud Platform. - -Cloud Speech to Text - - -The operator is defined in the :class:`airflow.gcp.operators.speech_to_text`
[GitHub] [airflow] mik-laj commented on a change in pull request #6086: [AIRFLOW-XXX] Display GCP integration in table
mik-laj commented on a change in pull request #6086: [AIRFLOW-XXX] Display GCP integration in table URL: https://github.com/apache/airflow/pull/6086#discussion_r324467572 ## File path: docs/integration.rst ## @@ -237,230 +235,290 @@ Airflow can be configured to read and write task logs in Google Cloud Storage. See :ref:`write-logs-gcp`. -GoogleCloudBaseHook +Operators and Hooks ''' All hooks are based on :class:`airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook`. - -BigQuery - - -The operators are defined in the following modules: - * :mod:`airflow.gcp.operators.bigquery` - * :mod:`airflow.gcp.sensors.bigquery` - * :mod:`airflow.operators.bigquery_to_bigquery` - * :mod:`airflow.operators.bigquery_to_gcs` - * :mod:`airflow.operators.bigquery_to_mysql` - -They also use :class:`airflow.gcp.hooks.bigquery.BigQueryHook` to communicate with Google Cloud Platform. - -BigQuery Data Transfer Service -'' -The operators are defined in the following module: - - * :mod:`airflow.gcp.operators.bigquery_dts` - * :mod:`airflow.gcp.sensors.bigquery_dts` - -The operator is defined in the :class:`airflow.gcp.operators.spanner` package. - -They also use :class:`airflow.gcp.hooks.bigquery_dts.BiqQueryDataTransferServiceHook` to communicate with Google Cloud Platform. - - -Cloud Spanner -' - -The operator is defined in the :class:`airflow.gcp.operators.spanner` package. - -They also use :class:`airflow.gcp.hooks.spanner.CloudSpannerHook` to communicate with Google Cloud Platform. - - -Cloud SQL -' - -The operator is defined in the :class:`airflow.gcp.operators.cloud_sql` package. - -They also use :class:`airflow.gcp.hooks.cloud_sql.CloudSqlDatabaseHook` and :class:`airflow.gcp.hooks.cloud_sql.CloudSqlHook` to communicate with Google Cloud Platform. - - -Cloud Bigtable -'' - -The operator is defined in the :class:`airflow.gcp.operators.bigtable` package. - -They also use :class:`airflow.gcp.hooks.bigtable.BigtableHook` to communicate with Google Cloud Platform. - -Cloud Build -''' - -The operator is defined in the :class:`airflow.gcp.operators.cloud_build` package. - -They also use :class:`airflow.gcp.hooks.cloud_build.CloudBuildHook` to communicate with Google Cloud Platform. - - -Compute Engine -'' - -The operators are defined in the :class:`airflow.gcp.operators.compute` package. - -They also use :class:`airflow.gcp.hooks.compute.GceHook` to communicate with Google Cloud Platform. - - -Cloud Functions -''' - -The operators are defined in the :class:`airflow.gcp.operators.functions` package. - -They also use :class:`airflow.gcp.hooks.functions.GcfHook` to communicate with Google Cloud Platform. - - -Cloud DataFlow -'' - -The operators are defined in the :class:`airflow.gcp.operators.dataflow` package. - -They also use :class:`airflow.gcp.hooks.dataflow.DataFlowHook` to communicate with Google Cloud Platform. - - -Cloud DataProc -'' - -The operators are defined in the :class:`airflow.gcp.operators.dataproc` package. - - -Cloud Datastore -''' - -The operators are defined in the :class:`airflow.gcp.operators.datastore` package. - -They also use :class:`airflow.gcp.hooks.datastore.DatastoreHook` to communicate with Google Cloud Platform. - - -Cloud ML Engine -''' - -The operators are defined in the :class:`airflow.gcp.operators.mlengine` package. - -They also use :class:`airflow.gcp.hooks.mlengine.MLEngineHook` to communicate with Google Cloud Platform. - -Cloud Storage -' - -The operators are defined in the following module: - - * :mod:`airflow.operators.local_to_gcs` - * :mod:`airflow.gcp.operators.gcs` - * :mod:`airflow.operators.gcs_to_bq` - * :mod:`airflow.operators.mysql_to_gcs` - * :mod:`airflow.gcp.sensors.gcs` - -They also use :class:`airflow.gcp.hooks.gcs.GoogleCloudStorageHook` to communicate with Google Cloud Platform. - - -Transfer Service - - -The operators are defined in the following module: - - * :mod:`airflow.gcp.operators.cloud_storage_transfer_service` - * :mod:`airflow.gcp.sensors.cloud_storage_transfer_service` - -They also use :class:`airflow.gcp.hooks.cloud_storage_transfer_service.GCPTransferServiceHook` to communicate with Google Cloud Platform. - - -Cloud Vision - - -The operator is defined in the :class:`airflow.gcp.operators.vision` package. - -They also use :class:`airflow.gcp.hooks.vision.CloudVisionHook` to communicate with Google Cloud Platform. - -Cloud Text to Speech - - -The operator is defined in the :class:`airflow.gcp.operators.text_to_speech` package. - -They also use :class:`airflow.gcp.hooks.text_to_speech.GCPTextToSpeechHook` to communicate with Google Cloud Platform. - -Cloud Speech to Text - - -The operator is defined in the :class:`airflow.gcp.operators.speech_to_text`
[GitHub] [airflow] clacke commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill
clacke commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill URL: https://github.com/apache/airflow/pull/5498#issuecomment-531573202 All DAGs are Schedule: None and the parent is triggered externally with parameters using [the REST plugin](https://github.com/teamclairvoyant/airflow-rest-api-plugin), which is supposed to be equivalent to running `airflow trigger_dag` using the CLI. I don't know which version of the plugin we are using, maybe that's something to look at as well. I'm glad to hear Airflow is not *supposed* to be acting the way I'm seeing it act. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5150) Implement POC of GitLab CI + GKE integration
[
https://issues.apache.org/jira/browse/AIRFLOW-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930163#comment-16930163
]
ASF GitHub Bot commented on AIRFLOW-5150:
-
potiuk commented on pull request #6115: [AIRFLOW-5150] Implement POC for GitLab
+ Kubernetes tests
URL: https://github.com/apache/airflow/pull/6115
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Implement POC of GitLab CI + GKE integration
>
>
> Key: AIRFLOW-5150
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5150
> Project: Apache Airflow
> Issue Type: Improvement
> Components: ci
>Affects Versions: 2.0.0
>Reporter: Jarek Potiuk
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5150) Implement POC of GitLab CI + GKE integration
[ https://issues.apache.org/jira/browse/AIRFLOW-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930170#comment-16930170 ] ASF subversion and git services commented on AIRFLOW-5150: -- Commit e640aa7f339db70be24096d08754b43470d7c36f in airflow's branch refs/heads/test-gitlab-ci from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=e640aa7 ] [AIRFLOW-5150] Implement POC for GitLab + Kubernetes tests > Implement POC of GitLab CI + GKE integration > > > Key: AIRFLOW-5150 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5150 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5150) Implement POC of GitLab CI + GKE integration
[ https://issues.apache.org/jira/browse/AIRFLOW-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930177#comment-16930177 ] ASF subversion and git services commented on AIRFLOW-5150: -- Commit 2de91e180764d42018a126805e727c0c21e3712a in airflow's branch refs/heads/test-gitlab-ci from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=2de91e1 ] [AIRFLOW-5150] Implement POC for GitLab + Kubernetes tests > Implement POC of GitLab CI + GKE integration > > > Key: AIRFLOW-5150 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5150 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [airflow] potiuk commented on a change in pull request #5837: [AIRFLOW-5223] Use KinD for Kubernetes in CI
potiuk commented on a change in pull request #5837: [AIRFLOW-5223] Use KinD for
Kubernetes in CI
URL: https://github.com/apache/airflow/pull/5837#discussion_r324486045
##
File path: scripts/ci/kubernetes/docker/build.sh
##
@@ -20,29 +20,14 @@ IMAGE=${IMAGE:-airflow}
TAG=${TAG:-latest}
DIRNAME=$(cd "$(dirname "$0")" && pwd)
AIRFLOW_ROOT="${DIRNAME}/../../../.."
-PYTHON_DOCKER_IMAGE=python:3.6-slim
set -e
-# Don't rebuild the image more than once on travis
-if [[ -n "${TRAVIS}" || -z "${AIRFLOW_CI_REUSE_K8S_IMAGE}" ]] && \
-docker image inspect "${IMAGE}:${TAG}" > /dev/null 2>/dev/null; then
- echo "Re-using existing image"
- exit 0
-fi
-
-if [[ "${VM_DRIVER:-none}" != "none" ]]; then
-if ENVCONFIG=$(minikube docker-env); then
- eval "${ENVCONFIG}"
-fi
-fi
-
echo "Airflow directory ${AIRFLOW_ROOT}"
echo "Airflow Docker directory ${DIRNAME}"
cd "${AIRFLOW_ROOT}"
-docker run -ti --rm -v "${AIRFLOW_ROOT}:/airflow" \
--w /airflow "${PYTHON_DOCKER_IMAGE}"
./scripts/ci/kubernetes/docker/compile.sh
+python setup.py compile_assets sdist -q
Review comment:
I see now. I understand that in your solution you do the same what dind
image does and instead of using an external dind image, you start docker engine
inside the airflow-testing image. I understand this is the idea.
That's a bit different to what initially thought. I think there is a big
value on making the airflow-testing image without dind-modifications. Instead
it could use docker engine run outside of the airflow-testing image in separate
`dind` image. The /var/lib/docker.sock is indeed not a good solution either - I
think using the docker engine from host in general is not a good idea
(especially if we move to GitLabCI/CloudBuild when we run everything inside
Kubernetes already). But we have another option - we can start separate ding
image and connect to it using DOCKER_HOST configuration. It will be using TCP
rather than socket, but the performance penalty should be small.
I've implemented something similar in GitLab CI POC. You can see the
implementation here: https://github.com/apache/airflow/pull/6115 . It might be
an inspiration for you:
* it does not use pass /var/lib/docker.sock (for the reasons mentioned by
you)
* it starts a separate `dind` image before running docker-compose command
(as a service in case of GitLab CI). This way from within kubernetes we can
reach it using DOCKER_HOST="tcp://localhost:2375" and dind image is run in the
same network (network_mode = host) as the script that runs docker-compose.
* I used it for a different purpose - I used it to build docker images from
within Kubernetes cluster. In case of GitLab build, the images are built by
another docker image (Docker-builder) which uses DOCKER_HOST to connect to the
docker engine run as dind image.
* But I think you could use the very same approach for kind. When you run
docker-compose, you could run the `dind` image as a separate image and connect
to it from within airflow-testing image (setting DOCKER_HOST appropriately).
I think this will be very small change to your implementation - primary
difference will be that you will not have to start/stop docker service - it
will be already running (this was the source of my original comment). And you
would not have to have /var/lib/docker as volume (it will be done in dind
image already).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #5837: [AIRFLOW-5223] Use KinD for Kubernetes in CI
potiuk commented on a change in pull request #5837: [AIRFLOW-5223] Use KinD for
Kubernetes in CI
URL: https://github.com/apache/airflow/pull/5837#discussion_r324486346
##
File path: Dockerfile
##
@@ -85,6 +92,24 @@ RUN curl -sL https://deb.nodesource.com/setup_10.x | bash -
\
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
+# Kubernetes dependencies
+RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add - \
+&& add-apt-repository "deb [arch=amd64]
https://download.docker.com/linux/debian stretch stable" \
+&& apt-get update \
+&& apt-get -y install --no-install-recommends docker-ce \
+&& apt-get autoremove -yqq --purge \
+&& apt-get clean && rm -rf /var/lib/apt/lists/*
+
+RUN curl -Lo kubectl \
+
"https://storage.googleapis.com/kubernetes-release/release/${KUBERNETES_VERSION}/bin/linux/amd64/kubectl;
\
+&& chmod +x kubectl \
+&& mv kubectl /usr/local/bin/kubectl
+
+RUN curl -Lo kind \
+
"https://github.com/kubernetes-sigs/kind/releases/download/${KIND_VERSION}/kind-linux-amd64;
\
+&& chmod +x kind \
+&& mv kind /usr/local/bin/kind
+
Review comment:
Yeah. I did not realise that minicluster was a different thing. I moved it
indeed in the meantime . I think this will be part of a different PR to use
micluster/hadoop etc. images separately from airflow testing. But let's leave
it for later.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #5837: [AIRFLOW-5223] Use KinD for Kubernetes in CI
potiuk commented on a change in pull request #5837: [AIRFLOW-5223] Use KinD for Kubernetes in CI URL: https://github.com/apache/airflow/pull/5837#discussion_r324486249 ## File path: Dockerfile ## @@ -372,4 +397,6 @@ EXPOSE 8080 ENTRYPOINT ["/usr/local/bin/dumb-init", "--", "/entrypoint.sh"] +VOLUME /var/lib/docker Review comment: As explained in further comment - I think we do not have to do this and we can run dind image separately rather than running the docker image inside airflow-testing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5150) Implement POC of GitLab CI + GKE integration
[ https://issues.apache.org/jira/browse/AIRFLOW-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930167#comment-16930167 ] ASF subversion and git services commented on AIRFLOW-5150: -- Commit 264ffe57dbd9f1870c10e0e800e29d024549dd52 in airflow's branch refs/heads/test-gitlab-ci from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=264ffe5 ] [AIRFLOW-5150] Implement POC for GitLab + Kubernetes tests > Implement POC of GitLab CI + GKE integration > > > Key: AIRFLOW-5150 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5150 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [airflow] potiuk commented on a change in pull request #5786: [AIRFLOW-5170] [AIRFLOW-5256] Consistent licences for python files and related pylint fixes
potiuk commented on a change in pull request #5786: [AIRFLOW-5170] [AIRFLOW-5256] Consistent licences for python files and related pylint fixes URL: https://github.com/apache/airflow/pull/5786#discussion_r324488944 ## File path: airflow/kubernetes/volume.py ## @@ -25,20 +25,22 @@ class Volume(K8SModel): +""" +Adds Kubernetes Volume to pod. allows pod to access features like ConfigMaps +and Persistent Volumes Review comment: Right! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #5786: [AIRFLOW-5170] [AIRFLOW-5256] Consistent licences for python files and related pylint fixes
potiuk commented on a change in pull request #5786: [AIRFLOW-5170] [AIRFLOW-5256] Consistent licences for python files and related pylint fixes URL: https://github.com/apache/airflow/pull/5786#discussion_r324488969 ## File path: airflow/kubernetes/volume.py ## @@ -25,20 +25,22 @@ class Volume(K8SModel): +""" +Adds Kubernetes Volume to pod. allows pod to access features like ConfigMaps +and Persistent Volumes +:param name: the name of the volume mount +:type name: str +:param configs: dictionary of any features needed for volume. +We purposely keep this vague since there are multiple volume types with changing Review comment: Fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] jasonlin2268 opened a new pull request #6116: Merge pull request #1 from apache/master
jasonlin2268 opened a new pull request #6116: Merge pull request #1 from
apache/master
URL: https://github.com/apache/airflow/pull/6116
fix
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] dstandish commented on a change in pull request #6104: [AIRFLOW-4574] allow providing private_key in SSHHook
dstandish commented on a change in pull request #6104: [AIRFLOW-4574] allow providing private_key in SSHHook URL: https://github.com/apache/airflow/pull/6104#discussion_r324483395 ## File path: airflow/contrib/hooks/ssh_hook.py ## @@ -160,24 +169,27 @@ def get_conn(self): 'against Man-In-The-Middle attacks') # Default is RejectPolicy client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) - -if self.password and self.password.strip(): -client.connect(hostname=self.remote_host, - username=self.username, - password=self.password, - key_filename=self.key_file, - timeout=self.timeout, - compress=self.compress, - port=self.port, - sock=self.host_proxy) -else: -client.connect(hostname=self.remote_host, - username=self.username, - key_filename=self.key_file, - timeout=self.timeout, - compress=self.compress, - port=self.port, - sock=self.host_proxy) +connect_kwargs = dict( +hostname=self.remote_host, +username=self.username, +timeout=self.timeout, +compress=self.compress, +port=self.port, +sock=self.host_proxy +) + +if self.password: +password = self.password.strip() +connect_kwargs.update(password=password) + +# prefer pkey over key_filename when both are given Review comment: Admittedly I was on the fence about this too. Ultimately of course I defer to you. **Reasoning for picking one** I guess I don't see the harm in trying at least one of them. I figured choosing one was better because it would at least try one of them, therefore it would fail in fewer circumstances. Though I understand throwing error would force user to resolve ambiguity. **Why pkey, if picking one** The choice of which one to pick, assuming we were to choose one, is probably less controversial: choosing the private key is better because the private key is actually a private key, while the path to file is just a path, and the file may or may not be there. **What does paramiko do?** I was curious and looked into paramiko. What does it do when given both? It appears that it picks pkey, but it's not super obvious to me: https://github.com/paramiko/paramiko/blob/master/paramiko/client.py#L655 **Suggestion** Perhaps better yet, is when given both, then pass both to paramiko, and let it do whatever it does. What you think? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dstandish commented on a change in pull request #6104: [AIRFLOW-4574] allow providing private_key in SSHHook
dstandish commented on a change in pull request #6104: [AIRFLOW-4574] allow providing private_key in SSHHook URL: https://github.com/apache/airflow/pull/6104#discussion_r324483395 ## File path: airflow/contrib/hooks/ssh_hook.py ## @@ -160,24 +169,27 @@ def get_conn(self): 'against Man-In-The-Middle attacks') # Default is RejectPolicy client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) - -if self.password and self.password.strip(): -client.connect(hostname=self.remote_host, - username=self.username, - password=self.password, - key_filename=self.key_file, - timeout=self.timeout, - compress=self.compress, - port=self.port, - sock=self.host_proxy) -else: -client.connect(hostname=self.remote_host, - username=self.username, - key_filename=self.key_file, - timeout=self.timeout, - compress=self.compress, - port=self.port, - sock=self.host_proxy) +connect_kwargs = dict( +hostname=self.remote_host, +username=self.username, +timeout=self.timeout, +compress=self.compress, +port=self.port, +sock=self.host_proxy +) + +if self.password: +password = self.password.strip() +connect_kwargs.update(password=password) + +# prefer pkey over key_filename when both are given Review comment: Admittedly I was on the fence about this too. Ultimately of course I defer to you. **Reasoning for picking one** I guess I don't see the harm in trying at least one of them. I figured choosing one was better because it would at least try one of them, therefore it would fail in fewer circumstances. Though I understand throwing error would force user to resolve ambiguity. **Why pkey, if picking one** The choice of which one to pick, assuming we were to choose one, is probably less controversial: choosing the private key is better because the private key is actually a private key, while the path to file is just a path, and the file may or may not be there. **What does paramiko do?** I was curious and looked into paramiko. What does it do when given both? It appears that it picks pkey, but it's not super obvious to me: https://github.com/paramiko/paramiko/blob/master/paramiko/client.py#L655 **Proposal** Perhaps better yet, is when given both, then pass both to paramiko, and let it do whatever it does. What you think? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #5837: [AIRFLOW-5223] Use KinD for Kubernetes in CI
potiuk commented on issue #5837: [AIRFLOW-5223] Use KinD for Kubernetes in CI URL: https://github.com/apache/airflow/pull/5837#issuecomment-531608665 Hey @gerardo -> I think maybe you can take a look at running the dind image separately. It would be simplification and it will make the airflow-testing images not having to run docker engine inside. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #5837: [AIRFLOW-5223] Use KinD for Kubernetes in CI
potiuk commented on a change in pull request #5837: [AIRFLOW-5223] Use KinD for
Kubernetes in CI
URL: https://github.com/apache/airflow/pull/5837#discussion_r324486431
##
File path: scripts/ci/local_ci_stop_environment.sh
##
@@ -45,7 +45,6 @@ export HOST_GROUP_ID
docker-compose \
-f "${MY_DIR}/docker-compose.yml" \
--f "${MY_DIR}/docker-compose-kubernetes.yml" \
Review comment:
Good!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-5150) Implement POC of GitLab CI + GKE integration
[ https://issues.apache.org/jira/browse/AIRFLOW-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930174#comment-16930174 ] ASF subversion and git services commented on AIRFLOW-5150: -- Commit f83f7e041c8c9c349a531082684bd5136c91fe6d in airflow's branch refs/heads/test-gitlab-ci from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f83f7e0 ] [AIRFLOW-5150] Implement POC for GitLab + Kubernetes tests > Implement POC of GitLab CI + GKE integration > > > Key: AIRFLOW-5150 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5150 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (AIRFLOW-5447) KubernetesExecutor hangs on task queueing
[ https://issues.apache.org/jira/browse/AIRFLOW-5447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930224#comment-16930224 ] Daniel Imberman edited comment on AIRFLOW-5447 at 9/16/19 4:21 AM: --- [~Yuval.Itzchakov] [~cwegrzyn]Thank you guys for getting this info to us. I THINK this might have to do with a bug in the k8s python client which requires "create" and "get" privileges for "pods/exec" [https://stackoverflow.com/questions/53827345/airflow-k8s-operator-xcom-handshake-status-403-forbidden] [https://github.com/kubernetes-client/python/issues/690] The reason I believe this is that this lack of running/updating of pods point to a failure or the KubernetesJobWatcher. When we finally started seeing similar problems we were seeing these failures from the JobWatcher [https://user-images.githubusercontent.com/1036482/64914385-2f0eca80-d71e-11e9-8f8b-44a1c8620b92.png]. I'm going to look into this further tomorrow and get back ASAP. was (Author: dimberman): [~Yuval.Itzchakov] [~cwegrzyn]Thank you guys for getting this info to us. I THINK this might have to do with a bug in the k8s kubernetes client which requires "create" and "get" privileges for "pods/exec" [https://stackoverflow.com/questions/53827345/airflow-k8s-operator-xcom-handshake-status-403-forbidden] [https://github.com/kubernetes-client/python/issues/690] The reason I believe this is that this lack of running/updating of pods point to a failure or the KubernetesJobWatcher. When we finally started seeing similar problems we were seeing these failures from the JobWatcher [https://user-images.githubusercontent.com/1036482/64914385-2f0eca80-d71e-11e9-8f8b-44a1c8620b92.png]. I'm going to look into this further tomorrow and get back ASAP. > KubernetesExecutor hangs on task queueing > - > > Key: AIRFLOW-5447 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5447 > Project: Apache Airflow > Issue Type: Bug > Components: executor-kubernetes >Affects Versions: 1.10.4, 1.10.5 > Environment: Kubernetes version v1.14.3, Airflow version 1.10.4-1.10.5 >Reporter: Henry Cohen >Assignee: Daniel Imberman >Priority: Blocker > > Starting in 1.10.4, and continuing in 1.10.5, when using the > KubernetesExecutor, with the webserver and scheduler running in the > kubernetes cluster, tasks are scheduled, but when added to the task queue, > the executor process hangs indefinitely. Based on log messages, it appears to > be stuck at this line > https://github.com/apache/airflow/blob/v1-10-stable/airflow/contrib/executors/kubernetes_executor.py#L761 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5447) KubernetesExecutor hangs on task queueing
[ https://issues.apache.org/jira/browse/AIRFLOW-5447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930224#comment-16930224 ] Daniel Imberman commented on AIRFLOW-5447: -- [~Yuval.Itzchakov] [~cwegrzyn]Thank you guys for getting this info to us. I THINK this might have to do with a bug in the k8s kubernetes client which requires "create" and "get" privileges for "pods/exec" [https://stackoverflow.com/questions/53827345/airflow-k8s-operator-xcom-handshake-status-403-forbidden] [https://github.com/kubernetes-client/python/issues/690] The reason I believe this is that this lack of running/updating of pods point to a failure or the KubernetesJobWatcher. When we finally started seeing similar problems we were seeing these failures from the JobWatcher [https://user-images.githubusercontent.com/1036482/64914385-2f0eca80-d71e-11e9-8f8b-44a1c8620b92.png]. I'm going to look into this further tomorrow and get back ASAP. > KubernetesExecutor hangs on task queueing > - > > Key: AIRFLOW-5447 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5447 > Project: Apache Airflow > Issue Type: Bug > Components: executor-kubernetes >Affects Versions: 1.10.4, 1.10.5 > Environment: Kubernetes version v1.14.3, Airflow version 1.10.4-1.10.5 >Reporter: Henry Cohen >Assignee: Daniel Imberman >Priority: Blocker > > Starting in 1.10.4, and continuing in 1.10.5, when using the > KubernetesExecutor, with the webserver and scheduler running in the > kubernetes cluster, tasks are scheduled, but when added to the task queue, > the executor process hangs indefinitely. Based on log messages, it appears to > be stuck at this line > https://github.com/apache/airflow/blob/v1-10-stable/airflow/contrib/executors/kubernetes_executor.py#L761 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [airflow] potiuk commented on a change in pull request #5837: [AIRFLOW-5223] Use KinD for Kubernetes in CI
potiuk commented on a change in pull request #5837: [AIRFLOW-5223] Use KinD for Kubernetes in CI URL: https://github.com/apache/airflow/pull/5837#discussion_r324486402 ## File path: scripts/ci/pylint_todo.txt ## @@ -462,8 +462,8 @@ ./tests/lineage/backend/test_atlas.py ./tests/lineage/test_lineage.py ./tests/macros/test_hive.py -./tests/minikube/test_kubernetes_executor.py -./tests/minikube/test_kubernetes_pod_operator.py +./tests/integration/kubernetes/test_kubernetes_executor.py Review comment: One more thing - you can now also remove it it from pylint_todo_txt (if all pylint issues are fixed) - this way it will be checked during pre-commits/travis build. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5150) Implement POC of GitLab CI + GKE integration
[ https://issues.apache.org/jira/browse/AIRFLOW-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930168#comment-16930168 ] ASF subversion and git services commented on AIRFLOW-5150: -- Commit 58fc65dda740fd5b06d590f7af0ad2ca308146a8 in airflow's branch refs/heads/test-gitlab-ci from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=58fc65d ] [AIRFLOW-5150] Implement POC for GitLab + Kubernetes tests > Implement POC of GitLab CI + GKE integration > > > Key: AIRFLOW-5150 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5150 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [airflow] potiuk commented on issue #5976: [AIRFLOW-5369] Add interactivity to pre-commits
potiuk commented on issue #5976: [AIRFLOW-5369] Add interactivity to pre-commits URL: https://github.com/apache/airflow/pull/5976#issuecomment-531610072 Hey @dimberman :)? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #5786: [AIRFLOW-5170] [AIRFLOW-5256] Consistent licences for python files and related pylint fixes
potiuk commented on issue #5786: [AIRFLOW-5170] [AIRFLOW-5256] Consistent licences for python files and related pylint fixes URL: https://github.com/apache/airflow/pull/5786#issuecomment-531613726 @feluelle -> I think it will be green soon:) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-5453) Improve reading inputs from Dataflow console
[ https://issues.apache.org/jira/browse/AIRFLOW-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5453. --- Fix Version/s: 2.0.0 Resolution: Fixed > Improve reading inputs from Dataflow console > > > Key: AIRFLOW-5453 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5453 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [airflow] potiuk commented on issue #6073: [AIRFLOW-5453] Improve reading inputs from Dataflow console
potiuk commented on issue #6073: [AIRFLOW-5453] Improve reading inputs from Dataflow console URL: https://github.com/apache/airflow/pull/6073#issuecomment-531589265 Already merged in separate PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk closed pull request #6073: [AIRFLOW-5453] Improve reading inputs from Dataflow console
potiuk closed pull request #6073: [AIRFLOW-5453] Improve reading inputs from Dataflow console URL: https://github.com/apache/airflow/pull/6073 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6086: [AIRFLOW-XXX] Display GCP integration in table
mik-laj commented on a change in pull request #6086: [AIRFLOW-XXX] Display GCP integration in table URL: https://github.com/apache/airflow/pull/6086#discussion_r324476450 ## File path: docs/integration.rst ## @@ -237,230 +235,290 @@ Airflow can be configured to read and write task logs in Google Cloud Storage. See :ref:`write-logs-gcp`. -GoogleCloudBaseHook +Operators and Hooks ''' All hooks are based on :class:`airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook`. - -BigQuery - - -The operators are defined in the following modules: - * :mod:`airflow.gcp.operators.bigquery` - * :mod:`airflow.gcp.sensors.bigquery` - * :mod:`airflow.operators.bigquery_to_bigquery` - * :mod:`airflow.operators.bigquery_to_gcs` - * :mod:`airflow.operators.bigquery_to_mysql` - -They also use :class:`airflow.gcp.hooks.bigquery.BigQueryHook` to communicate with Google Cloud Platform. - -BigQuery Data Transfer Service -'' -The operators are defined in the following module: - - * :mod:`airflow.gcp.operators.bigquery_dts` - * :mod:`airflow.gcp.sensors.bigquery_dts` - -The operator is defined in the :class:`airflow.gcp.operators.spanner` package. - -They also use :class:`airflow.gcp.hooks.bigquery_dts.BiqQueryDataTransferServiceHook` to communicate with Google Cloud Platform. - - -Cloud Spanner -' - -The operator is defined in the :class:`airflow.gcp.operators.spanner` package. - -They also use :class:`airflow.gcp.hooks.spanner.CloudSpannerHook` to communicate with Google Cloud Platform. - - -Cloud SQL -' - -The operator is defined in the :class:`airflow.gcp.operators.cloud_sql` package. - -They also use :class:`airflow.gcp.hooks.cloud_sql.CloudSqlDatabaseHook` and :class:`airflow.gcp.hooks.cloud_sql.CloudSqlHook` to communicate with Google Cloud Platform. - - -Cloud Bigtable -'' - -The operator is defined in the :class:`airflow.gcp.operators.bigtable` package. - -They also use :class:`airflow.gcp.hooks.bigtable.BigtableHook` to communicate with Google Cloud Platform. - -Cloud Build -''' - -The operator is defined in the :class:`airflow.gcp.operators.cloud_build` package. - -They also use :class:`airflow.gcp.hooks.cloud_build.CloudBuildHook` to communicate with Google Cloud Platform. - - -Compute Engine -'' - -The operators are defined in the :class:`airflow.gcp.operators.compute` package. - -They also use :class:`airflow.gcp.hooks.compute.GceHook` to communicate with Google Cloud Platform. - - -Cloud Functions -''' - -The operators are defined in the :class:`airflow.gcp.operators.functions` package. - -They also use :class:`airflow.gcp.hooks.functions.GcfHook` to communicate with Google Cloud Platform. - - -Cloud DataFlow -'' - -The operators are defined in the :class:`airflow.gcp.operators.dataflow` package. - -They also use :class:`airflow.gcp.hooks.dataflow.DataFlowHook` to communicate with Google Cloud Platform. - - -Cloud DataProc -'' - -The operators are defined in the :class:`airflow.gcp.operators.dataproc` package. - - -Cloud Datastore -''' - -The operators are defined in the :class:`airflow.gcp.operators.datastore` package. - -They also use :class:`airflow.gcp.hooks.datastore.DatastoreHook` to communicate with Google Cloud Platform. - - -Cloud ML Engine -''' - -The operators are defined in the :class:`airflow.gcp.operators.mlengine` package. - -They also use :class:`airflow.gcp.hooks.mlengine.MLEngineHook` to communicate with Google Cloud Platform. - -Cloud Storage -' - -The operators are defined in the following module: - - * :mod:`airflow.operators.local_to_gcs` - * :mod:`airflow.gcp.operators.gcs` - * :mod:`airflow.operators.gcs_to_bq` - * :mod:`airflow.operators.mysql_to_gcs` - * :mod:`airflow.gcp.sensors.gcs` - -They also use :class:`airflow.gcp.hooks.gcs.GoogleCloudStorageHook` to communicate with Google Cloud Platform. - - -Transfer Service - - -The operators are defined in the following module: - - * :mod:`airflow.gcp.operators.cloud_storage_transfer_service` - * :mod:`airflow.gcp.sensors.cloud_storage_transfer_service` - -They also use :class:`airflow.gcp.hooks.cloud_storage_transfer_service.GCPTransferServiceHook` to communicate with Google Cloud Platform. - - -Cloud Vision - - -The operator is defined in the :class:`airflow.gcp.operators.vision` package. - -They also use :class:`airflow.gcp.hooks.vision.CloudVisionHook` to communicate with Google Cloud Platform. - -Cloud Text to Speech - - -The operator is defined in the :class:`airflow.gcp.operators.text_to_speech` package. - -They also use :class:`airflow.gcp.hooks.text_to_speech.GCPTextToSpeechHook` to communicate with Google Cloud Platform. - -Cloud Speech to Text - - -The operator is defined in the :class:`airflow.gcp.operators.speech_to_text`
[GitHub] [airflow] adankro opened a new pull request #6105: [AIRFLOW-5495] Remove unneeded parens in dataproc.py
adankro opened a new pull request #6105: [AIRFLOW-5495] Remove unneeded parens
in dataproc.py
URL: https://github.com/apache/airflow/pull/6105
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-5495) Remove unneeded parens in dataproc.py
[
https://issues.apache.org/jira/browse/AIRFLOW-5495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930082#comment-16930082
]
ASF GitHub Bot commented on AIRFLOW-5495:
-
adankro commented on pull request #6105: [AIRFLOW-5495] Remove unneeded parens
in dataproc.py
URL: https://github.com/apache/airflow/pull/6105
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Remove unneeded parens in dataproc.py
> -
>
> Key: AIRFLOW-5495
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5495
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.10.6
>Reporter: Jakob Homan
>Assignee: Adan Christian Rosales Ornelas
>Priority: Minor
> Labels: ccoss2019, newbie
>
> Note: This ticket's being created to facilitate a new contributor's workshop
> for Airflow. After the workshop has completed, I'll mark these all available
> for anyone that might like to take them on.
> The parens around {{self.custom_image_project_id}} don't need to be there; we
> should remove them.
> airflow/gcp/operators/dataproc.py:409
> {code:java}
> elif self.custom_image:
> project_id = self.custom_image_project_id if
> (self.custom_image_project_id) else self.project_id
> custom_image_url = 'https://www.googleapis.com/compute/beta/projects/' \
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [airflow] adankro commented on issue #6105: [AIRFLOW-5495] Remove unneeded parens in dataproc.py
adankro commented on issue #6105: [AIRFLOW-5495] Remove unneeded parens in dataproc.py URL: https://github.com/apache/airflow/pull/6105#issuecomment-531598199 @jghoman This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] jghoman commented on a change in pull request #6105: [AIRFLOW-5495] Remove unneeded parens in dataproc.py
jghoman commented on a change in pull request #6105: [AIRFLOW-5495] Remove unneeded parens in dataproc.py URL: https://github.com/apache/airflow/pull/6105#discussion_r324480968 ## File path: airflow/gcp/operators/dataproc.py ## @@ -407,7 +407,7 @@ def _build_cluster_data(self): cluster_data['config']['softwareConfig']['imageVersion'] = self.image_version elif self.custom_image: -project_id = self.custom_image_project_id if (self.custom_image_project_id) else self.project_id +project_id = self.custom_image_project_id if self.custom_image_project_id else self.project_id Review comment: Hello from Guadalajara, @mik-laj ! We're running a new contributor hackathon. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6104: [AIRFLOW-4574] allow providing private_key in SSHHook
mik-laj commented on a change in pull request #6104: [AIRFLOW-4574] allow providing private_key in SSHHook URL: https://github.com/apache/airflow/pull/6104#discussion_r324481025 ## File path: airflow/contrib/hooks/ssh_hook.py ## @@ -160,24 +169,27 @@ def get_conn(self): 'against Man-In-The-Middle attacks') # Default is RejectPolicy client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) - -if self.password and self.password.strip(): -client.connect(hostname=self.remote_host, - username=self.username, - password=self.password, - key_filename=self.key_file, - timeout=self.timeout, - compress=self.compress, - port=self.port, - sock=self.host_proxy) -else: -client.connect(hostname=self.remote_host, - username=self.username, - key_filename=self.key_file, - timeout=self.timeout, - compress=self.compress, - port=self.port, - sock=self.host_proxy) +connect_kwargs = dict( +hostname=self.remote_host, +username=self.username, +timeout=self.timeout, +compress=self.compress, +port=self.port, +sock=self.host_proxy +) + +if self.password: +password = self.password.strip() +connect_kwargs.update(password=password) + +# prefer pkey over key_filename when both are given Review comment: Why do you prefer one format over another? I think it's worth throwing an exception if two mutually exclusive parameters are given. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #5837: [AIRFLOW-5223] Use KinD for Kubernetes in CI
potiuk commented on a change in pull request #5837: [AIRFLOW-5223] Use KinD for Kubernetes in CI URL: https://github.com/apache/airflow/pull/5837#discussion_r324481094 ## File path: Dockerfile ## @@ -18,6 +18,8 @@ # Base image for the whole Docker file ARG APT_DEPS_IMAGE="airflow-apt-deps-ci-slim" ARG PYTHON_BASE_IMAGE="python:3.6-slim-stretch" +ARG KUBERNETES_VERSION="v1.15.0" Review comment: Hello @gerardo -> sorry for such big delays but I've been super busy with customer project and I had very little time. Yes. It should be moved to the new ci_buikd_iinstall_deps.sh file. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6105: [AIRFLOW-5495] Remove unneeded parens in dataproc.py
mik-laj commented on a change in pull request #6105: [AIRFLOW-5495] Remove unneeded parens in dataproc.py URL: https://github.com/apache/airflow/pull/6105#discussion_r324481153 ## File path: airflow/gcp/operators/dataproc.py ## @@ -407,7 +407,7 @@ def _build_cluster_data(self): cluster_data['config']['softwareConfig']['imageVersion'] = self.image_version elif self.custom_image: -project_id = self.custom_image_project_id if (self.custom_image_project_id) else self.project_id +project_id = self.custom_image_project_id if self.custom_image_project_id else self.project_id Review comment: Hello from Warsaw, Poland. I wish you lots of fun. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #5837: [AIRFLOW-5223] Use KinD for Kubernetes in CI
potiuk commented on a change in pull request #5837: [AIRFLOW-5223] Use KinD for
Kubernetes in CI
URL: https://github.com/apache/airflow/pull/5837#discussion_r324481177
##
File path: Dockerfile
##
@@ -33,6 +35,9 @@ ENV PYTHON_BASE_IMAGE=${PYTHON_BASE_IMAGE}
ARG AIRFLOW_VERSION="2.0.0.dev0"
ENV AIRFLOW_VERSION=$AIRFLOW_VERSION
+ARG KUBERNETES_VERSION
Review comment:
Same here -> the new script is there to speed up the build slightly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] jghoman merged pull request #6105: [AIRFLOW-5495] Remove unneeded parens in dataproc.py
jghoman merged pull request #6105: [AIRFLOW-5495] Remove unneeded parens in dataproc.py URL: https://github.com/apache/airflow/pull/6105 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5495) Remove unneeded parens in dataproc.py
[
https://issues.apache.org/jira/browse/AIRFLOW-5495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930091#comment-16930091
]
ASF subversion and git services commented on AIRFLOW-5495:
--
Commit 7351df8be5975dc7c4438d81f2d2bc35848a3900 in airflow's branch
refs/heads/master from Adan Rosales
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=7351df8 ]
[AIRFLOW-5495] Remove unneeded parens in dataproc.py (#6105)
* AIRFLOW-5495 removing the parens
* [AIRFLOW-5495] Update airflow/gcp/operators/dataproc.py
Co-Authored-By: Kamil Breguła
> Remove unneeded parens in dataproc.py
> -
>
> Key: AIRFLOW-5495
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5495
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.10.6
>Reporter: Jakob Homan
>Assignee: Adan Christian Rosales Ornelas
>Priority: Minor
> Labels: ccoss2019, newbie
>
> Note: This ticket's being created to facilitate a new contributor's workshop
> for Airflow. After the workshop has completed, I'll mark these all available
> for anyone that might like to take them on.
> The parens around {{self.custom_image_project_id}} don't need to be there; we
> should remove them.
> airflow/gcp/operators/dataproc.py:409
> {code:java}
> elif self.custom_image:
> project_id = self.custom_image_project_id if
> (self.custom_image_project_id) else self.project_id
> custom_image_url = 'https://www.googleapis.com/compute/beta/projects/' \
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5495) Remove unneeded parens in dataproc.py
[
https://issues.apache.org/jira/browse/AIRFLOW-5495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930092#comment-16930092
]
ASF subversion and git services commented on AIRFLOW-5495:
--
Commit 7351df8be5975dc7c4438d81f2d2bc35848a3900 in airflow's branch
refs/heads/master from Adan Rosales
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=7351df8 ]
[AIRFLOW-5495] Remove unneeded parens in dataproc.py (#6105)
* AIRFLOW-5495 removing the parens
* [AIRFLOW-5495] Update airflow/gcp/operators/dataproc.py
Co-Authored-By: Kamil Breguła
> Remove unneeded parens in dataproc.py
> -
>
> Key: AIRFLOW-5495
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5495
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.10.6
>Reporter: Jakob Homan
>Assignee: Adan Christian Rosales Ornelas
>Priority: Minor
> Labels: ccoss2019, newbie
>
> Note: This ticket's being created to facilitate a new contributor's workshop
> for Airflow. After the workshop has completed, I'll mark these all available
> for anyone that might like to take them on.
> The parens around {{self.custom_image_project_id}} don't need to be there; we
> should remove them.
> airflow/gcp/operators/dataproc.py:409
> {code:java}
> elif self.custom_image:
> project_id = self.custom_image_project_id if
> (self.custom_image_project_id) else self.project_id
> custom_image_url = 'https://www.googleapis.com/compute/beta/projects/' \
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5495) Remove unneeded parens in dataproc.py
[
https://issues.apache.org/jira/browse/AIRFLOW-5495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930090#comment-16930090
]
ASF subversion and git services commented on AIRFLOW-5495:
--
Commit 7351df8be5975dc7c4438d81f2d2bc35848a3900 in airflow's branch
refs/heads/master from Adan Rosales
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=7351df8 ]
[AIRFLOW-5495] Remove unneeded parens in dataproc.py (#6105)
* AIRFLOW-5495 removing the parens
* [AIRFLOW-5495] Update airflow/gcp/operators/dataproc.py
Co-Authored-By: Kamil Breguła
> Remove unneeded parens in dataproc.py
> -
>
> Key: AIRFLOW-5495
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5495
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.10.6
>Reporter: Jakob Homan
>Assignee: Adan Christian Rosales Ornelas
>Priority: Minor
> Labels: ccoss2019, newbie
>
> Note: This ticket's being created to facilitate a new contributor's workshop
> for Airflow. After the workshop has completed, I'll mark these all available
> for anyone that might like to take them on.
> The parens around {{self.custom_image_project_id}} don't need to be there; we
> should remove them.
> airflow/gcp/operators/dataproc.py:409
> {code:java}
> elif self.custom_image:
> project_id = self.custom_image_project_id if
> (self.custom_image_project_id) else self.project_id
> custom_image_url = 'https://www.googleapis.com/compute/beta/projects/' \
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5495) Remove unneeded parens in dataproc.py
[
https://issues.apache.org/jira/browse/AIRFLOW-5495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930089#comment-16930089
]
ASF GitHub Bot commented on AIRFLOW-5495:
-
jghoman commented on pull request #6105: [AIRFLOW-5495] Remove unneeded parens
in dataproc.py
URL: https://github.com/apache/airflow/pull/6105
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Remove unneeded parens in dataproc.py
> -
>
> Key: AIRFLOW-5495
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5495
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.10.6
>Reporter: Jakob Homan
>Assignee: Adan Christian Rosales Ornelas
>Priority: Minor
> Labels: ccoss2019, newbie
>
> Note: This ticket's being created to facilitate a new contributor's workshop
> for Airflow. After the workshop has completed, I'll mark these all available
> for anyone that might like to take them on.
> The parens around {{self.custom_image_project_id}} don't need to be there; we
> should remove them.
> airflow/gcp/operators/dataproc.py:409
> {code:java}
> elif self.custom_image:
> project_id = self.custom_image_project_id if
> (self.custom_image_project_id) else self.project_id
> custom_image_url = 'https://www.googleapis.com/compute/beta/projects/' \
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [airflow] potiuk commented on a change in pull request #5837: [AIRFLOW-5223] Use KinD for Kubernetes in CI
potiuk commented on a change in pull request #5837: [AIRFLOW-5223] Use KinD for Kubernetes in CI URL: https://github.com/apache/airflow/pull/5837#discussion_r324481317 ## File path: Dockerfile ## @@ -51,11 +56,13 @@ RUN apt-get update \ && apt-get install -y --no-install-recommends \ curl \ gnupg2 \ + apt-transport-https \ Review comment: I see this is already moved to the 'airflow-apt-deps-ci' stage. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #5837: [AIRFLOW-5223] Use KinD for Kubernetes in CI
potiuk commented on a change in pull request #5837: [AIRFLOW-5223] Use KinD for Kubernetes in CI URL: https://github.com/apache/airflow/pull/5837#discussion_r324481094 ## File path: Dockerfile ## @@ -18,6 +18,8 @@ # Base image for the whole Docker file ARG APT_DEPS_IMAGE="airflow-apt-deps-ci-slim" ARG PYTHON_BASE_IMAGE="python:3.6-slim-stretch" +ARG KUBERNETES_VERSION="v1.15.0" Review comment: Hello @gerardo -> sorry for such big delays but I've been super busy with customer project and I had very little time. Yes. It should be moved to the new ci_build_install_deps.sh file. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #5837: [AIRFLOW-5223] Use KinD for Kubernetes in CI
potiuk commented on a change in pull request #5837: [AIRFLOW-5223] Use KinD for Kubernetes in CI URL: https://github.com/apache/airflow/pull/5837#discussion_r324481380 ## File path: Dockerfile ## @@ -85,6 +92,24 @@ RUN curl -sL https://deb.nodesource.com/setup_10.x | bash - \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* +# Kubernetes dependencies Review comment: This too should be moved to the new ci_build_install_deps.sh file. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj merged pull request #6078: [AIRFLOW-5459] Use a dynamic tmp location in Dataflow operator
mik-laj merged pull request #6078: [AIRFLOW-5459] Use a dynamic tmp location in Dataflow operator URL: https://github.com/apache/airflow/pull/6078 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (AIRFLOW-5459) use a dynamic tmp location in dataflow operator
[ https://issues.apache.org/jira/browse/AIRFLOW-5459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula closed AIRFLOW-5459. -- Resolution: Fixed > use a dynamic tmp location in dataflow operator > --- > > Key: AIRFLOW-5459 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5459 > Project: Apache Airflow > Issue Type: Bug > Components: gcp >Affects Versions: 1.10.5 >Reporter: Ben Marengo >Assignee: Ben Marengo >Priority: Major > Fix For: 1.10.6 > > > use a dynamic tmp location in dataflow operator -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5459) use a dynamic tmp location in dataflow operator
[ https://issues.apache.org/jira/browse/AIRFLOW-5459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930093#comment-16930093 ] ASF GitHub Bot commented on AIRFLOW-5459: - mik-laj commented on pull request #6078: [AIRFLOW-5459] Use a dynamic tmp location in Dataflow operator URL: https://github.com/apache/airflow/pull/6078 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > use a dynamic tmp location in dataflow operator > --- > > Key: AIRFLOW-5459 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5459 > Project: Apache Airflow > Issue Type: Bug > Components: gcp >Affects Versions: 1.10.5 >Reporter: Ben Marengo >Assignee: Ben Marengo >Priority: Major > Fix For: 1.10.6 > > > use a dynamic tmp location in dataflow operator -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5459) use a dynamic tmp location in dataflow operator
[ https://issues.apache.org/jira/browse/AIRFLOW-5459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930094#comment-16930094 ] ASF subversion and git services commented on AIRFLOW-5459: -- Commit ae9608dae3f137cd13ffdede4c1c1b3ca3767263 in airflow's branch refs/heads/master from marengaz [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=ae9608d ] [AIRFLOW-5459] Use a dynamic tmp location in Dataflow operator > use a dynamic tmp location in dataflow operator > --- > > Key: AIRFLOW-5459 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5459 > Project: Apache Airflow > Issue Type: Bug > Components: gcp >Affects Versions: 1.10.5 >Reporter: Ben Marengo >Assignee: Ben Marengo >Priority: Major > Fix For: 1.10.6 > > > use a dynamic tmp location in dataflow operator -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5445) Reduce the required resources for the Kubernetes's sidecar
[ https://issues.apache.org/jira/browse/AIRFLOW-5445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930095#comment-16930095 ] ASF GitHub Bot commented on AIRFLOW-5445: - mik-laj commented on pull request #6062: [AIRFLOW-5445] Reduce the required resources for the Kubernetes's sidecar URL: https://github.com/apache/airflow/pull/6062 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Reduce the required resources for the Kubernetes's sidecar > -- > > Key: AIRFLOW-5445 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5445 > Project: Apache Airflow > Issue Type: Improvement > Components: core, executor-kubernetes >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [airflow] mik-laj commented on issue #6062: [AIRFLOW-5445] Reduce the required resources for the Kubernetes's sidecar
mik-laj commented on issue #6062: [AIRFLOW-5445] Reduce the required resources
for the Kubernetes's sidecar
URL: https://github.com/apache/airflow/pull/6062#issuecomment-531600031
I checked carefully and `resource: {}` does not work properly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj merged pull request #6062: [AIRFLOW-5445] Reduce the required resources for the Kubernetes's sidecar
mik-laj merged pull request #6062: [AIRFLOW-5445] Reduce the required resources for the Kubernetes's sidecar URL: https://github.com/apache/airflow/pull/6062 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5489) bash_senor: Remove unneeded assignment of variable
[
https://issues.apache.org/jira/browse/AIRFLOW-5489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930097#comment-16930097
]
ASF GitHub Bot commented on AIRFLOW-5489:
-
getras1998 commented on pull request #6106: [AIRFLOW-5489] Remove unneeded
assignment of variable
URL: https://github.com/apache/airflow/pull/6106
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5489
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> bash_senor: Remove unneeded assignment of variable
> --
>
> Key: AIRFLOW-5489
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5489
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.10.6
>Reporter: Jakob Homan
>Priority: Minor
> Labels: ccoss2019, newbie
>
> Note: This ticket's being created to facilitate a new contributor's workshop
> for Airflow. After the workshop has completed, I'll mark these all available
> for anyone that might like to take them on.
> The `line` variable is assigned to `''` but then immediately reassigned in
> the loop. This first assignment should be deleted.
> airflow/contrib/sensors/bash_sensor.py:83
> {code:java}
> line = ''
> for line in iter(sp.stdout.readline, b''):
> line = line.decode(self.output_encoding).strip()
> self.log.info(line)
> sp.wait()
> self.log.info("Command exited with return code %s", sp.returncode) {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Closed] (AIRFLOW-5445) Reduce the required resources for the Kubernetes's sidecar
[ https://issues.apache.org/jira/browse/AIRFLOW-5445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula closed AIRFLOW-5445. -- Fix Version/s: 1.10.6 Resolution: Fixed > Reduce the required resources for the Kubernetes's sidecar > -- > > Key: AIRFLOW-5445 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5445 > Project: Apache Airflow > Issue Type: Improvement > Components: core, executor-kubernetes >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > Fix For: 1.10.6 > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (AIRFLOW-5445) Reduce the required resources for the Kubernetes's sidecar
[ https://issues.apache.org/jira/browse/AIRFLOW-5445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930096#comment-16930096 ] ASF subversion and git services commented on AIRFLOW-5445: -- Commit 7b5cf442e4a279403200118ebbafd41be0dd9389 in airflow's branch refs/heads/master from Kamil Breguła [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=7b5cf44 ] [AIRFLOW-5445] Reduce the required resources for the Kubernetes's sidecar (#6062) > Reduce the required resources for the Kubernetes's sidecar > -- > > Key: AIRFLOW-5445 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5445 > Project: Apache Airflow > Issue Type: Improvement > Components: core, executor-kubernetes >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [airflow] getras1998 opened a new pull request #6106: [AIRFLOW-5489] Remove unneeded assignment of variable
getras1998 opened a new pull request #6106: [AIRFLOW-5489] Remove unneeded
assignment of variable
URL: https://github.com/apache/airflow/pull/6106
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5489
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] getras1998 commented on issue #6106: [AIRFLOW-5489] Remove unneeded assignment of variable
getras1998 commented on issue #6106: [AIRFLOW-5489] Remove unneeded assignment of variable URL: https://github.com/apache/airflow/pull/6106#issuecomment-531600315 @jghoman This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6105: [AIRFLOW-5495] Remove unneeded parens in dataproc.py
codecov-io commented on issue #6105: [AIRFLOW-5495] Remove unneeded parens in dataproc.py URL: https://github.com/apache/airflow/pull/6105#issuecomment-531600949 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6105?src=pr=h1) Report > Merging [#6105](https://codecov.io/gh/apache/airflow/pull/6105?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/5f0b1f442367e6b5cb795f55d5e6ee1921ba8228?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6105?src=pr=tree) ```diff @@Coverage Diff@@ ## master #6105 +/- ## = - Coverage 80.11% 80.1% -0.01% = Files 607 607 Lines 34906 34906 = - Hits27964 27961 -3 - Misses 69426945 +3 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6105?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/gcp/operators/dataproc.py](https://codecov.io/gh/apache/airflow/pull/6105/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2RhdGFwcm9jLnB5) | `86.55% <100%> (ø)` | :arrow_up: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6105/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `74.28% <0%> (-0.31%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6105/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.8% <0%> (-0.19%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6105?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6105?src=pr=footer). Last update [5f0b1f4...9341a34](https://codecov.io/gh/apache/airflow/pull/6105?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5492) Missing docstring for hive .py
[
https://issues.apache.org/jira/browse/AIRFLOW-5492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930105#comment-16930105
]
ASF GitHub Bot commented on AIRFLOW-5492:
-
pedroortizm commented on pull request #6107: [AIRFLOW-5492] added missing
docstrings
URL: https://github.com/apache/airflow/pull/6107
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-5492\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5492
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-5492\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Missing docstring for hive .py
> --
>
> Key: AIRFLOW-5492
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5492
> Project: Apache Airflow
> Issue Type: Bug
> Components: utils
>Affects Versions: 1.10.6
>Reporter: Jakob Homan
>Priority: Minor
> Labels: ccoss2019, newbie
>
> Note: This ticket's being created to facilitate a new contributor's workshop
> for Airflow. After the workshop has completed, I'll mark these all available
> for anyone that might like to take them on.
> We need to add doc_strings for both {{schema}} and {{metastore_conn_id}}
> airflow/macros/hive.py:83
> {code:java}
> def closest_ds_partition(
> table, ds, before=True, schema="default",
> metastore_conn_id='metastore_default'):
> """
> This function finds the date in a list closest to the target date.
> An optional parameter can be given to get the closest before or after.
> :param table: A hive table name
> :type table: str
> :param ds: A datestamp ``%Y-%m-%d`` e.g. ``-mm-dd``
> :type ds: list[datetime.date]
> :param before: closest before (True), after (False) or either side of ds
> :type before: bool or None
> :returns: The closest date
> :rtype: str or None {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [airflow] mik-laj merged pull request #6086: [AIRFLOW-XXX] Display GCP integration in table
mik-laj merged pull request #6086: [AIRFLOW-XXX] Display GCP integration in table URL: https://github.com/apache/airflow/pull/6086 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] pedroortizm opened a new pull request #6107: [AIRFLOW-5492] added missing docstrings
pedroortizm opened a new pull request #6107: [AIRFLOW-5492] added missing
docstrings
URL: https://github.com/apache/airflow/pull/6107
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-5492\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5492
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-5492\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] pedroortizm commented on issue #6107: [AIRFLOW-5492] added missing docstrings
pedroortizm commented on issue #6107: [AIRFLOW-5492] added missing docstrings URL: https://github.com/apache/airflow/pull/6107#issuecomment-531601283 @jghoman This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Mordeck opened a new pull request #6108: [AIRFLOW-5491] mark_tasks pydoc is incorrect
Mordeck opened a new pull request #6108: [AIRFLOW-5491] mark_tasks pydoc is
incorrect
URL: https://github.com/apache/airflow/pull/6108
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj merged pull request #6108: [AIRFLOW-5491] mark_tasks pydoc is incorrect
mik-laj merged pull request #6108: [AIRFLOW-5491] mark_tasks pydoc is incorrect URL: https://github.com/apache/airflow/pull/6108 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5491) mark_tasks pydoc is incorrect
[
https://issues.apache.org/jira/browse/AIRFLOW-5491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930107#comment-16930107
]
ASF GitHub Bot commented on AIRFLOW-5491:
-
mik-laj commented on pull request #6108: [AIRFLOW-5491] mark_tasks pydoc is
incorrect
URL: https://github.com/apache/airflow/pull/6108
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> mark_tasks pydoc is incorrect
> -
>
> Key: AIRFLOW-5491
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5491
> Project: Apache Airflow
> Issue Type: Bug
> Components: security
>Affects Versions: 1.10.6
>Reporter: Jakob Homan
>Priority: Minor
> Labels: ccoss2019, newbie
>
> Note: This ticket's being created to facilitate a new contributor's workshop
> for Airflow. After the workshop has completed, I'll mark these all available
> for anyone that might like to take them on.
> The pydoc for set_state is incorrect; it thinks the first param named {{task
> instead of }}{{tasks}} (which is used in the code, and the doc itself thinks
> this is a single task instead of an iterable.
> airflow/api/common/experimental/mark_tasks.py:62
> {code:java}
> def set_state(
> tasks: Iterable[BaseOperator],
> execution_date: datetime.datetime,
> upstream: bool = False,
> downstream: bool = False,
> future: bool = False,
> past: bool = False,
> state: str = State.SUCCESS,
> commit: bool = False,
> session=None): # pylint: disable=too-many-arguments,too-many-locals
> """
> Set the state of a task instance and if needed its relatives. Can set
> state
> for future tasks (calculated from execution_date) and retroactively
> for past tasks. Will verify integrity of past dag runs in order to create
> tasks that did not exist. It will not create dag runs that are missing
> on the schedule (but it will as for subdag dag runs if needed).
> :param task: the task from which to work. task.task.dag needs to be set
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
