[jira] [Commented] (AIRFLOW-2524) Airflow integration with AWS Sagemaker
[
https://issues.apache.org/jira/browse/AIRFLOW-2524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16585028#comment-16585028
]
ASF GitHub Bot commented on AIRFLOW-2524:
-
troychen728 opened a new pull request #3767: [AIRFLOW-2524]Add SageMaker Batch
Inference
URL: https://github.com/apache/incubator-airflow/pull/3767
Make sure you have checked _all_ steps below.
### JIRA
- [X] My PR addresses the following [Airflow
JIRA](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title.
- https://issues.apache.org/jira/browse/AIRFLOW-2524
### Description
- [X] Here are some details about my PR, including screenshots of any UI
changes:
- This PR allows user to start a Amazon SageMaker Transform job (Batch
Inference) using the SageMakerCreateTransformJobOperator
- User can also check the progress(state) of the transform job through
the SageMakerTranformSensor
### Tests
- [X] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
- tests/contrib/hooks/test_sagemaker_hook.py
-
tests/contrib/operators/test_sagemaker_create_transform_job_operator.py
- tests/contrib/sensors/test_sagemaker_transform_sensor.py
### Commits
- [X] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
### Documentation
- [X] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [X] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Airflow integration with AWS Sagemaker
> --
>
> Key: AIRFLOW-2524
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2524
> Project: Apache Airflow
> Issue Type: Improvement
> Components: aws, contrib
>Reporter: Rajeev Srinivasan
>Assignee: Yang Yu
>Priority: Major
> Labels: AWS
> Fix For: 2.0.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Would it be possible to orchestrate an end to end AWS Sagemaker job using
> Airflow.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] troychen728 opened a new pull request #3767: [AIRFLOW-2524]Add SageMaker Batch Inference
troychen728 opened a new pull request #3767: [AIRFLOW-2524]Add SageMaker Batch
Inference
URL: https://github.com/apache/incubator-airflow/pull/3767
Make sure you have checked _all_ steps below.
### JIRA
- [X] My PR addresses the following [Airflow
JIRA](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title.
- https://issues.apache.org/jira/browse/AIRFLOW-2524
### Description
- [X] Here are some details about my PR, including screenshots of any UI
changes:
- This PR allows user to start a Amazon SageMaker Transform job (Batch
Inference) using the SageMakerCreateTransformJobOperator
- User can also check the progress(state) of the transform job through
the SageMakerTranformSensor
### Tests
- [X] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
- tests/contrib/hooks/test_sagemaker_hook.py
-
tests/contrib/operators/test_sagemaker_create_transform_job_operator.py
- tests/contrib/sensors/test_sagemaker_transform_sensor.py
### Commits
- [X] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
### Documentation
- [X] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [X] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] feng-tao commented on issue #3763: [AIRFLOW-2915] Add example DAG for GoogleCloudStorageToBigQueryOperator
feng-tao commented on issue #3763: [AIRFLOW-2915] Add example DAG for GoogleCloudStorageToBigQueryOperator URL: https://github.com/apache/incubator-airflow/pull/3763#issuecomment-414099828 If I don't have gcp dependency, what happens when I set load_example_dag to true? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dalejung commented on issue #3723: [AIRFLOW-2876] Update Tenacity to 4.12
dalejung commented on issue #3723: [AIRFLOW-2876] Update Tenacity to 4.12 URL: https://github.com/apache/incubator-airflow/pull/3723#issuecomment-414085029 @r39132 is the futures backport installed in your 3.6 install? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #3766: [AIRFLOW-2917] Set AIRFLOW__CORE__SQL_ALCHEMY_CONN only when needed
codecov-io commented on issue #3766: [AIRFLOW-2917] Set AIRFLOW__CORE__SQL_ALCHEMY_CONN only when needed URL: https://github.com/apache/incubator-airflow/pull/3766#issuecomment-414063517 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3766?src=pr=h1) Report > Merging [#3766](https://codecov.io/gh/apache/incubator-airflow/pull/3766?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/738d27edfeddb0dc8712cf2e2db72dedb4354821?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3766?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#3766 +/- ## === Coverage 77.67% 77.67% === Files 204 204 Lines 1584915849 === Hits1231012310 Misses 3539 3539 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3766?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3766?src=pr=footer). Last update [738d27e...4446886](https://codecov.io/gh/apache/incubator-airflow/pull/3766?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2917) Set AIRFLOW__CORE__SQL_ALCHEMY_CONN only when needed for k8s executor
[
https://issues.apache.org/jira/browse/AIRFLOW-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584783#comment-16584783
]
ASF GitHub Bot commented on AIRFLOW-2917:
-
ckljohn opened a new pull request #3766: [AIRFLOW-2917] Set
AIRFLOW__CORE__SQL_ALCHEMY_CONN only when needed
URL: https://github.com/apache/incubator-airflow/pull/3766
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-2917
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
In Kubernetes executor, `AIRFLOW_CORE_SQL_ALCHEMY_CONN` is set as an
environment variable even when it is specified in configmap or secrets.
Only when `airflow_configmap` is not provided and
`AIRFLOW__CORE__SQL_ALCHEMY_CONN` not in secrets, it is set as an env var.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Set AIRFLOW__CORE__SQL_ALCHEMY_CONN only when needed for k8s executor
> -
>
> Key: AIRFLOW-2917
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2917
> Project: Apache Airflow
> Issue Type: Improvement
> Components: executor
>Affects Versions: 1.10.0
>Reporter: John Cheng
>Priority: Minor
>
> In Kubernetes executor, `AIRFLOW__CORE__SQL_ALCHEMY_CONN` is set as an
> environment variable even when it is specified in configmap or secrets.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] ckljohn opened a new pull request #3766: [AIRFLOW-2917] Set AIRFLOW__CORE__SQL_ALCHEMY_CONN only when needed
ckljohn opened a new pull request #3766: [AIRFLOW-2917] Set
AIRFLOW__CORE__SQL_ALCHEMY_CONN only when needed
URL: https://github.com/apache/incubator-airflow/pull/3766
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-2917
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
In Kubernetes executor, `AIRFLOW_CORE_SQL_ALCHEMY_CONN` is set as an
environment variable even when it is specified in configmap or secrets.
Only when `airflow_configmap` is not provided and
`AIRFLOW__CORE__SQL_ALCHEMY_CONN` not in secrets, it is set as an env var.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (AIRFLOW-2917) Set AIRFLOW__CORE__SQL_ALCHEMY_CONN only when needed for k8s executor
[ https://issues.apache.org/jira/browse/AIRFLOW-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Cheng updated AIRFLOW-2917: Description: In Kubernetes executor, `AIRFLOW__CORE__SQL_ALCHEMY_CONN` is set as an environment variable even when it is specified in configmap or secrets. (was: In Kubernetes executor, AIRFLOW__CORE__SQL_ALCHEMY_CONN is set as an environment variable even when it is specified in configmap or secrets.) > Set AIRFLOW__CORE__SQL_ALCHEMY_CONN only when needed for k8s executor > - > > Key: AIRFLOW-2917 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2917 > Project: Apache Airflow > Issue Type: Improvement > Components: executor >Affects Versions: 1.10.0 >Reporter: John Cheng >Priority: Minor > > In Kubernetes executor, `AIRFLOW__CORE__SQL_ALCHEMY_CONN` is set as an > environment variable even when it is specified in configmap or secrets. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2917) Set AIRFLOW__CORE__SQL_ALCHEMY_CONN only when needed for k8s executor
John Cheng created AIRFLOW-2917: --- Summary: Set AIRFLOW__CORE__SQL_ALCHEMY_CONN only when needed for k8s executor Key: AIRFLOW-2917 URL: https://issues.apache.org/jira/browse/AIRFLOW-2917 Project: Apache Airflow Issue Type: Improvement Components: executor Affects Versions: 1.10.0 Reporter: John Cheng In Kubernetes executor, AIRFLOW__CORE__SQL_ALCHEMY_CONN is set as an environment variable even when it is specified in configmap or secrets. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] codecov-io commented on issue #3765: [AIRFLOW-2245] Add remote_host of SSH/SFTP operator as templated field
codecov-io commented on issue #3765: [AIRFLOW-2245] Add remote_host of SSH/SFTP operator as templated field URL: https://github.com/apache/incubator-airflow/pull/3765#issuecomment-414056215 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3765?src=pr=h1) Report > Merging [#3765](https://codecov.io/gh/apache/incubator-airflow/pull/3765?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/738d27edfeddb0dc8712cf2e2db72dedb4354821?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3765?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#3765 +/- ## === Coverage 77.67% 77.67% === Files 204 204 Lines 1584915849 === Hits1231012310 Misses 3539 3539 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3765?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3765?src=pr=footer). Last update [738d27e...1d4f374](https://codecov.io/gh/apache/incubator-airflow/pull/3765?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #3765: [AIRFLOW-2245] Add remote_host of SSH/SFTP operator as templated field
codecov-io edited a comment on issue #3765: [AIRFLOW-2245] Add remote_host of SSH/SFTP operator as templated field URL: https://github.com/apache/incubator-airflow/pull/3765#issuecomment-414056215 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3765?src=pr=h1) Report > Merging [#3765](https://codecov.io/gh/apache/incubator-airflow/pull/3765?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/738d27edfeddb0dc8712cf2e2db72dedb4354821?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3765?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#3765 +/- ## === Coverage 77.67% 77.67% === Files 204 204 Lines 1584915849 === Hits1231012310 Misses 3539 3539 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3765?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3765?src=pr=footer). Last update [738d27e...1d4f374](https://codecov.io/gh/apache/incubator-airflow/pull/3765?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2245) Add remote_host of SSH/SFTP operator as templated field
[
https://issues.apache.org/jira/browse/AIRFLOW-2245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584754#comment-16584754
]
ASF GitHub Bot commented on AIRFLOW-2245:
-
ckljohn opened a new pull request #3765: [AIRFLOW-2245] Add remote_host of
SSH/SFTP operator as templated field
URL: https://github.com/apache/incubator-airflow/pull/3765
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-2245
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
It added `remote_host` to the templated fields .`remote_host` can be
obtained at the beginning of the DAG and pass to SSH/SFTP operator with xcom.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
My PR does not add tests because this is a small change, and there are no
tests currently in-place to test the templated fields.
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add remote_host of SSH/SFTP operator as templated field
> ---
>
> Key: AIRFLOW-2245
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2245
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators
>Reporter: John Cheng
>Priority: Trivial
>
> The remote host can be obtained at the beginning of the DAG and pass to
> SSH/SFTP operator with xcom.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2245) Add remote_host of SSH/SFTP operator as templated field
[ https://issues.apache.org/jira/browse/AIRFLOW-2245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Cheng updated AIRFLOW-2245: Issue Type: Improvement (was: Wish) > Add remote_host of SSH/SFTP operator as templated field > --- > > Key: AIRFLOW-2245 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2245 > Project: Apache Airflow > Issue Type: Improvement >Reporter: John Cheng >Priority: Trivial > > The remote host can be obtained at the beginning of the DAG and pass to > SSH/SFTP operator with xcom. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2245) Add remote_host of SSH/SFTP operator as templated field
[ https://issues.apache.org/jira/browse/AIRFLOW-2245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Cheng updated AIRFLOW-2245: Priority: Trivial (was: Minor) > Add remote_host of SSH/SFTP operator as templated field > --- > > Key: AIRFLOW-2245 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2245 > Project: Apache Airflow > Issue Type: Wish >Reporter: John Cheng >Priority: Trivial > > The remote host can be obtained at the beginning of the DAG and pass to > SSH/SFTP operator with xcom. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2245) Add remote_host of SSH/SFTP operator as templated field
[ https://issues.apache.org/jira/browse/AIRFLOW-2245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Cheng updated AIRFLOW-2245: Description: The remote host can be obtained at the beginning of the DAG and pass to SSH/SFTP operator with xcom. (was: Add Jinja support for remote_host of SSH operator and hook) Summary: Add remote_host of SSH/SFTP operator as templated field (was: Add Jinja support for remote_host of SSH operator and hook) > Add remote_host of SSH/SFTP operator as templated field > --- > > Key: AIRFLOW-2245 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2245 > Project: Apache Airflow > Issue Type: Wish >Reporter: John Cheng >Priority: Minor > > The remote host can be obtained at the beginning of the DAG and pass to > SSH/SFTP operator with xcom. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] codecov-io commented on issue #3764: [AIRFLOW-2916] Arg `verify` for AwsHook() & S3 sensors/operators
codecov-io commented on issue #3764: [AIRFLOW-2916] Arg `verify` for AwsHook() & S3 sensors/operators URL: https://github.com/apache/incubator-airflow/pull/3764#issuecomment-414046987 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3764?src=pr=h1) Report > Merging [#3764](https://codecov.io/gh/apache/incubator-airflow/pull/3764?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/738d27edfeddb0dc8712cf2e2db72dedb4354821?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `71.42%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3764?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3764 +/- ## == - Coverage 77.67% 77.66% -0.01% == Files 204 204 Lines 1584915856 +7 == + Hits1231012315 +5 - Misses 3539 3541 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3764?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/sensors/s3\_prefix\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3764/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3MzX3ByZWZpeF9zZW5zb3IucHk=) | `38.88% <0%> (-2.29%)` | :arrow_down: | | [airflow/sensors/s3\_key\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3764/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3MzX2tleV9zZW5zb3IucHk=) | `30.3% <0%> (-0.95%)` | :arrow_down: | | [airflow/operators/redshift\_to\_s3\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/3764/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcmVkc2hpZnRfdG9fczNfb3BlcmF0b3IucHk=) | `95.55% <100%> (+0.1%)` | :arrow_up: | | [airflow/operators/s3\_to\_hive\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/3764/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvczNfdG9faGl2ZV9vcGVyYXRvci5weQ==) | `94.01% <100%> (+0.05%)` | :arrow_up: | | [airflow/operators/s3\_file\_transform\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/3764/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvczNfZmlsZV90cmFuc2Zvcm1fb3BlcmF0b3IucHk=) | `93.87% <100%> (+0.26%)` | :arrow_up: | | [airflow/operators/s3\_to\_redshift\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/3764/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvczNfdG9fcmVkc2hpZnRfb3BlcmF0b3IucHk=) | `100% <100%> (ø)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3764?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3764?src=pr=footer). Last update [738d27e...142e1ff](https://codecov.io/gh/apache/incubator-airflow/pull/3764?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2916) Add argument `verify` for AwsHook() and S3 related sensors/operators
[
https://issues.apache.org/jira/browse/AIRFLOW-2916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584722#comment-16584722
]
ASF GitHub Bot commented on AIRFLOW-2916:
-
XD-DENG opened a new pull request #3764: [AIRFLOW-2916] Arg `verify` for
AwsHook() & S3 sensors/operators
URL: https://github.com/apache/incubator-airflow/pull/3764
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-2916
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
### Background
The `AwsHook()` and `S3`-related operators/sensors are depending on package
`boto3`.
In `boto3`, when we initiate a `client` or a `resource`, argument `verify`
is provided (https://boto3.readthedocs.io/en/latest/reference/core/session.html
).
### Why do we need this
It is useful when
- users want to use a different CA cert bundle than the one used by
`botocore`.
- users want to have **'--no-verify-ssl'**. This is especially useful when
we're using on-premises S3 or other implementations of object storage, like
IBM's Cloud Object Storage.
### Backward Compatibility
The default value here is always `None`, which is also the default value in
`boto3`, so that backward compatibility is ensured too.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add argument `verify` for AwsHook() and S3 related sensors/operators
>
>
> Key: AIRFLOW-2916
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2916
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks, operators
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Minor
>
> The AwsHook() and S3-related operators/sensors are depending on package boto3.
> In boto3, when we initiate a client or a resource, argument `verify` is
> provided (https://boto3.readthedocs.io/en/latest/reference/core/session.html
> ).
> It is useful when
> # users want to use a different CA cert bundle than the one used by botocore.
> # users want to have '--no-verify-ssl'. This is especially useful when we're
> using on-premises S3 or other implementations of object storage, like IBM's
> Cloud Object Storage.
> However, this feature is not provided in Airflow for S3 yet.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] XD-DENG opened a new pull request #3764: [AIRFLOW-2916] Arg `verify` for AwsHook() & S3 sensors/operators
XD-DENG opened a new pull request #3764: [AIRFLOW-2916] Arg `verify` for
AwsHook() & S3 sensors/operators
URL: https://github.com/apache/incubator-airflow/pull/3764
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-2916
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
### Background
The `AwsHook()` and `S3`-related operators/sensors are depending on package
`boto3`.
In `boto3`, when we initiate a `client` or a `resource`, argument `verify`
is provided (https://boto3.readthedocs.io/en/latest/reference/core/session.html
).
### Why do we need this
It is useful when
- users want to use a different CA cert bundle than the one used by
`botocore`.
- users want to have **'--no-verify-ssl'**. This is especially useful when
we're using on-premises S3 or other implementations of object storage, like
IBM's Cloud Object Storage.
### Backward Compatibility
The default value here is always `None`, which is also the default value in
`boto3`, so that backward compatibility is ensured too.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-2916) Add argument `verify` for AwsHook() and S3 related sensors/operators
Xiaodong DENG created AIRFLOW-2916: -- Summary: Add argument `verify` for AwsHook() and S3 related sensors/operators Key: AIRFLOW-2916 URL: https://issues.apache.org/jira/browse/AIRFLOW-2916 Project: Apache Airflow Issue Type: Improvement Components: hooks, operators Reporter: Xiaodong DENG Assignee: Xiaodong DENG The AwsHook() and S3-related operators/sensors are depending on package boto3. In boto3, when we initiate a client or a resource, argument `verify` is provided (https://boto3.readthedocs.io/en/latest/reference/core/session.html ). It is useful when # users want to use a different CA cert bundle than the one used by botocore. # users want to have '--no-verify-ssl'. This is especially useful when we're using on-premises S3 or other implementations of object storage, like IBM's Cloud Object Storage. However, this feature is not provided in Airflow for S3 yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
