JavierLopezT opened a new pull request #9246:
URL: https://github.com/apache/airflow/pull/9246
This PR includes several changes:
- New functionality for deleting the S3 file after the COPY
- New functionality for making a check of duplicates based on the primary
keys of the table
- New functionality of executing queries prior to the copy (for instance a
create statement)
- Color changed to a more vivid one
- Addition of s3_key as template_field
- Addition of .sql as template ext (for previous_queries)
- Modified a part in the copy statement, that will makes prior workflows
incompatible. Not sure if I have to propose a discussion about this point
somewhere else (If so, I will delete this part from this PR) or just create a
note in UPDATING.md. The prior code was:
```
COPY {self.schema}.{self.table}
FROM 's3://{self.s3_bucket}/{self.s3_key}/{self.table}'
with credentials
'aws_access_key_id={credentials.access_key};aws_secret_access_key={credentials.secret_key}'
{copy_options};
```
And the new is:
```
COPY {self.schema}.{self.table}
FROM 's3://{self.s3_bucket}/{self.s3_key}'
with credentials
'aws_access_key_id={credentials.access_key};aws_secret_access_key={credentials.secret_key}'
{copy_options};
```
So we don't have to have the table as the file name. I have read here and
somewhere else complains about the existence of the table at the end of the
from.
I have not made any tests yet because I am not sure if you will find all of
these features useful.
---
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [ ] Unit tests coverage for changes (not needed for documentation changes)
- [x] Target Github ISSUE in description if exists
- [x] Commits follow "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
