[
https://issues.apache.org/jira/browse/AIRFLOW-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor resolved AIRFLOW-2862.
----------------------------------------
Resolution: Duplicate
> S3ToRedshiftTransfer Copy Command Flexibility
> ---------------------------------------------
>
> Key: AIRFLOW-2862
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2862
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators
> Reporter: Micheal Ascah
> Assignee: Micheal Ascah
> Priority: Minor
>
> Currently, the S3ToRedshiftTransfer class requires that the target table to
> be loaded is suffixed to the end of the S3 key provided.
> It doesn't seem justifiable that the operator should require the file be
> named by any convention. The S3 bucket + S3 key should be all that is needed.
> This makes it possible to load any S3 Key into a Redshift table, rather than
> only files that have the table name at the end of the S3 key.
> The S3 key parameter should also be template-able so that files created in S3
> using timestamps from macros in other tasks in the current DAG run can be
> used to identify files when loading from S3 to Redshift.
> The command template should change from
> {code:java}
> COPY {schema}.{table}
> FROM 's3://{s3_bucket}/{s3_key}/{table}'
> with credentials
> 'aws_access_key_id={access_key};aws_secret_access_key={secret_key}'
> {copy_options};{code}
> To
>
> {code:java}
> COPY {schema}.{table}
> FROM 's3://{s3_bucket}/{s3_key}'
> with credentials
> 'aws_access_key_id={access_key};aws_secret_access_key={secret_key}'
> {copy_options};
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
