jack created AIRFLOW-2679:
-----------------------------
Summary: GoogleCloudStorageToBigQueryOperator to support UPSERT
Key: AIRFLOW-2679
URL: https://issues.apache.org/jira/browse/AIRFLOW-2679
Project: Apache Airflow
Issue Type: Improvement
Reporter: jack
Fix For: 1.10.0, 1.10
Currently the
{color:#222222}GoogleCloudStorageToBigQueryOp{color}{color:#222222}erator
support incremental load using
*{color:#404040}max_id_key{color}*{color:#404040} {color}.{color}
{color:#222222}However many systems actually needs "UPSERT" in terms of - if
row exists update it, if not insert/copy it.{color}
{color:#222222}Currently the operator assumes that we only need to insert new
data, it can't handle update of data. Most of the time data is not static it
changes with time. Yesterday order status was NEW today it's Processing,
tomorrow it's SENT in a month it will be REFUNDED etc... {color}
{color:#222222} {color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
