tianjianjiang opened a new pull request #15026: URL: https://github.com/apache/airflow/pull/15026
Compatibility issues: For `datetime`-related values: - MySQL's range starts from year 1000, except when it is an error value that uses year 0000; - BigQuery's range starts from year 0001. JSON/CSV dumps for BigQuery import: - https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-json#limitations > When you load JSON or CSV data, values in TIMESTAMP columns must use a dash (-) separator for the date portion of the timestamp, and the date must be in the following format: YYYY-MM-DD (year-month-day). The hh:mm:ss (hour-minute-second) portion of the timestamp must use a colon (:) separator. - https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-json#details_of_loading_json_data > If you provide a schema, BigQuery also accepts Unix Epoch time for timestamp values. However, schema autodetection will not detect this case, and will treat the value as a numeric or string type instead. When no `schema` provided, the current version of the operator assumes that MySQL's DATETIME and DATE values becoming BigQuery's TIMESTAMP, which doesn't accept Unix Epoch integer. Should there be a `schema`, the current version of the operator sends integers and floats to BigQuery's DATETIME/DATE/TIMESTAMP and TIME, respectively. Issues are: - For DATETIME/DATE, BigQuery will throw errors; - For TIMESTAMP, although BigQuery accepts Unix Epoch integer in this case, `calendar.timegm()` will still cause troubles; ```python >>> calendar.timegm((1000,1,1,0,0,0)) -30610224000 ``` - For TIME, BigQuery may silently convert to incorrect values. --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information. In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
