sanxore commented on PR #25691: URL: https://github.com/apache/airflow/pull/25691#issuecomment-1220286818
> > If I remember correctly we had to modify things to be able to export them to `BigQuery` with the correct field type. > > > > * You are updating the `date` case, do we also want not to isoformat `datetime` in this case ? [here](https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/transfers/postgres_to_gcs.py#L143) > > * Is is possible to confirm that these fields are getting exported with the correct type to bigquery with this modification ? (DATETIME, DATE and TIME) > > * we have the issue with datetime too. ( I will add the fix to the PR ) > * This Operator have no interaction with BigQuery. The goal of this Operator is to write data to GCS file system with the right type if we write data to a binary format ( like parquet) or with the right format if we write data to a string format ( like csv ) > > Converting time, detetime and date to a string format had broken the Operator for parquet as export format Yes i confirm this field will have the right type in BigQuery. If you see my terminal screenshot I parsed the schema of parquet file I exported. The physical type is INT32 an logical type is date is exactly what's date for BigQuery in we use a `bq load` for managed and external table on parquet files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
