pierrejeambrun commented on PR #25691: URL: https://github.com/apache/airflow/pull/25691#issuecomment-1220097978
I took a look. Can't find what changed. I found working extract from August 2022, with `PostgresToGCSOperator` to parquet format with `date` and `datetime` in the schema. :thinking: The schema generated by this operator use type that are safe for BigQuery (`_write_local_schema_file` will for instance use `field_to_bigquery`, using the underlying `type_map` mapping db types to BigQuery types). This makes sure we can import into `BigQuery` an exported Schema to `GCS`. Parquet has on top of that an additional mapping, for mapping BigQuery types to pyarrow types. (See `_convert_parquet_schema`). I would expect parquet export to be successful when columns are dates, but also be able to import this to bigquery with a correct schema definition. (This is how it works for csv and json export) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
