[
https://issues.apache.org/jira/browse/AIRFLOW-6891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik closed AIRFLOW-6891.
-------------------------------
Resolution: Cannot Reproduce
I couldn't reproduce the issue from StackOverflow user
> GCS to BQ operator fails when JSON is the source format
> -------------------------------------------------------
>
> Key: AIRFLOW-6891
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6891
> Project: Apache Airflow
> Issue Type: Bug
> Components: gcp
> Affects Versions: 1.10.9
> Reporter: Kaxil Naik
> Assignee: Kaxil Naik
> Priority: Major
>
> From
> https://stackoverflow.com/questions/60358764/airflow-gcs-to-bq-operator-fails-when-json-is-the-source-format
> I have a GoogleCloudStorageToBigQueryOperator operator running on airflow in
> a dag. It works perfect when working CSV files... I am now trying to ingest a
> JSON file, and I'm receiving errors: such like:
> *skipLeadingRows* is not a valid src_fmt_configs for type
> *NEWLINE_DELIMITED_JSON*
> The weird thing is that I'm not calling *skipLeadingRows* in my calling. as
> below:
>
> {noformat}
> load_Users_to_GBQ = GoogleCloudStorageToBigQueryOperator(
> task_id='Table1_GCS_to_GBQ',
> bucket='bucket1',
> source_objects=['table*.json'],
> source_format='NEWLINE_DELIMITED_JSON',
> destination_project_dataset_table='DB.table1',
> autodetect=False,
> schema_fields=[
> {'name': 'fieldid', 'type': 'integer', 'mode': 'NULLABLE'},
> {'name': 'filed2', 'type': 'integer', 'mode': 'NULLABLE'},
> {'name': 'field3', 'type': 'string', 'mode': 'NULLABLE'},
> {'name': 'field4', 'type': 'string', 'mode': 'NULLABLE'},
> {'name': 'field5', 'type': 'string', 'mode': 'NULLABLE'}
> ],
> write_disposition='WRITE_TRUNCATE',
> google_cloud_storage_conn_id='Conn1',
> bigquery_conn_id='Conn1',
> dag=dag)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
