Hello, I am generating a data to be written in new BQ table with a specific schema. The data is generated at DoFn function.
My question is: what is the recommended format of data that I should return from DnFn (getValuesStrFn bellow) ? is it dictionary? list? other? I tried list and str and it fired an error. The pipeline is: p = beam.Pipeline(options=options) (p | 'Read From Data Frame' >> beam.Create(cellLinesTable.values.tolist()) | 'call Get Value Str' >> beam.ParDo(getValuesStrFn(colList)) | 'write to BQ' >> beam.io.Write(beam.io.BigQuerySink(dataset='dataset_cell_lines',table='cell_lines_table', schema=schema_bq))) Thanks, -- Eila www.orielresearch.org https://www.meetu <https://www.meetup.com/Deep-Learning-In-Production/>p.co <https://www.meetup.com/Deep-Learning-In-Production/> m/Deep-Learning-In-Production/ <https://www.meetup.com/Deep-Learning-In-Production/>
