update:
I tried the following options:
1. return dict from DnFn and error was fired:
newRowDictlist = newRowDict #[newRowDict]
return(newRowDictlist)
2. return list with dict in it
newRowDictlist = [newRowDict]
return(newRowDictlist)
On Mon, Aug 13, 2018 at 12:51 PM, OrielResearch Eila Arich-Landkof <
[email protected]> wrote:
> Hello,
>
> I am generating a data to be written in new BQ table with a specific
> schema. The data is generated at DoFn function.
>
> My question is: what is the recommended format of data that I should
> return from DnFn (getValuesStrFn bellow) ? is it dictionary? list? other?
> I tried list and str and it fired an error.
>
>
> The pipeline is:
> p = beam.Pipeline(options=options)
> (p | 'Read From Data Frame' >> beam.Create(cellLinesTable.values.tolist())
> | 'call Get Value Str' >> beam.ParDo(getValuesStrFn(colList))
> | 'write to BQ' >> beam.io.Write(beam.io.BigQuerySink(dataset='dataset_
> cell_lines',table='cell_lines_table', schema=schema_bq)))
> Thanks,
> --
> Eila
> www.orielresearch.org
> https://www.meetu <https://www.meetup.com/Deep-Learning-In-Production/>
> p.co <https://www.meetup.com/Deep-Learning-In-Production/>m/Deep-
> Learning-In-Production/
> <https://www.meetup.com/Deep-Learning-In-Production/>
>
>
>
--
Eila
www.orielresearch.org
https://www.meetu <https://www.meetup.com/Deep-Learning-In-Production/>p.co
<https://www.meetup.com/Deep-Learning-In-Production/>
m/Deep-Learning-In-Production/
<https://www.meetup.com/Deep-Learning-In-Production/>
