[the previous email was send too early by mistake]
update:
I tried the following options:
1. return dict from DnFn and error was fired:
newRowDictlist = newRowDict #[newRowDict]
return(newRowDictlist)
and the following warning:
*Returning a dict from a ParDo or FlatMap is discouraged. Please use list("...*
2. return list with dict in it
newRowDictlist = [newRowDict]
return(newRowDictlist)
No error was generated. I see the table but the data hasn't been populated
yet. BQ normal delay as far as I know
Since I can not see the BQ full result....could you please let me know if I
am writing the data at the right format to BQ (I had no issues writing it
to other type of outputs)
Thanks for any help,
Eila
On Mon, Aug 13, 2018 at 1:55 PM, OrielResearch Eila Arich-Landkof <
[email protected]> wrote:
> update:
>
> I tried the following options:
>
> 1. return dict from DnFn and error was fired:
> newRowDictlist = newRowDict #[newRowDict]
> return(newRowDictlist)
>
> 2. return list with dict in it
> newRowDictlist = [newRowDict]
> return(newRowDictlist)
>
>
>
> On Mon, Aug 13, 2018 at 12:51 PM, OrielResearch Eila Arich-Landkof <
> [email protected]> wrote:
>
>> Hello,
>>
>> I am generating a data to be written in new BQ table with a specific
>> schema. The data is generated at DoFn function.
>>
>> My question is: what is the recommended format of data that I should
>> return from DnFn (getValuesStrFn bellow) ? is it dictionary? list? other?
>> I tried list and str and it fired an error.
>>
>>
>> The pipeline is:
>> p = beam.Pipeline(options=options)
>> (p | 'Read From Data Frame' >> beam.Create(cellLinesTable.val
>> ues.tolist())
>> | 'call Get Value Str' >> beam.ParDo(getValuesStrFn(colList))
>> | 'write to BQ' >> beam.io.Write(beam.io.BigQuery
>> Sink(dataset='dataset_cell_lines',table='cell_lines_table',
>> schema=schema_bq)))
>> Thanks,
>> --
>> Eila
>> www.orielresearch.org
>> https://www.meetu <https://www.meetup.com/Deep-Learning-In-Production/>
>> p.co <https://www.meetup.com/Deep-Learning-In-Production/>m/Deep-Le
>> arning-In-Production/
>> <https://www.meetup.com/Deep-Learning-In-Production/>
>>
>>
>>
>
>
> --
> Eila
> www.orielresearch.org
> https://www.meetu <https://www.meetup.com/Deep-Learning-In-Production/>
> p.co <https://www.meetup.com/Deep-Learning-In-Production/>m/Deep-
> Learning-In-Production/
> <https://www.meetup.com/Deep-Learning-In-Production/>
>
>
>
--
Eila
www.orielresearch.org
https://www.meetu <https://www.meetup.com/Deep-Learning-In-Production/>p.co
<https://www.meetup.com/Deep-Learning-In-Production/>
m/Deep-Learning-In-Production/
<https://www.meetup.com/Deep-Learning-In-Production/>
