[
https://issues.apache.org/jira/browse/BEAM-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370395#comment-16370395
]
Thomas Groh commented on BEAM-2767:
-----------------------------------
The underlying cause of this is due to the encoding round trip performed by the
DirectRunner between steps, which is not usually performed in Dataflow.
Accessing the repeated record (or any record) as a {{Map<String, Object>}}
field will execute successfully on both of these runners, as a {{TableRow}}
implements the {{Map<String, Object>}} interface.
Records are read as type "TableRow", but when they are encoded they are encoded
as a simple JSON map. Because the JSON coder does not recognize the types of
the fields of the map, it deserializes the record as a simple map type.
> BigQueryIO result different for REPEATED field between DirectRunner and
> DataflowRunner
> --------------------------------------------------------------------------------------
>
> Key: BEAM-2767
> URL: https://issues.apache.org/jira/browse/BEAM-2767
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp, runner-dataflow, runner-direct
> Affects Versions: 2.0.0
> Reporter: Andre
> Assignee: Thomas Groh
> Priority: Minor
>
> When running a query against BigQueryIO with a REPEATED RECORD field the
> behavior is different between DirectRunner and DataflowRunner. The field
> containing the repeated record has to be cast to access the records.
> Apparently the following implementations work for each runner but I would
> expect them to be the same as my pipeline otherwise only runs on one.
> DirectRunner:
> {code:java}
> ArrayList<LinkedHashMap> orderLines = (ArrayList<LinkedHashMap>)
> c.element().get("RepeatedField");
> {code}
> DataflowRunner:
> {code:java}
> ImmutableList<TableRow> orderLines = (ImmutableList<TableRow>)
> c.element().get("RepeatedField");
> {code}
>
> For example when using the ImmutableList implementation on DirectRunner the
> following exception is thrown:
> {code:java}
> java.lang.ClassCastException: java.util.ArrayList cannot be cast to
> com.google.common.collect.ImmutableList
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
