MiguelAnzoWizeline commented on a change in pull request #14586:
URL: https://github.com/apache/beam/pull/14586#discussion_r644373119
##########
File path:
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageReadTest.java
##########
@@ -1351,4 +1353,20 @@ public void testReadFromBigQueryIOWithTrimmedSchema()
throws Exception {
p.run();
}
+
+ private static org.apache.arrow.vector.types.pojo.Field field(
Review comment:
Hi @TheNeuralBit I’m still having some problems with this
implementation, I followed the examples you gave me, which helped me understand
more the Arrow side, but when I was implementing it I found that
`ReadRowsResponse` uses a different `ArrowRecordBatch` implementation found in
`com.google.cloud.bigquery.storage.v1.ArrowRecordBatch` and with that now I
have two questions:
- In order to convert from
`org.apache.arrow.vector.ipc.message.ArrowRecordBatch` to
`com.google.cloud.bigquery.storage.v1.ArrowRecordBatch`, from what I can see I
would need to write the `org.apache.arrow.vector.ipc.message.ArrowRecordBatch`
to an InputStream and then
[parse](https://googleapis.dev/java/google-cloud-bigquerystorage/1.8.3/com/google/cloud/bigquery/storage/v1/ArrowRecordBatch.html#parseFrom-com.google.protobuf.CodedInputStream-)
[it](https://googleapis.dev/java/google-cloud-bigquerystorage/1.8.3/com/google/cloud/bigquery/storage/v1/ArrowRecordBatch.html#parseFrom-java.io.InputStream-)
to `com.google.cloud.bigquery.storage.v1.ArrowRecordBatch`, is that correct?
and is there a specific coder needed to do that? I found the `Avro`
implementation uses a `GenericDatumWriter`. Is there any other more
straightforward way to do that conversion?
- Is it correct for the Arrow
[code](https://github.com/apache/beam/blob/ed63ed19c67c7c0a1ce7c2888c44fb1238419c32/sdks/java/extensions/arrow/src/main/java/org/apache/beam/sdk/extensions/arrow/ArrowConversion.java)
to use `org.apache.arrow.vector.ipc.message.ArrowRecordBatch` instead of
`com.google.cloud.bigquery.storage.v1.ArrowRecordBatch`?
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
