Hey Matthew, Thanks also for sharing the code that you're working on. What are your plans with the connector? I could imagine that others would also be interested, so perhaps you wanted to add it to https://flink-packages.org/ ?
Best regards, Martijn Visser https://twitter.com/MartijnVisser82 https://github.com/MartijnVisser On Wed, 6 Apr 2022 at 08:36, Matthew Brown <[email protected]> wrote: > Thanks Robert! > > All working now. > > Turns out an incorrect log4j2 configuration was slurping up a warning > about converting org.apache.avro.util.Utf8 to the internal flink data > type. > I've drawn some inspiration from flink-avro and added a converter that > converts and re-orders the Avro encoded BigQuery results to the correct > flink RowData. > > Cheers, > Matt. > > On Tue, Apr 5, 2022 at 9:28 PM Robert Metzger <[email protected]> wrote: > >> Hi Matt, >> >> At first glance your code looks fine. I guess you'll need to follow the >> codepaths more with the debugger. >> Have you made sure that "reachedEnd()" returns false? >> >> >> On Tue, Apr 5, 2022 at 9:42 AM Matthew Brown <[email protected]> wrote: >> >>> Hi all, >>> >>> I'm attempting to build a Table API connector for BigQuery using the >>> BigQuery Storage API ( >>> https://cloud.google.com/bigquery/docs/reference/storage). >>> >>> I've got a base structure built out at >>> https://github.com/mnbbrown/bigquery-connector >>> There's a couple of things I have to do yet like correcting the mapping >>> between the BigQuery avro schema and flink TypeInformation, and add a test >>> suite. >>> >>> I've added it to an internal project I'm working on and "nextRecord" on >>> the InputFormat is never called. I can see open/close, >>> openInputFormat/closeInputFormat, etc being called correctly. >>> >>> Does anybody have any debugging tips? >>> >>> Thanks, >>> Matt. >>> >>> -- >>> -- >>> AU: +61 459 493 730 >>> UK: +44 7927 618921 >>> @mnbbrown >>> >> > > -- > -- > AU: +61 459 493 730 > UK: +44 7927 618921 > @mnbbrown >
