Hi Brian, Many thanks for your mail.
Yes I figured that one out in the end from the docs, but many thanks for confirming. I did subsequently discover some other issues with protoBuf-derived schemas (essentially they don’t seem to be properly supported by BigQueryIO.Write or allow for optional fields) but I posted a separate message on the dev channel covering this. Kind regards, Rob From: Brian Hulette [mailto:[email protected]] Sent: 18 August 2020 20:50 To: user Subject: Re: Registering Protobuf schema ********************************************* "This is an external email. Do you know who has sent it? Can you be sure that any links and attachments contained within it are safe? If in any doubt, use the Phishing Reporter Button in your Outlook client or forward the email as an attachment to ~ I've Been Phished" ********************************************* Hi Robert, Sorry for the late reply on this. I think you should be able to do this by registering it in your pipeline's SchemaRegistry manually, like so: Pipeline p; p.getSchemaRegistry().registerSchemaProvider(Fx.class, ProtoMessageSchema.class); Of course this isn't quite as nice as just adding the DefualtSchema annotation to a class you control. Maybe we should consider some global config that would always use schemas for proto-generated classes. Brian On Sun, Jul 12, 2020 at 9:56 AM Kaymak, Tobias <[email protected]<mailto:[email protected]>> wrote: This sounds like it is related to the problem I'm trying to solve. (In my case having a Java POJO containing a protobuf backed-class and trying to generate a Beam Schema from it.) I would be very interested to a solution to this as well :) On Tue, Jul 7, 2020 at 2:22 PM <[email protected]<mailto:[email protected]>> wrote: Hi All, I have a BEAM pipeline where I am reading data from some parquet files and converting them into a different format based on protobuf generated classes. I wish to associate a schema (derived from the protobuf classes) for my PCollections. What is the appropriate way to do this with protobuf-generated classes? Code excerpt: PCollection<Fx> result = input.apply("FXFilePattern", FileIO.match().filepattern(fxDataFilePattern)) .apply("FXReadMatches", FileIO.readMatches()) .apply("FXReadParquetFile", ParquetIO.readFiles(fxAvroSchema)) .apply("MapFXToProto", ParDo.of(MapFXToProto.of())); boolean hasSchema = result.hasSchema(); // returns false With thanks in advance. Kind regards, Rob Robert Butcher Technical Architect | Foundry/SRS | NatWest Markets WeWork, 10 Devonshire Square, London, EC2M 4AE Mobile +44 (0) 7414 730866<tel:+44%207414%20730866> This email is classified as CONFIDENTIAL unless otherwise stated. This communication and any attachments are confidential and intended solely for the addressee. If you are not the intended recipient please advise us immediately and delete it. Unless specifically stated in the message or otherwise indicated, you may not duplicate, redistribute or forward this message and any attachments are not intended for distribution to, or use by any person or entity in any jurisdiction or country where such distribution or use would be contrary to local law or regulation. NatWest Markets Plc or any affiliated entity ("NatWest Markets") accepts no responsibility for any changes made to this message after it was sent. Unless otherwise specifically indicated, the contents of this communication and its attachments are for information purposes only and should not be regarded as an offer or solicitation to buy or sell a product or service, confirmation of any transaction, a valuation, indicative price or an official statement. Trading desks may have a position or interest that is inconsistent with any views expressed in this message. In evaluating the information contained in this message, you should know that it could have been previously provided to other clients and/or internal NatWest Markets personnel, who could have already acted on it. NatWest Markets cannot provide absolute assurances that all electronic communications (sent or received) are secure, error free, not corrupted, incomplete or virus free and/or that they will not be lost, mis-delivered, destroyed, delayed or intercepted/decrypted by others. Therefore NatWest Markets disclaims all liability with regards to electronic communications (and the contents therein) if they are corrupted, lost destroyed, delayed, incomplete, mis-delivered, intercepted, decrypted or otherwise misappropriated by others. Any electronic communication that is conducted within or through NatWest Markets systems will be subject to being archived, monitored and produced to regulators and in litigation in accordance with NatWest Markets’ policy and local laws, rules and regulations. Unless expressly prohibited by local law, electronic communications may be archived in countries other than the country in which you are located, and may be treated in accordance with the laws and regulations of the country of each individual included in the entire chain. Copyright NatWest Markets Plc. All rights reserved. See https://www.nwm.com/disclaimer for further risk disclosure. This communication and any attachments are confidential and intended solely for the addressee. If you are not the intended recipient please advise us immediately and delete it. Unless specifically stated in the message or otherwise indicated, you may not duplicate, redistribute or forward this message and any attachments are not intended for distribution to, or use by any person or entity in any jurisdiction or country where such distribution or use would be contrary to local law or regulation. NatWest Markets Plc, NatWest Markets N.V., NatWest Markets Securities Japan Limited and/or NatWest Markets Securities Inc. (collectively "NatWest Markets") accepts no responsibility for any changes made to this message after it was sent. This communication, where prepared by the sales and trading desk or desk strategists, may be marketing material, desk strategy and/or trader commentary. It is not a product of the research department. This material may constitute an invitation to consider entering into a derivatives transaction under U.S. CFTC Regulations sections 1.71 and 23.605, where applicable, but is not a binding offer to buy/sell any financial instrument. The views of the author may differ from others at NatWest Markets. Unless otherwise specifically indicated, the contents of this communication and its attachments are for information purposes only and should not be regarded as an offer or solicitation to buy or sell a product or service, confirmation of any transaction, a valuation, indicative price or an official statement. Trading desks may have a position or interest that is inconsistent with any views expressed in this message. In evaluating the information contained in this message, you should know that it could have been previously provided to other clients and/or internal NatWest Markets personnel, who could have already acted on it. NatWest Markets cannot provide absolute assurances that all electronic communications (sent or received) are secure, error free, not corrupted, incomplete or virus free and/or that they will not be lost, mis-delivered, destroyed, delayed or intercepted/decrypted by others. Therefore NatWest Markets disclaims all liability with regards to electronic communications (and the contents therein) if they are corrupted, lost destroyed, delayed, incomplete, mis-delivered, intercepted, decrypted or otherwise misappropriated by others. Any electronic communication that is conducted within or through NatWest Markets systems will be subject to being archived, monitored and produced to regulators and in litigation in accordance with NatWest Markets’ policy and local laws, rules and regulations. Unless expressly prohibited by local law, electronic communications may be archived in countries other than the country in which you are located, and may be treated in accordance with the laws and regulations of the country of each individual included in the entire chain. Copyright © NatWest Markets Plc. All rights reserved. See https://www.nwm.com/disclaimer for further risk disclosure (the agency arrangements referred to in the further risk disclosure between NatWest Markets Plc and NatWest Markets N.V. are not applicable to branches of NatWest Markets N.V.).
