This sounds like it is related to the problem I'm trying to solve. (In my case having a Java POJO containing a protobuf backed-class and trying to generate a Beam Schema from it.)
I would be very interested to a solution to this as well :) On Tue, Jul 7, 2020 at 2:22 PM <[email protected]> wrote: > Hi All, > > > > I have a BEAM pipeline where I am reading data from some parquet files and > converting them into a different format based on protobuf generated classes. > > > > I wish to associate a schema (derived from the protobuf classes) for my > PCollections. What is the appropriate way to do this with > protobuf-generated classes? > > > > Code excerpt: > > > > PCollection<Fx> result = input.apply("FXFilePattern", FileIO.*match* > ().filepattern(fxDataFilePattern)) > .apply("FXReadMatches", FileIO.*readMatches*()) > .apply("FXReadParquetFile", ParquetIO.*readFiles*(fxAvroSchema)) > .apply("MapFXToProto", ParDo.*of*(MapFXToProto.*of*())); > boolean hasSchema = result.hasSchema(); // returns false > > > > With thanks in advance. > > > > Kind regards, > > > > Rob > > > > *Robert Butcher* > > *Technical Architect | Foundry/SRS | NatWest Markets* > > WeWork, 10 Devonshire Square, London, EC2M 4AE > > Mobile +44 (0) 7414 730866 > > > > This email is classified as *CONFIDENTIAL* unless otherwise stated. > > > > This communication and any attachments are confidential and intended > solely for the addressee. If you are not the intended recipient please > advise us immediately and delete it. Unless specifically stated in the > message or otherwise indicated, you may not duplicate, redistribute or > forward this message and any attachments are not intended for distribution > to, or use by any person or entity in any jurisdiction or country where > such distribution or use would be contrary to local law or regulation. > NatWest Markets Plc or any affiliated entity ("NatWest Markets") accepts > no responsibility for any changes made to this message after it was sent. > Unless otherwise specifically indicated, the contents of this > communication and its attachments are for information purposes only and > should not be regarded as an offer or solicitation to buy or sell a product > or service, confirmation of any transaction, a valuation, indicative price > or an official statement. Trading desks may have a position or interest > that is inconsistent with any views expressed in this message. In > evaluating the information contained in this message, you should know that > it could have been previously provided to other clients and/or internal > NatWest Markets personnel, who could have already acted on it. > NatWest Markets cannot provide absolute assurances that all electronic > communications (sent or received) are secure, error free, not corrupted, > incomplete or virus free and/or that they will not be lost, mis-delivered, > destroyed, delayed or intercepted/decrypted by others. Therefore NatWest > Markets disclaims all liability with regards to electronic communications > (and the contents therein) if they are corrupted, lost destroyed, delayed, > incomplete, mis-delivered, intercepted, decrypted or otherwise > misappropriated by others. > Any electronic communication that is conducted within or through NatWest > Markets systems will be subject to being archived, monitored and produced > to regulators and in litigation in accordance with NatWest Markets’ policy > and local laws, rules and regulations. Unless expressly prohibited by local > law, electronic communications may be archived in countries other than the > country in which you are located, and may be treated in accordance with the > laws and regulations of the country of each individual included in the > entire chain. > Copyright NatWest Markets Plc. All rights reserved. See > https://www.nwm.com/disclaimer for further risk disclosure. >
