I'm not sure I completely understand the question. Are you saying that you want each element to write to only one file, guaranteeing that two elements are never written to the same file?
On Mon, Dec 2, 2019 at 11:53 AM Christopher Larsen < christopher.lar...@quantiphi.com> wrote: > Hi All, > > TL/DR: can you extend FileIO.sink<T> to write one or more file per element > instead of one or more elements per file? > > In working with Thrift files we have found that since a .thrift file needs > to be compiled to generate code the order of the contents of the file are > important (ie, the namespace and includes elements need to come before > definitions are defined). > > The issue that we are facing is that by implementing FileIO.sink<Document> > we cannot determine how many Document objects are written to a file since > this is determined by the runner. This can result in more than one Document > being written to a file which will cause compilation errors. > > We know that this can be controlled by writeDynamic but since we believe > the default behavior for the connector should be to output a Document to > one or more files (depending on sharding) we were wondering how to best > accomplish this. > > Best, > Chris > > *This message contains information that may be privileged or confidential > and is the property of the Quantiphi Inc and/or its affiliates**. It is > intended only for the person to whom it is addressed. **If you are not > the intended recipient, any review, dissemination, distribution, copying, > storage or other use of all or any portion of this message is strictly > prohibited. If you received this message in error, please immediately > notify the sender by reply e-mail and delete this message in its * > *entirety* >
