lukecwik commented on issue #22945: URL: https://github.com/apache/beam/issues/22945#issuecomment-1234919430
An idea to fix this would be to swap out the concatenate combiner with one which outputs the records and the metadata record for PCollection list view ensuring that the metadata record only exists for the set of records that are being written. This would work for accumulating and discarding mode and we could ensure that there is only a single metadata record for all the values greatly simplifying down the access code since we wouldn't need to perform the mapping from index to position and sub position since there would only ever be one single metadata record. This should be pipeline update compatible since the coder that we used would have metadata records and records already encoded in it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
