Re: Bulk storage of protobuf records in files

2023-06-05 Thread Shammon FY
Hi Ryan, What I usually encounter is writing Protobuf format data to systems such as Kafka, and I have never encountered writing to a file yet. Best, Shammon FY On Mon, Jun 5, 2023 at 10:50 PM Martijn Visser wrote: > Hey Ryan, > > I've never encountered a use case for writing Protobuf

Re: Bulk storage of protobuf records in files

2023-06-05 Thread Martijn Visser
Hey Ryan, I've never encountered a use case for writing Protobuf encoded files to a filesystem. Best regards, Martijn On Fri, May 26, 2023 at 6:39 PM Ryan Skraba via user wrote: > Hello all! > > I discovered while investigating FLINK-32008[1] that we can write to the > filesystem connector

Bulk storage of protobuf records in files

2023-05-26 Thread Ryan Skraba via user
Hello all! I discovered while investigating FLINK-32008[1] that we can write to the filesystem connector with the protobuf format, but today, the resulting file is pretty unlikely to be useful or rereadable. There's no real standard for storing many protobuf messages in a single file container,