Hi Dev's I was trying to understand the transformations created for the following pipeline, which seems to be pretty simple from the looks of it. But the graph created seems to be pretty complex. I have attached a rough sketch of the graph that I understood from debugging the code below [1]. Was a little bit puzzled as to why so many transformations are introduced for the write() operation, is this the normal behavior for I/O operations or am I missing something? doesn't this introduced a lot of unwanted overhead to a simple operation?
PCollection<String> result = p.apply(GenerateSequence.from(0).to(10)) .apply( ParDo.of( new DoFn<Long, String>() { @ProcessElement public void processElement(ProcessContext c) throws Exception { c.output(c.element().toString()); } })); result.apply(TextIO.write().to(new URI(resultPath).getPath() + "/part")); [1] beam graph <https://docs.google.com/drawings/d/1Ptk8XQiiee5vymXrUZMYNQIS8iEnexKD4ucbZ4Uk-CE/edit?usp=drive_web> Best Regards, Pulasthi -- Pulasthi S. Wickramasinghe PhD Candidate | Research Assistant School of Informatics and Computing | Digital Science Center Indiana University, Bloomington cell: 224-386-9035