Hi Dev's

I was trying to understand the transformations created for the
following pipeline, which seems to be pretty simple from the looks of it.
But the graph created seems to be pretty complex. I have attached a rough
sketch of the graph that I understood from debugging the code below [1].
Was a little bit puzzled as to why so many transformations are introduced
for the write() operation, is this the normal behavior for I/O operations
or am I missing something? doesn't this introduced a lot of unwanted
overhead to a simple operation?

PCollection<String> result =
        p.apply(GenerateSequence.from(0).to(10))
            .apply(
                ParDo.of(
                    new DoFn<Long, String>() {
                      @ProcessElement
                      public void processElement(ProcessContext c) throws
Exception {
                        c.output(c.element().toString());
                      }
                    }));

    result.apply(TextIO.write().to(new URI(resultPath).getPath() +
"/part"));


[1]
 beam graph
<https://docs.google.com/drawings/d/1Ptk8XQiiee5vymXrUZMYNQIS8iEnexKD4ucbZ4Uk-CE/edit?usp=drive_web>

Best Regards,
Pulasthi
-- 
Pulasthi S. Wickramasinghe
PhD Candidate  | Research Assistant
School of Informatics and Computing | Digital Science Center
Indiana University, Bloomington
cell: 224-386-9035

Reply via email to