We have been using Beam for a bit now. However we just turned on the dataflow shuffle service and were very surprised that the shuffled data amounts were quadruple the amounts we expected.
Turns out that the file writing TextIO is doing shuffles within itself. Is there a way to prevent shuffling in the writing phase? Thanks, Shannon Duncan
