Which direct runner? They are language specific. On Mon, Mar 13, 2023, 11:27 AM [email protected] <[email protected]> wrote:
> Hi guys, > > We are trying to run our pipeline using direct runner and the input > dataset is a large amount of HDFS files (few hundred of GB data) > > We experienced OOM issue crash. Then inside the direct runner document, I > realized direct runner loads the whole dataset into the memory. > > Is there any way we can avoid this OOM issue? > > Regards > > ------------------------------------------------------------- > > Wilson(Xiaoshuang) Wang > Sr. Software Engineer >
