GitHub user zhangxffff added a comment to the discussion: Add a new backend: Bolt
> @zhangxffff thanks for the explanation! > > > For the shuffle reader > > I'm curious about `Morsel Driven` mentioned above, I guess Bolt use parallel > mode to execute task (Gluten use serial mode). does fetching raw shuffle > data, decompression, deserialization all done in one operator? how many > driver does shuffle read operator generate? if only one driver of shuffle > read op, seems no major difference with Gluten, if two, fetching, > decompression, deserialization all in one driver? After offload as a Bolt operator, shuffle read support multiple drivers, each driver's shuffle reader get different InputStreams (since Gluten get an iterator of InputStream from ShuffleBlockFetcherIterator, which contains multiple InputStreams), so they can fetching data, decompression, deserialization in parallel. GitHub link: https://github.com/apache/incubator-gluten/discussions/10929#discussioncomment-15037342 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
