That's right, it is much easier to control reducers compared to mappers. Mappers are controlled by hadoop without a similar simple knob. You can set some config parameters to give hadoop an idea of how many map tasks to launch.
This stack overflow may be helpful: https://stackoverflow.com/questions/6885441/setting-the-number-of-map-tasks-and-reduce-tasks On Tue, Aug 28, 2018 at 6:40 PM Tianshan Cui <[email protected]> wrote: > Hi scalding users, > > I am curious that is there a good way to control the number of mappers per > individual MR stage? It seems like we can easily control the number of > reducers via withReducers. However, I didn't find there is a good way to > do the similar for mappers. I know we could setup the job config and tune > the split size, but that will affect the whole flow. > > Any ideas? Maybe I missed something? :) > > Thanks, > Tianshan > > -- > You received this message because you are subscribed to the Google Groups > "Scalding Development" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- P. Oscar Boykin, Ph.D. | http://twitter.com/posco | http://pobox.com/~boykin -- You received this message because you are subscribed to the Google Groups "Scalding Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
