Thanks Sean.
On Thu, Feb 19, 2015 at 5:33 PM, Sean Owen <so...@cloudera.com> wrote:
> I think that the newer Hadoop API does not expose this suggested min
> partitions parameter like the old one did. I believe you can try
> setting mapreduce.input.fileinputformat.split.{min,max}size instead on
> the Hadoop Configuration to suggest a max/min split size, and
> therefore bound the number of partitions you get back.
>
> On Thu, Feb 19, 2015 at 11:07 AM, twinkle sachdeva
> <twinkle.sachd...@gmail.com> wrote:
> > Hi,
> >
> > In our job, we need to process the data in small chunks, so as to avoid
> GC
> > and other stuff. For this, we are using old API of hadoop as that let us
> > specify parameter like minPartitions.
> >
> > Does any one knows, If there a way to do the same via newHadoopAPI also?
> > How that way will be different from older API?
> >
> > I am little bit aware of split size stuff, but not much aware regarding
> any
> > promise that minimum number of partitions criteria gets satisfied or not.
> >
> > Any pointers will be of help.
> >
> > Thanks,
> > Twinkle
>
