Generally the number of tasks is a hint to InputFormat.getSplits() based on number of splits wanted and it can potentially return different number of splits.
If you see ExportInputFormat, the split size is determined by combined file sizes divided by requested num mappers. This is a integer division and there is a potential for getting additional splits. Thanks Venkat On Tue, Jul 22, 2014 at 11:41 PM, David Robson <david.rob...@software.dell.com> wrote: > Yes OraOop is not enabled by default - and also this particular issue is only > when using partitioning - so they would have turned on some extra flags. Then > they also have to be using 1 more mapper than requested as you said - so it's > a pretty specific combination of events - so I don't think we need to hold up > the release for it. > > Do you recall what the behaviour should be in regards to this - the > documentation leads me to believe if we request 4 mappers we should get 4 > mappers? If this is only a request and we may get more - perhaps we should > update the documentation about this. Otherwise we could change the code to > guarantee no more than 4 mappers? > > -----Original Message----- > From: Abraham Elmahrek [mailto:a...@cloudera.com] > Sent: Wednesday, 23 July 2014 3:16 PM > To: dev@sqoop.apache.org > Subject: Re: Final Jiras and Release Date > > Hey guys, > > AFAIK we can fall back onto the original connector? With that being said, > OraOop is an awesome connector... So i'm completely open to waiting for a fix > if it isn't too large. > > David, I'm assuming you're referring to the "-m" option? I do recall some > cases where there may be one more split than desired if splitting is not > clean. Hopefully Venkat has more insight! > > -Abe > > > On Tue, Jul 22, 2014 at 9:42 PM, David Robson < > david.rob...@software.dell.com> wrote: > >> Hey Venkat, >> >> I had a look into that issue - it seems the problem is even though you >> request 4 mappers - Sqoop runs 5 mappers. Is the number of mappers >> meant to be guaranteed? For example if I say 4 mappers, should I get 4 >> mappers? I guess there could be potential to get less mappers if the >> data was for example 1 row - then that would only be processed by 1 >> mapper. But should it be possible to get more mappers than requested? >> >> OraOop is assuming there will be no more mappers than requested - so >> if this is a valid scenario we would need to modify OraOop. On the >> other hand if this is unexpected behaviour then we should look at why >> the num mappers is not working? >> >> Either way I don't think we need to fix it for this release - as it >> seems customers would be unlikely to hit this issue. >> >> David >> >> -----Original Message----- >> From: Venkat Ranganathan [mailto:vranganat...@hortonworks.com] >> Sent: Wednesday, 23 July 2014 1:04 PM >> To: dev@sqoop.apache.org >> Subject: Re: Final Jiras and Release Date >> >> Abe >> >> Do we want to target SQOOP-1388 also for 1.4.5 - the one that Vidya >> has raised for one of the Oracle connector export failure? >> >> My avro patch is in RB. >> >> Thanks >> >> Venkat >> >> On Tue, Jul 22, 2014 at 6:12 PM, Abraham Elmahrek <a...@cloudera.com> >> wrote: >> > Hey guys, >> > >> > It looks like we still have a couple of Jiras still open. Let's see >> > how things look tomorrow, but let's have these be the last two Jiras >> > slotted for this release. >> > >> > -Abe >> > >> > >> > On Fri, Jul 18, 2014 at 1:20 PM, Venkat Ranganathan < >> > vranganat...@hortonworks.com> wrote: >> > >> >> Sounds good. I have already uploaded the patch for SQOOP-1358. It >> >> needs to be reviewed by a committer and then committed if no >> >> further changes are needed (or more work needed otherwise). >> >> >> >> Thanks for driving the release! >> >> >> >> Venkat >> >> >> >> On Fri, Jul 18, 2014 at 10:58 AM, Jarek Jarcec Cecho >> >> <jar...@apache.org> >> >> wrote: >> >> > Seems reasonable to me, thank you for driving the release Abe! >> >> > >> >> > Jarcec >> >> > >> >> > On Jul 18, 2014, at 10:40 AM, Abraham Elmahrek <a...@cloudera.com> >> wrote: >> >> > >> >> >> Hey guys, >> >> >> >> >> >> I haven't seen a -1 on this suggestion. So let's create a >> >> >> release branch come Wednesday July 23rd. >> >> >> >> >> >> -Abe >> >> >> >> >> >> >> >> >> On Wed, Jul 16, 2014 at 11:38 AM, Abraham Elmahrek >> >> >> <a...@cloudera.com> >> >> wrote: >> >> >> >> >> >>> Are there any objections to this branch date? >> >> >>> >> >> >>> -Abe >> >> >>> >> >> >>> >> >> >>> On Mon, Jul 14, 2014 at 1:22 PM, Abraham Elmahrek >> >> >>> <a...@apache.org> >> >> wrote: >> >> >>> >> >> >>>> Hey guys, >> >> >>>> >> >> >>>> It looks like there are a couple of Jiras that are in progress >> >> >>>> and slotted for the 1.4.5 release. If we aim for July 23rd to >> >> >>>> create a >> >> release >> >> >>>> branch, does that give every one enough time to finish up what >> >> >>>> they're working on? >> >> >>>> >> >> >>>> See https://issues.apache.org/jira/browse/SQOOP-1353 for more >> >> >>>> information. >> >> >>>> >> >> >>>> -Abe >> >> >>>> >> >> >>> >> >> >>> >> >> > >> >> >> >> -- >> >> CONFIDENTIALITY NOTICE >> >> NOTICE: This message is intended for the use of the individual or >> >> entity to which it is addressed and may contain information that is >> >> confidential, privileged and exempt from disclosure under >> >> applicable law. If the reader of this message is not the intended >> >> recipient, you are hereby notified that any printing, copying, >> >> dissemination, distribution, disclosure or forwarding of this >> >> communication is strictly prohibited. If you have received this >> >> communication in error, please contact the sender immediately and >> >> delete it from your >> system. Thank You. >> >> >> >> -- >> CONFIDENTIALITY NOTICE >> NOTICE: This message is intended for the use of the individual or >> entity to which it is addressed and may contain information that is >> confidential, privileged and exempt from disclosure under applicable >> law. If the reader of this message is not the intended recipient, you >> are hereby notified that any printing, copying, dissemination, >> distribution, disclosure or forwarding of this communication is >> strictly prohibited. If you have received this communication in error, >> please contact the sender immediately and delete it from your system. Thank >> You. >> -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
