It may be some kind of hostname name or reverse lookup delay, either on the origination or destination side.
On Thu, Aug 20, 2009 at 10:43 AM, Raghu Angadi <[email protected]>wrote: > Ananth T. Sarathy wrote: > >> it's on s3. and it always happens. >> > > I have no experience with S3. You might want to check out S3 forums. It > can't be normal for S3 either.. there must be something missing > (configuration, ACLs... ). > > Raghu. > > > Ananth T Sarathy >> >> >> On Wed, Aug 19, 2009 at 4:35 PM, Raghu Angadi <[email protected]> >> wrote: >> >> Ananth T. Sarathy wrote: >>> >>> Also, I just want to clear... the delay seems to at the intial >>>> >>>> (read = in.read(buf)) >>>> >>>> It the file on HDFS (over S3) or S3? >>> >>> Does it always happen? >>> >>> Raghu. >>> >>> >>> after the first time into the loop it flies... >>> >>>> Ananth T Sarathy >>>> >>>> >>>> On Wed, Aug 19, 2009 at 1:58 PM, Raghu Angadi <[email protected]> >>>> wrote: >>>> >>>> Edward Capriolo wrote: >>>> >>>>> On Wed, Aug 19, 2009 at 11:11 AM, Edward Capriolo < >>>>> >>>>>> [email protected] >>>>>> >>>>>> wrote: >>>>>>> It would be as fast as underlying filesystem goes. >>>>>>> >>>>>>> I would not agree with that statement. There is overhead. >>>>>>>> You might be misinterpreting my comment. There is of course some >>>>>>>> over >>>>>>>> >>>>>>> head >>>>> (at the least the procedure calls).. depending on you underlying >>>>> filesystem, >>>>> there could be extra buffer copies and CRC overhead. But none of that >>>>> explains transfer as slow as 1 MBps (if my interpretation of of results >>>>> is >>>>> correct). >>>>> >>>>> Raghu. >>>>> >>>>> >>>>> >>>>> >> > -- Pro Hadoop, a book to guide you from beginner to hadoop mastery, http://www.amazon.com/dp/1430219424?tag=jewlerymall www.prohadoopbook.com a community for Hadoop Professionals
