Not sure, I will look into it. My prediction is that we go with HDFS rather than S3 linked destination tables soon anyway. In the mean time we're going to instrument the code to just retry failed queries.
- Neal On Thu, Aug 6, 2009 at 4:03 PM, Zheng Shao <[email protected]> wrote: > Hi Neal, > > It seems like that the exception is thrown at: > org.apache.hadoop.fs.s3.S3Exception:>> > org.jets3t.service.S3ServiceException > > Is there any parameter to adjust the s3 file system? > > Zheng > > On Thu, Aug 6, 2009 at 7:32 AM, Neal Richter<[email protected]> wrote: > > Any response on this? We had some hadoop output to S3 failures last week > as > > well. Should I post to hive-dev list? > > > > Thanks! - Neal > > > > On Thu, Jul 30, 2009 at 7:11 PM, Neal Richter <[email protected]> > wrote: > >> > >> Below is the output of Hive for an INSERT-SELECT from one 'EXTERNAL' > table > >> to another. This is running in EC2 and the external tables have > partitions > >> registered as path-keys in S3. The final upload of data to S3 fails. > This > >> does not always happen, yet when it does the only option seems to be to > >> re-run the job. > >> > >> Is there another way? Perhaps a way to tell hive to retry data-uploads > to > >> S3 if a failure occurs? - Neal > >> > >> > >> [snip] > >> map = 100%, reduce =52% > >> map = 100%, reduce =88% > >> map = 100%, reduce =93% > >> map = 100%, reduce =100% > >> map = 100%, reduce =49% > >> map = 100%, reduce =100% > >> map = 100%, reduce =45% > >> map = 100%, reduce =86% > >> map = 100%, reduce =100% > >> map = 100%, reduce =67% > >> map = 100%, reduce =12% > >> map = 100%, reduce =14% > >> map = 100%, reduce =0% > >> map = 100%, reduce =17% > >> map = 100%, reduce =0% > >> map = 100%, reduce =45% > >> Ended Job = job_200907281406_0315 > >> Job Commit failed with exception > >> > 'org.apache.hadoop.hive.ql.metadata.HiveException(org.apache.hadoop.fs.s3.S3Exception: > >> org.jets3t.service.S3ServiceException: Encountered too many S3 Internal > >> Server errors (6), aborting request.)' > >> FAILED: Execution Error, return code 3 from > >> org.apache.hadoop.hive.ql.exec.ExecDriver > >> > > > > > > > > -- > Yours, > Zheng >
