[
https://issues.apache.org/jira/browse/HADOOP-997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12472832
]
[EMAIL PROTECTED] commented on HADOOP-997:
------------------------------------------
I tried the patch and got this:
[EMAIL PROTECTED]:/mnt/hadoop-trunk$ ./bin/hadoop fs -fs s3://AWS_ID:[EMAIL
PROTECTED] -put /mnt/search/searcher.dir/index-sorted /1998/20070207040835-1998
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.fs.s3.$Proxy0.initialize(Unknown Source)
at org.apache.hadoop.fs.s3.S3FileSystem.initialize(S3FileSystem.java:63)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:160)
at org.apache.hadoop.fs.FileSystem.getNamed(FileSystem.java:119)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:91)
at org.apache.hadoop.fs.FsShell.init(FsShell.java:39)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:796)
at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:189)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:895)
Caused by: java.lang.IllegalAccessException: Class
org.apache.hadoop.io.retry.RetryInvocationHandler can not access a member of
class org.apache.hadoop.fs.s3.FileSystemStore with modifiers "public abstract"
at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:65)
at java.lang.reflect.Method.invoke(Method.java:578)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:62)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:42)
... 9 more
Before the patch, the above command-line worked.
[EMAIL PROTECTED]:/mnt/hadoop-trunk$ svn info
...
Revision: 507114
...
Other comments on patch,
+ The retry mechanism looks sweet.
+ Is it true that the Jets3tFileSystemStore#initialize runs after the retry
proxy has been set up so initialization will be retried if an Exception? Is
this intentional? Perhaps initialization should be outside the purview of
retry.
+ Minor comment: In version 2 of the retry mechanism, you'd be able to narrow
retries to happen only for certain exception types: e.g. for IOExceptions only
rather than for Exception as its currently written.
+ In S3OutputStream#newBackupFile, you replace File createTempFile with your
own tmp file name fabricator. Any reason why you didn't stay using
createTempFile, just use the override with signature instead:
http://java.sun.com/j2se/1.5.0/docs/api/java/io/File.html#createTempFile(java.lang.String,%20java.lang.String,%20java.io.File)?
(You do this in a few places in this patch).
+ In S3FileSystem.java#createDefaultStore, should the retry params be exposed
as configurables? (I'm fine if the rejoinder is that this makes for too much
configuration).
+ On initialize, should Jets3tFileSystemStore clean up its tmp directory? (If
crash, deleteOnExit won't get a chance to run).
+ If an IOException in retreiveBlock in Jets3tFileSystemStore, you do a
closeQuietly on the out stream. Its followed by a finally that does
closeQuietly on both the in and out stream. I don't follow whats going on
here. Why not just leave all closing to the finally block? (Suggest adding a
comment on why the special case handling of out stream in IOE block).
Good stuff Tom.
> Implement S3 retry mechanism for failed block transfers
> -------------------------------------------------------
>
> Key: HADOOP-997
> URL: https://issues.apache.org/jira/browse/HADOOP-997
> Project: Hadoop
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.11.0
> Reporter: Tom White
> Assigned To: Tom White
> Attachments: HADOOP-997.patch
>
>
> HADOOP-882 improves S3FileSystem so that when certain communications problems
> with S3 occur the operation is retried. However, the retry mechanism cannot
> handle a block transfer failure, since blocks may be very large and we don't
> want to buffer them in memory. This improvement is to write a wrapper (using
> java.lang.reflect.Proxy if possible - see discussion in HADOOP-882) that can
> retry block transfers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
