[
https://issues.apache.org/jira/browse/HADOOP-16998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17087838#comment-17087838
]
Steve Loughran commented on HADOOP-16998:
-----------------------------------------
can tag this with the specific version of hadoop you are having problems with.
Try hadoop branch-3/trunk if not already done -that is, something with
HADOOP-16785 in, which tried to harden thos close work. If that's not enough,
at least it has a start with where to begin testing this.
Patches up on github as PRs for review, thanks.
> WASB : NativeAzureFsOutputStream#close() throwing
> java.lang.IllegalArgumentException instead of IOE which causes HBase RS to
> get aborted
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-16998
> URL: https://issues.apache.org/jira/browse/HADOOP-16998
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
> Reporter: Anoop Sam John
> Assignee: Anoop Sam John
> Priority: Major
> Attachments: HADOOP-16998.patch
>
>
> During HFile create, at the end when called close() on the OutputStream,
> there is some pending data to get flushed. When this flush happens, an
> Exception is thrown back from Storage. The Azure-storage SDK layer will throw
> back IOE. (Even if it is a StorageException thrown from the Storage, the SDK
> converts it to IOE.) But at HBase, we end up getting IllegalArgumentException
> which causes the RS to get aborted. If we get back IOE, the flush will get
> retried instead of aborting RS.
> The reason is this
> NativeAzureFsOutputStream uses Azure-storage SDK's BlobOutputStreamInternal.
> But the BlobOutputStreamInternal is wrapped within a SyncableDataOutputStream
> which is a FilterOutputStream. During the close op, NativeAzureFsOutputStream
> calls close on SyncableDataOutputStream and it uses below method from
> FilterOutputStream
> {code}
> public void close() throws IOException {
> try (OutputStream ostream = out) {
> flush();
> }
> }
> {code}
> Here the flush call caused an IOE to be thrown to here. The finally will
> issue close call on ostream (Which is an instance of BlobOutputStreamInternal)
> When BlobOutputStreamInternal#close() is been called, if there was any
> exception already occured on that Stream, it will throw back the same
> Exception
> {code}
> public synchronized void close() throws IOException {
> try {
> // if the user has already closed the stream, this will throw a
> STREAM_CLOSED exception
> // if an exception was thrown by any thread in the
> threadExecutor, realize it now
> this.checkStreamState();
> ...
> }
> private void checkStreamState() throws IOException {
> if (this.lastError != null) {
> throw this.lastError;
> }
> }
> {code}
> So here both try and finally block getting Exceptions and Java uses
> Throwable#addSuppressed()
> Within this method if both Exceptions are same objects, it throws back
> IllegalArgumentException
> {code}
> public final synchronized void addSuppressed(Throwable exception) {
> if (exception == this)
> throw new
> IllegalArgumentException(SELF_SUPPRESSION_MESSAGE, exception);
> ....
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
