[
https://issues.apache.org/jira/browse/HDFS-10931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Clampffer updated HDFS-10931:
-----------------------------------
Attachment: HDFS-10931.HDFS-8707.001.patch
New patch. The last patch would cause sporadic failures due to errors injected
into the read pipeline during the mini stress test. In that test we run the
stress test twice: 1 with error injection and 1 without. I've removed the
EXPECT_ZERO on the failure count of the error injected case because while the
RPC engine can generally retry on errors however the read path wont and is
bound to cause false positives. What the error injected test is really looking
for is crashes due to unexpected states in the read case.
To make testing more robust in the future we could use separate error injection
for the read pipeline and the RPC layer. Then it would be possible to assert
that the number of failed reads is <= the number of injected read pipeline
errors. Barring a large run of consecutive simulated failures RPC is unlikely
to cause a false positive.
> libhdfs++: Fix object lifecycle issues in the BlockReader
> ---------------------------------------------------------
>
> Key: HDFS-10931
> URL: https://issues.apache.org/jira/browse/HDFS-10931
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
> Reporter: James Clampffer
> Assignee: James Clampffer
> Priority: Critical
> Attachments: HDFS-10931.HDFS-8707.000.patch,
> HDFS-10931.HDFS-8707.001.patch
>
>
> The BlockReader can work itself into a a state during AckRead (possibly other
> stages as well) where the pipeline posts a task for asio with a pointer back
> into itself, then promptly calls "delete this" without canceling the asio
> request. The asio task finishes and tries to acquire the lock at the address
> where the DataNodeConnection used to live - but the DN connection is no
> longer valid so it's scribbling on some arbitrary bit of memory. On some
> platforms the underlying address used by the mutex state will be handed out
> to future mutexes so the scribble breaks that state and all the locks in that
> process start misbehaving.
> This can be reproduced by using the patch from HDFS-8790 and adding more
> worker threads + a lot more reader threads.
> I'm going to fix this in two parts:
> 1) Duct tape + superglue patch to make sure that all top level continuations
> in the block reader pipeline hold a shared_ptr to the DataNodeConnection.
> Nested continuations also get a copy of the shared_ptr to make sure the
> connection is alive. This at least keeps the connection alive so that it can
> keep returning asio::operation_aborted.
> 2) The continuation stuff needs a lot of work to make sure this type of bug
> doesn't keep popping up. We've already fixed these issues in the RPC code.
> This will most likely need to be split into a few jiras.
> - Continuation "framework" can be slimmed down quite a bit, perhaps even
> removed. Near zero documentation + many implied contracts = constant bug
> chasing.
> - Add comments to actually describe what's going on in the networking code.
> This bug took significantly longer than it should have to track down because
> I hadn't worked on the BlockReader in a while.
> - No more "delete this".
> - Flatten out nested continuations e.g. the guts of BlockReaderImpl::AckRead.
> It's unclear why they were implemented like this in the first place and
> there's no comments to indicate that this was intentional.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
