[
https://issues.apache.org/jira/browse/HDFS-14316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16779089#comment-16779089
]
Deshui Yu commented on HDFS-14316:
----------------------------------
Hi, [~elgoiri], in RouterClientProtocol.java, line 238~251:
238 try {
239 RemoteLocation createLocation = rpcServer.getCreateLocation(src);
240 return (HdfsFileStatus) rpcClient.invokeSingle(createLocation,
method);
241 } catch (Exception e) {
242 if (!rpcServer.isPathFaultTolerant(src)) {
243 throw e;
244 }
245 LOG.error("Cannot create {}: {}", src, e.getMessage());
246 List<RemoteLocation> locations =
247 rpcServer.getLocationsForPath(src, true);
248 LOG.info("{} supports failed subclusters, trying: {}", src,
locations);
249 return rpcClient.invokeSequential(
250 locations, method, HdfsFileStatus.class, null);
251 }
We use try-fail-retry mode to handle subcluster failure and continue to write
in next available(maybe) subcluster. Here're some concerns from my perspective:
1. Can we make sure the RemoteLocation list returned in line 246 will not
contain the failed subcluster anymore (especially in Random Order), or another
failed subcluster? If not, maybe line 249 will also throw exception in catch{}
clause.
2. try-fail-retry mode is straight-forward and simple to implement, but the
latency is increased once a subclass fail since the create operation requires
2-rounds rpc call to namenodes. Since router has a global view about the
availability of subclusters, can we utilize this global knowledge to skip the
failed subcluster rather than try-fail-retry?
> RBF: Support unavailable subclusters for mount points with multiple
> destinations
> --------------------------------------------------------------------------------
>
> Key: HDFS-14316
> URL: https://issues.apache.org/jira/browse/HDFS-14316
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Íñigo Goiri
> Assignee: Íñigo Goiri
> Priority: Major
> Attachments: HDFS-14316-HDFS-13891.000.patch,
> HDFS-14316-HDFS-13891.001.patch
>
>
> Currently mount points with multiple destinations (e.g., HASH_ALL) fail
> writes when the destination subcluster is down. We need an option to allow
> writing in other subclusters when one is down.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
