[
https://issues.apache.org/jira/browse/HADOOP-13726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833516#comment-15833516
]
Manjunath Anand edited comment on HADOOP-13726 at 1/22/17 1:39 PM:
-------------------------------------------------------------------
I saw the new method computeIfAbsent introduced in ConcurrentHashMap as part of
Java 8 and felt appropriate for this scenario. Here's the implementation:-
{code}
private final ConcurrentMap<Key, FileSystem> map = new ConcurrentHashMap<>();
private FileSystem getInternal(URI uri, Configuration conf, Key key)
throws IOException{
FileSystem fs;
fs = map.get(key);
if (fs != null) {
return fs;
}
IOException[] ie = new IOException[1];
map.computeIfAbsent(key, k -> {
FileSystem fsc = null;
try {
fsc = createFileSystem(uri, conf);
fsc.key = key;
} catch (IOException e) {
ie[0] = e;
}
return fsc;
});
if(ie[0] != null) throw ie[0];
if (conf.getBoolean(
FS_AUTOMATIC_CLOSE_KEY, FS_AUTOMATIC_CLOSE_DEFAULT)) {
toAutoClose.add(key);
}
return map.get(key);
}
{code}
This avoids synchronization and wont stop all the threads , just those threads
referring to the same key and only during computeIfAbsent. Please let me know
your thoughts.
was (Author: manju_hadoop):
I saw the new method computeIfAbsent introduced in ConcurrentHashMap as part of
Java 8 and felt appropriate for this scenario. Here's the implementation:-
{code}
private final ConcurrentMap<Key, FileSystem> map = new ConcurrentHashMap<>();
private FileSystem getInternal(URI uri, Configuration conf, Key key)
throws IOException{
FileSystem fs;
fs = map.get(key);
if (fs != null) {
return fs;
}
IOException[] ie = new IOException[1];
map.computeIfAbsent(key, k -> {
FileSystem fsc = null;
try {
fsc = createFileSystem(uri, conf);
fsc.key = key;
} catch (IOException e) {
ie[0] = e;
}
return fsc;
});
if(ie[0] != null) throw ie[0];
if (conf.getBoolean(
FS_AUTOMATIC_CLOSE_KEY, FS_AUTOMATIC_CLOSE_DEFAULT)) {
toAutoClose.add(key);
}
return map.get(key);
}
{code}
This avoids synchronization and wont stop all the threads , just those threads
referring to the same key. Please let me know your thoughts.
> Enforce that FileSystem initializes only a single instance of the requested
> FileSystem.
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-13726
> URL: https://issues.apache.org/jira/browse/HADOOP-13726
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Reporter: Chris Nauroth
>
> The {{FileSystem}} cache is intended to guarantee reuse of instances by
> multiple call sites or multiple threads. The current implementation does
> provide this guarantee, but there is a brief race condition window during
> which multiple threads could perform redundant initialization. If the file
> system implementation has expensive initialization logic, then this is
> wasteful. This issue proposes to eliminate that race condition and guarantee
> initialization of only a single instance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
