[jira] [Commented] (HADOOP-8292) TableMapping does not refresh when topology is updated
[
https://issues.apache.org/jira/browse/HADOOP-8292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461462#comment-13461462
]
Harsh J commented on HADOOP-8292:
-
{code}
+ protected void clearCache() {
+cache.clear();
+ }
{code}
Is there a way to also invoke a cache clearance dynamically (say via a
dfsadmin command)? Should we have one?
Patch appears alright to me. Mind checking it once again and rebasing?
Todd's question:
bq. How does this interact with the HDFS topology code which needs to check
when a cluster changes from single-rack to multi-rack? When a node's topology
changes, don't we need to re-check replication policies for all the blocks,
etc? Maybe this isn't a new issue, but it's certainly strange.
Currently the fsck begins warning as soon as the mapping changes in its
perspective (when it checks for it). Things such as balancer, etc. go crazy
after that but doesn't cause a downtime as such. Perhaps this needs to be
discussed in a separate JIRA?
TableMapping does not refresh when topology is updated
--
Key: HADOOP-8292
URL: https://issues.apache.org/jira/browse/HADOOP-8292
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Philip Zeyliger
Assignee: Alejandro Abdelnur
Attachments: HADOOP-8292.patch, HADOOP-8292.patch
HADOOP-7030 introduced TableMapping, an implementation of DNSToSwitchMapping
which uses a file to map from IPs/hosts to their racks. It's intended to
replace ScriptBasedMapping for cases where the latter was just a complicated
way of looking up the rack in a file.
Though there was discussion of it on the JIRA, the TableMapping
implementation is not 'refreshable'. i.e., if you want to add a host to your
cluster, and that host wasn't in the topology file to begin with, it will
never be added.
TableMapping should refresh, either based on a command that can be executed,
or, perhaps, if the file on disk changes.
I'll also point out that TableMapping extends CachedDNSToSwitchMapping, but,
since it does no refreshing, I don't see what the caching gets you: I think
the cache ends up being a second copy of the underlying map, always.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8292) TableMapping does not refresh when topology is updated
[ https://issues.apache.org/jira/browse/HADOOP-8292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13271527#comment-13271527 ] Alejandro Abdelnur commented on HADOOP-8292: Forgot to mention, I'm using a daemon thread with a while(true) loop because the DNSToSwitchMapping interface does not have lifecycle methods (init/destroy) I could use to shutdown the thread. This could be done, but I'd argue as part of another JIRA TableMapping does not refresh when topology is updated -- Key: HADOOP-8292 URL: https://issues.apache.org/jira/browse/HADOOP-8292 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.0 Reporter: Philip Zeyliger Assignee: Alejandro Abdelnur Attachments: HADOOP-8292.patch, HADOOP-8292.patch HADOOP-7030 introduced TableMapping, an implementation of DNSToSwitchMapping which uses a file to map from IPs/hosts to their racks. It's intended to replace ScriptBasedMapping for cases where the latter was just a complicated way of looking up the rack in a file. Though there was discussion of it on the JIRA, the TableMapping implementation is not 'refreshable'. i.e., if you want to add a host to your cluster, and that host wasn't in the topology file to begin with, it will never be added. TableMapping should refresh, either based on a command that can be executed, or, perhaps, if the file on disk changes. I'll also point out that TableMapping extends CachedDNSToSwitchMapping, but, since it does no refreshing, I don't see what the caching gets you: I think the cache ends up being a second copy of the underlying map, always. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8292) TableMapping does not refresh when topology is updated
[ https://issues.apache.org/jira/browse/HADOOP-8292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13271579#comment-13271579 ] Hadoop QA commented on HADOOP-8292: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526177/HADOOP-8292.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.fs.viewfs.TestViewFsTrash +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/967//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/967//console This message is automatically generated. TableMapping does not refresh when topology is updated -- Key: HADOOP-8292 URL: https://issues.apache.org/jira/browse/HADOOP-8292 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.0 Reporter: Philip Zeyliger Assignee: Alejandro Abdelnur Attachments: HADOOP-8292.patch, HADOOP-8292.patch HADOOP-7030 introduced TableMapping, an implementation of DNSToSwitchMapping which uses a file to map from IPs/hosts to their racks. It's intended to replace ScriptBasedMapping for cases where the latter was just a complicated way of looking up the rack in a file. Though there was discussion of it on the JIRA, the TableMapping implementation is not 'refreshable'. i.e., if you want to add a host to your cluster, and that host wasn't in the topology file to begin with, it will never be added. TableMapping should refresh, either based on a command that can be executed, or, perhaps, if the file on disk changes. I'll also point out that TableMapping extends CachedDNSToSwitchMapping, but, since it does no refreshing, I don't see what the caching gets you: I think the cache ends up being a second copy of the underlying map, always. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8292) TableMapping does not refresh when topology is updated
[ https://issues.apache.org/jira/browse/HADOOP-8292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13270998#comment-13270998 ] Hadoop QA commented on HADOOP-8292: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526085/HADOOP-8292.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.fs.viewfs.TestViewFsTrash +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/963//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/963//console This message is automatically generated. TableMapping does not refresh when topology is updated -- Key: HADOOP-8292 URL: https://issues.apache.org/jira/browse/HADOOP-8292 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.0 Reporter: Philip Zeyliger Assignee: Alejandro Abdelnur Attachments: HADOOP-8292.patch HADOOP-7030 introduced TableMapping, an implementation of DNSToSwitchMapping which uses a file to map from IPs/hosts to their racks. It's intended to replace ScriptBasedMapping for cases where the latter was just a complicated way of looking up the rack in a file. Though there was discussion of it on the JIRA, the TableMapping implementation is not 'refreshable'. i.e., if you want to add a host to your cluster, and that host wasn't in the topology file to begin with, it will never be added. TableMapping should refresh, either based on a command that can be executed, or, perhaps, if the file on disk changes. I'll also point out that TableMapping extends CachedDNSToSwitchMapping, but, since it does no refreshing, I don't see what the caching gets you: I think the cache ends up being a second copy of the underlying map, always. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8292) TableMapping does not refresh when topology is updated
[ https://issues.apache.org/jira/browse/HADOOP-8292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13271000#comment-13271000 ] Todd Lipcon commented on HADOOP-8292: - Few quick notes (didn't look in serious detail yet): - is it OK to be doing file system access while holding this lock, and in the hot path of resolve? I worry that this might slow down client requests, for example. - I think we should avoid reading the file if the modification time is within the last couple of seconds -- with some editors and config management systems, updating a file might temporarily leave it in an empty state before re-filling it again with the new data. Well behaved systems won't do that, but I think it's better for us to be resilient to it than for us to end up loading an empty topology mapping. - clearCache() is called from resolve() without the lock held. That might cause multiple threads to call clear() on a map at once, which might result in an exception or something. How does this interact with the HDFS topology code which needs to check when a cluster changes from single-rack to multi-rack? When a node's topology changes, don't we need to re-check replication policies for all the blocks, etc? Maybe this isn't a new issue, but it's certainly strange. TableMapping does not refresh when topology is updated -- Key: HADOOP-8292 URL: https://issues.apache.org/jira/browse/HADOOP-8292 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.0 Reporter: Philip Zeyliger Assignee: Alejandro Abdelnur Attachments: HADOOP-8292.patch HADOOP-7030 introduced TableMapping, an implementation of DNSToSwitchMapping which uses a file to map from IPs/hosts to their racks. It's intended to replace ScriptBasedMapping for cases where the latter was just a complicated way of looking up the rack in a file. Though there was discussion of it on the JIRA, the TableMapping implementation is not 'refreshable'. i.e., if you want to add a host to your cluster, and that host wasn't in the topology file to begin with, it will never be added. TableMapping should refresh, either based on a command that can be executed, or, perhaps, if the file on disk changes. I'll also point out that TableMapping extends CachedDNSToSwitchMapping, but, since it does no refreshing, I don't see what the caching gets you: I think the cache ends up being a second copy of the underlying map, always. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
