Hi Sumit,
very interesting. I will certainly explore the possibility of using
dfs.inernal.nameservices (leaving federation for another day :)
You think a fallback to dfs.nameservices (the current behaivour), if
internal nameservices is not set, is needed?
Regards,
Henning
Am 01/04/16 um 20:30 schrieb Sumit Mohanty:
Hi Henning.
Its a very good problem that you brought up.
https://community.hortonworks.com/questions/8989/how-to-use-name-service-id-between-to-clusters.html
talks about "dfs.internal.nameservices" and if Ambari populates this and uses
it for all its internal scripts/alerts then that may be a solution.
This, as you notice also says "nameservices" (plural) which points to HDFS
federation. That is probably a problem for a different day.
In any case, in absence of HDFS federation, we can have Ambari refer to
"dfs.internal.nameservices" for its own use. The requirement will be to have NN
HA wizard and Blueprint deployment populate this property.
AMBARI-15615 modified the NN HA wizard to populate the property. But as far as
I can see no other changes have happened.
I was not aware of hdfs_site['dfs.namenode.shared.edits.dir'] as a possible solution but
talking to few folks from HDFS it seems that populating/using
"dfs.internal.nameservices" will be more desirable.
Do you want to explore the possibility of using "dfs.internal.nameservices"?
thanks
Sumit
________________________________________
From: Henning Kropp <[email protected]>
Sent: Friday, April 01, 2016 10:38 AM
To: [email protected]
Subject: Support for Multiple Nameservices
Hi,
we noticed that throughout Ambari hdfs_site['dfs.nameservices'] is
treated as a string denoting just one nameservice. It is possible to
configure multiple namservices for example to support seamless distcp
between two HA clusters. In general this makes it possible to use
multiple nameservices wiht hdfs.
This makes dfs.nameservices a comma separated string holding mulitple
namservices. When doing this with the current Ambari release this leads
to multiple problems. One is that Ambari marks the restart of HDFS as
failed, even though the restart was succesful. The reason for that is
that hdfs_resources.py is not working this a comma separated list of
nameservices AMBARI-15506.
We created an umbrella JIRA to track the other issues AMBARI-15507.
Problems with Blueprint install, because the DN's were registered with
the other cluster AMBARI-15509. Web alerting for NNs does not work
AMBARI-15508.
There might be other places where dfs.namservices is treated just a
string? How can web alerting be refactored to work with multiple
nameservices?
Also I would appreciate any feedback about the function to resolve the
current nameservice for the current cluster.
For AMBARI-15506 I defined the following nameservice resolution:
1. split names by ','
2. for all services check if string is also contained in
dfs.namenode.shared.edits.dir . Typically this are
jounalnode1,journalnode2,journalnode3:port/servicename. Here it would
probably be better to verify the name with fs.defaultFS, but this is
part of core-site not hdfs-site, which would add a separate dependency.
For namenode_ha_utils.py for me this seemed like an issue, because
refactoring all the python scripts to also include the core-site seemed
much more involved.
3. A default fallback the first string in the list is used as the
nameservice.
Thanks and regards,
Henning
