[
https://issues.apache.org/jira/browse/AMBARI-25672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
h.s updated AMBARI-25672:
-------------------------
Description:
step 1:
# delete a host from a kerberos cluster ,not a master host
# stop all the service on the host,
# use api delete host
step 2:
# prepare a host, install agent
# add a node to the cluster use api and install service
# regenerate_keytab
# ambari hang at preparing operations/hostname/preparing operations
it is because step1.3 cannot completely clear all this host kerberos idetities
in both database(mysql ) and kdc(kdc.admin)
* in mysql
there are 3 table kkp_mapping_service, kerberos_keytab_principal,
kerberos_keytab,kerberos_principal, host related kerberos identities in these
tables must be deleted completely,
* in kdc ,
{code:java}
kadmin.local
listprincs *hostnanme*{code}
will find related identies not deleted completely
some services kerberos identies in mysql and kdc can be deleted but some
sevices not,
if not all service kerberos identies deleted completely,if any service kerberos
identities left ,next time add a host to this cluster, will hang at preparing
operations
delete host api call chain in ambari-server
{code:java}
org.apache.ambari.server.api.services.HostService#deleteHost
org.apache.ambari.server.api.services.BaseService#handleRequest
org.apache.ambari.server.api.services.BaseRequest#process
org.apache.ambari.server.api.handlers.BaseManagementHandler#handleRequest
org.apache.ambari.server.api.handlers.DeleteHandler#persist
org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl#delete
org.apache.ambari.server.controller.internal.ClusterControllerImpl#deleteResources
org.apache.ambari.server.controller.internal.AbstractAuthorizedResourceProvider#deleteResources
org.apache.ambari.server.controller.internal.HostResourceProvider#deleteResourcesAuthorized
org.apache.ambari.server.controller.internal.HostResourceProvider#deleteHosts
A=org.apache.ambari.server.controller.internal.HostResourceProvider#processDeleteHostRequests
{code}
A=org.apache.ambari.server.controller.internal.HostResourceProvider#processDeleteHostRequests
has some main step
{code:java}
A1=org.apache.ambari.server.controller.AmbariManagementControllerImpl#deleteHostComponents
//this step will delete components and their kerbers identities
A2=org.apache.ambari.server.state.cluster.ClustersImpl#deleteHost //this step
will delete host from mysql{code}
A1=org.apache.ambari.server.controller.AmbariManagementControllerImpl#deleteHostComponents
call chain
{code:java}
org.apache.ambari.server.state.ServiceComponentImpl#deleteServiceComponentHosts
A1-1=org.apache.ambari.server.state.svccomphost.ServiceComponentHostImpl#delete
{code}
A1-1=org.apache.ambari.server.state.svccomphost.ServiceComponentHostImpl#delete
call chain
{code:java}
org.apache.ambari.server.state.cluster.ClusterImpl#removeServiceComponentHost
A1-1-1=eventPublisher.publish(event); //publish
ServiceComponentUninstalledEvent,org.apache.ambari.server.controller.utilities.KerberosIdentityCleaner#componentRemoved
will deal this event,and delete components kerberos identites,these event once
publish,next line code will execute,not wait these event finish,
{code}
A1-1-1=org.apache.ambari.server.controller.utilities.KerberosIdentityCleaner#componentRemoved
call chain
{code:java}
org.apache.ambari.server.controller.utilities.RemovableIdentities#remove
org.apache.ambari.server.controller.KerberosHelperImpl#deleteIdentities(org.apache.ambari.server.state.Cluster,
java.util.List<org.apache.ambari.server.serveraction.kerberos.Component>,
java.util.Set<java.lang.String>)
org.apache.ambari.server.controller.KerberosHelperImpl#validateKDCCredentials(org.apache.ambari.server.controller.KerberosDetails,
org.apache.ambari.server.state.Cluster) //check KDC administrator credentials
org.apache.ambari.server.controller.DeleteIdentityHandler#addDeleteIdentityStages
//add stage in prepare delete identies
{code}
A2=org.apache.ambari.server.state.cluster.ClustersImpl#deleteHost call chain
{code:java}
org.apache.ambari.server.state.cluster.ClustersImpl#deleteHostEntityRelationships
org.apache.ambari.server.state.cluster.ClustersImpl#unmapHostFromClusters
org.apache.ambari.server.state.cluster.ClustersImpl#unmapHostClusterEntities
//delete host cluster mapping
org.apache.ambari.server.orm.dao.KerberosKeytabPrincipalDAO#removeByHost
hostDAO.remove(entity); // Note, if the host is still heartbeating, then new
records will be re-inserted into the hosts and hoststate tables
{code}
there are 3 reason why some service kerberos identies can not be deleted
* one, lost kdc.admin.credential , maybe caused by ambari-server restart
* second,
was:
step 1:
# delete a host from a kerberos cluster ,not a master host
# stop all the service on the host,
# use api delete host
step 2:
# prepare a host, install agent
# add a node to the cluster use api and install service
# regenerate_keytab
# ambari hang at preparing operations/hostname/preparing operations
it is because step1.3 cannot completely clear all this host kerberos idetities
in both database(mysql ) and kdc(kdc.admin)
* in mysql
there are 3 table kkp_mapping_service, kerberos_keytab_principal,
kerberos_keytab,kerberos_principal, host related kerberos identities in these
tables must be deleted completely,
* in kdc ,
{code:java}
kadmin.local
listprincs *hostnanme*{code}
will find related identies not deleted completely
some services kerberos identies in mysql and kdc can be deleted but some
sevices not,
if not all service kerberos identies deleted completely,if any service kerberos
identities left ,next time add a host to this cluster, will hang at preparing
operations
delete host api call chain in ambari-server
{code:java}
org.apache.ambari.server.api.services.HostService#deleteHost
org.apache.ambari.server.api.services.BaseService#handleRequest
org.apache.ambari.server.api.services.BaseRequest#process
org.apache.ambari.server.api.handlers.BaseManagementHandler#handleRequest
org.apache.ambari.server.api.handlers.DeleteHandler#persist
org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl#delete
org.apache.ambari.server.controller.internal.ClusterControllerImpl#deleteResources
org.apache.ambari.server.controller.internal.AbstractAuthorizedResourceProvider#deleteResources
org.apache.ambari.server.controller.internal.HostResourceProvider#deleteResourcesAuthorized
org.apache.ambari.server.controller.internal.HostResourceProvider#deleteHosts
A=org.apache.ambari.server.controller.internal.HostResourceProvider#processDeleteHostRequests
{code}
A=org.apache.ambari.server.controller.internal.HostResourceProvider#processDeleteHostRequests
has some main step
{code:java}
A1=org.apache.ambari.server.controller.AmbariManagementControllerImpl#deleteHostComponents
//this step will delete components and their kerbers identities
A2=org.apache.ambari.server.state.cluster.ClustersImpl#deleteHost //this step
will delete host from mysql{code}
A1=org.apache.ambari.server.controller.AmbariManagementControllerImpl#deleteHostComponents
call chain
{code:java}
org.apache.ambari.server.state.ServiceComponentImpl#deleteServiceComponentHosts
A1-1=org.apache.ambari.server.state.svccomphost.ServiceComponentHostImpl#delete
{code}
A1-1=org.apache.ambari.server.state.svccomphost.ServiceComponentHostImpl#delete
call chain
{code:java}
org.apache.ambari.server.state.cluster.ClusterImpl#removeServiceComponentHost
A1-1-1=eventPublisher.publish(event); //publish
ServiceComponentUninstalledEvent,org.apache.ambari.server.controller.utilities.KerberosIdentityCleaner#componentRemoved
will deal this event,and delete components kerberos identites,these event once
publish,next line code will execute,not wait these event finish,
{code}
A1-1-1=org.apache.ambari.server.controller.utilities.KerberosIdentityCleaner#componentRemoved
call chain
{code:java}
org.apache.ambari.server.controller.utilities.RemovableIdentities#remove
org.apache.ambari.server.controller.KerberosHelperImpl#deleteIdentities(org.apache.ambari.server.state.Cluster,
java.util.List<org.apache.ambari.server.serveraction.kerberos.Component>,
java.util.Set<java.lang.String>)
org.apache.ambari.server.controller.KerberosHelperImpl#validateKDCCredentials(org.apache.ambari.server.controller.KerberosDetails,
org.apache.ambari.server.state.Cluster) //check KDC administrator credentials
org.apache.ambari.server.controller.DeleteIdentityHandler#addDeleteIdentityStages
//add stage in prepare delete identies
{code}
A2=org.apache.ambari.server.state.cluster.ClustersImpl#deleteHost call chain
{code:java}
org.apache.ambari.server.state.cluster.ClustersImpl#deleteHostEntityRelationships
org.apache.ambari.server.state.cluster.ClustersImpl#unmapHostFromClusters
org.apache.ambari.server.state.cluster.ClustersImpl#unmapHostClusterEntities
//delete host cluster mapping
org.apache.ambari.server.orm.dao.KerberosKeytabPrincipalDAO#removeByHost
hostDAO.remove(entity); // Note, if the host is still heartbeating, then new
records will be re-inserted into the hosts and hoststate tables
{code}
there are 3 reason why some service kerberos identies can not be deleted
one, lost kdc.admin.credential ,
> delete host from a kerberos cluster not completely clear all identies in
> database and kdc
> -----------------------------------------------------------------------------------------
>
> Key: AMBARI-25672
> URL: https://issues.apache.org/jira/browse/AMBARI-25672
> Project: Ambari
> Issue Type: Bug
> Components: ambari-server
> Affects Versions: 2.7.3
> Reporter: h.s
> Priority: Major
>
> step 1:
> # delete a host from a kerberos cluster ,not a master host
> # stop all the service on the host,
> # use api delete host
> step 2:
> # prepare a host, install agent
> # add a node to the cluster use api and install service
> # regenerate_keytab
> # ambari hang at preparing operations/hostname/preparing operations
> it is because step1.3 cannot completely clear all this host kerberos
> idetities in both database(mysql ) and kdc(kdc.admin)
> * in mysql
> there are 3 table kkp_mapping_service, kerberos_keytab_principal,
> kerberos_keytab,kerberos_principal, host related kerberos identities in these
> tables must be deleted completely,
> * in kdc ,
> {code:java}
> kadmin.local
> listprincs *hostnanme*{code}
> will find related identies not deleted completely
> some services kerberos identies in mysql and kdc can be deleted but some
> sevices not,
> if not all service kerberos identies deleted completely,if any service
> kerberos identities left ,next time add a host to this cluster, will hang at
> preparing operations
>
> delete host api call chain in ambari-server
> {code:java}
> org.apache.ambari.server.api.services.HostService#deleteHost
> org.apache.ambari.server.api.services.BaseService#handleRequest
> org.apache.ambari.server.api.services.BaseRequest#process
> org.apache.ambari.server.api.handlers.BaseManagementHandler#handleRequest
> org.apache.ambari.server.api.handlers.DeleteHandler#persist
> org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl#delete
> org.apache.ambari.server.controller.internal.ClusterControllerImpl#deleteResources
> org.apache.ambari.server.controller.internal.AbstractAuthorizedResourceProvider#deleteResources
> org.apache.ambari.server.controller.internal.HostResourceProvider#deleteResourcesAuthorized
> org.apache.ambari.server.controller.internal.HostResourceProvider#deleteHosts
> A=org.apache.ambari.server.controller.internal.HostResourceProvider#processDeleteHostRequests
> {code}
>
> A=org.apache.ambari.server.controller.internal.HostResourceProvider#processDeleteHostRequests
> has some main step
> {code:java}
> A1=org.apache.ambari.server.controller.AmbariManagementControllerImpl#deleteHostComponents
> //this step will delete components and their kerbers identities
> A2=org.apache.ambari.server.state.cluster.ClustersImpl#deleteHost //this step
> will delete host from mysql{code}
>
>
> A1=org.apache.ambari.server.controller.AmbariManagementControllerImpl#deleteHostComponents
> call chain
> {code:java}
> org.apache.ambari.server.state.ServiceComponentImpl#deleteServiceComponentHosts
> A1-1=org.apache.ambari.server.state.svccomphost.ServiceComponentHostImpl#delete
> {code}
> A1-1=org.apache.ambari.server.state.svccomphost.ServiceComponentHostImpl#delete
> call chain
> {code:java}
> org.apache.ambari.server.state.cluster.ClusterImpl#removeServiceComponentHost
> A1-1-1=eventPublisher.publish(event); //publish
> ServiceComponentUninstalledEvent,org.apache.ambari.server.controller.utilities.KerberosIdentityCleaner#componentRemoved
> will deal this event,and delete components kerberos identites,these event
> once publish,next line code will execute,not wait these event finish,
> {code}
> A1-1-1=org.apache.ambari.server.controller.utilities.KerberosIdentityCleaner#componentRemoved
> call chain
> {code:java}
> org.apache.ambari.server.controller.utilities.RemovableIdentities#remove
> org.apache.ambari.server.controller.KerberosHelperImpl#deleteIdentities(org.apache.ambari.server.state.Cluster,
> java.util.List<org.apache.ambari.server.serveraction.kerberos.Component>,
> java.util.Set<java.lang.String>)
> org.apache.ambari.server.controller.KerberosHelperImpl#validateKDCCredentials(org.apache.ambari.server.controller.KerberosDetails,
> org.apache.ambari.server.state.Cluster) //check KDC administrator credentials
> org.apache.ambari.server.controller.DeleteIdentityHandler#addDeleteIdentityStages
> //add stage in prepare delete identies
> {code}
>
> A2=org.apache.ambari.server.state.cluster.ClustersImpl#deleteHost call chain
> {code:java}
> org.apache.ambari.server.state.cluster.ClustersImpl#deleteHostEntityRelationships
> org.apache.ambari.server.state.cluster.ClustersImpl#unmapHostFromClusters
> org.apache.ambari.server.state.cluster.ClustersImpl#unmapHostClusterEntities
> //delete host cluster mapping
> org.apache.ambari.server.orm.dao.KerberosKeytabPrincipalDAO#removeByHost
> hostDAO.remove(entity); // Note, if the host is still heartbeating, then new
> records will be re-inserted into the hosts and hoststate tables
> {code}
> there are 3 reason why some service kerberos identies can not be deleted
> * one, lost kdc.admin.credential , maybe caused by ambari-server restart
> * second,
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
