----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63937/#review191596 -----------------------------------------------------------
Ship it! Ship It! - Robert Levas On Nov. 21, 2017, 2:28 a.m., Swapan Shridhar wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/63937/ > ----------------------------------------------------------- > > (Updated Nov. 21, 2017, 2:28 a.m.) > > > Review request for Ambari, Jayush Luniya, Madhuvanthi Radhakrishnan, and > Robert Levas. > > > Bugs: AMBARI-22472 > https://issues.apache.org/jira/browse/AMBARI-22472 > > > Repository: ambari > > > Description > ------- > > **Background:** > YARN NodeManager currently has: > > - 2 identities in 2.5 stack, namely : > **'/HIVE/HIVE_SERVER/hive_server_hive'** and **'llap_zk_hive'**. > -- **/HIVE/HIVE_SERVER/hive_server_hive** is a reference from HIVE_SERVER, > whereas > -- **llap_zk_hive** creates same principal as above in a separate keytab file. > > - and 3 identities in 2.6 stack: > *'/HIVE/HIVE_SERVER/hive_server_hive'* and *'llap_zk_hive'*. > -- **/HIVE/HIVE_SERVER/hive_server_hive** is a reference from HIVE_SERVER, > whereas > -- **llap_zk_hive** and **llap_task_hive** creates same principal as above in > a separate keytab file. > > **Issue:** Recreating same identities in different files creates issues while > AMbari upgrade from 2.5 to 2.6, as the *llap_zk_hive* are not > refreshed/updated after the upgrade. Thus, HSI fails to come up. > > **Fix:** > > **For HDP 2.5:** Make **llap_zk_hive** point as a reference pointing to > /HIVE/HIVE_SERVER/hive_server_hive, so that we have one identity getting > created only at one place and one keytab file. > > **For HDP 2.6:** Make **llap_zk_hive** and **llap_task_hive** point as a > reference pointing to /HIVE/HIVE_SERVER/hive_server_hive, so that we have one > identity getting created only at one place and one keytab file. > > > Diffs > ----- > > > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java > 96ce807 > ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/kerberos.json > af6bda6 > ambari-server/src/main/resources/stacks/HDP/2.6/services/YARN/kerberos.json > e0417bf > > ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog260Test.java > be04cd5 > > ambari-server/src/test/resources/kerberos/test_kerberos_descriptor_ranger_kms.json > e17e121 > > > Diff: https://reviews.apache.org/r/63937/diff/2/ > > > Testing > ------- > > **TESTING:** > > |||||||||||||||||||||||||| Ambari 2.5, HDP 2.5 before upgrade: > |||||||||||||||||||||||||| > > > {code:title=From /etc/hive2/cong/conf.server/hive-site.xml} > <property> > <name>hive.llap.daemon.keytab.file</name> > <value>/etc/security/keytabs/hive.service.keytab</value> > </property> > > <property> > <name>hive.llap.daemon.service.principal</name> > <value>hive/[email protected]</value> > </property> > > <property> > <name>hive.llap.zk.sm.keytab.file</name> > <value>/etc/security/keytabs/hive.llap.zk.sm.keytab</value> > </property> > > <property> > <name>hive.llap.zk.sm.principal</name> > <value>hive/[email protected]</value> > </property> > {code} > > > |||||||||||||||||||||||||| Upgrade to Ambari-2.6 > |||||||||||||||||||||||||| > > > **Logs: Ambari Server Upgrade** > > [root@swap-qqq-1 ~]# ambari-server upgrade > Using python /usr/bin/python > Upgrading ambari-server > INFO: Upgrade Ambari Server > INFO: Updating Ambari Server properties in ambari.properties ... > INFO: Updating Ambari Server properties in ambari-env.sh ... > WARNING: Original file ambari-env.sh kept > INFO: Fixing database objects owner > Ambari Server configured for Embedded Postgres. Confirm you have made a > backup of the Ambari Server database [y/n] (y)? y > INFO: Upgrading database schema > INFO: Return code from schema upgrade command, retcode = 0 > INFO: Schema upgrade completed > Adjusting ambari-server permissions and ownership... > Ambari Server 'upgrade' completed successfully. > [root@swap-qqq-1 ~]# > [root@swap-qqq-1 ~]# > [root@swap-qqq-1 ~]# > [root@swap-qqq-1 ~]# > [root@swap-qqq-1 ~]# ambari-server --version > 2.6.0.0-267 > [root@swap-qqq-1 ~]# > > > **Logs : Updating Kerberos descriptors** > > 21 Nov 2017 01:01:20,438 INFO [main] UpgradeCatalog260:675 - Updating YARN's > HSI Kerberos Descriptor .... > 21 Nov 2017 01:01:20,438 INFO [main] UpgradeCatalog260:687 - Retrieved > HIVE->HIVE_SERVER kerberos descriptor. Name = hive_server_hive > 21 Nov 2017 01:01:20,438 INFO [main] UpgradeCatalog260:707 - Retrieved > YARN->NODEMANAGER kerberos descriptor to be updated. Name = llap_zk_hive > 21 Nov 2017 01:01:20,439 INFO [main] UpgradeCatalog260:712 - Updated > 'llap_zk_hive' identity descriptor reference = > '/HIVE/HIVE_SERVER/hive_server_hive' > 21 Nov 2017 01:01:20,439 INFO [main] UpgradeCatalog260:715 - Updated > 'llap_zk_hive' principal descriptor value = 'null' > 21 Nov 2017 01:01:20,439 INFO [main] UpgradeCatalog260:720 - Updated > 'llap_zk_hive' keytab descriptor file = 'null' > 21 Nov 2017 01:01:20,439 INFO [main] UpgradeCatalog260:723 - Updated > 'llap_zk_hive' keytab descriptor owner name = 'null' > 21 Nov 2017 01:01:20,439 INFO [main] UpgradeCatalog260:725 - Updated > 'llap_zk_hive' keytab descriptor owner access = 'null' > 21 Nov 2017 01:01:20,439 INFO [main] UpgradeCatalog260:727 - Updated > 'llap_zk_hive' keytab descriptor group name = 'null' > 21 Nov 2017 01:01:20,439 INFO [main] UpgradeCatalog260:729 - Updated > 'llap_zk_hive' keytab descriptor group access = 'null' > 21 Nov 2017 01:01:20,439 INFO [main] UpgradeCatalog260:733 - Updated > 'yarnKerberosDescUpdatedList' = [hive.llap.zk.sm.keytab.file, > hive.llap.task.keytab.file] > > **Logs : Updated HSI config 'hive.llap.zk.sm.keytab.file'** > > 18 Nov 2017 07:25:54,073 INFO [main] UpgradeCatalog260:767 - Updated HSI > config 'hive.llap.zk.sm.keytab.file' = > /etc/security/keytabs/hive.service.keytab > > > **From UI**: > > Changed hive.llap.zk.sm.keytab.file : > https://issues.apache.org/jira/secure/attachment/12898329/Screen%20Shot%202017-11-17%20at%2011.44.41%20PM.png > > HSI up : > https://issues.apache.org/jira/secure/attachment/12898328/Screen%20Shot%202017-11-17%20at%2011.44.55%20PM.png > > > ------------------------------------ > > > UT test runs for Ambari 2.6 and HDP 2.6 (which includes **llap_zk_hive** and > **llap_task_hive**): > > > **UpgradeCatalog260Test::testUpdateKerberosDescriptorArtifact()** > > 2017-11-20 13:09:45,366 INFO [main] upgrade.AbstractUpgradeCatalog > (AbstractUpgradeCatalog.java:updateConfigurationPropertiesForCluster(573)) - > Applying configuration with tag 'version1511212185365' and configType > 'ranger-kms-audit' to cluster 'cl1' > 2017-11-20 13:09:45,367 INFO [main] upgrade.AbstractUpgradeCatalog > (AbstractUpgradeCatalog.java:updateConfigurationPropertiesForCluster(595)) - > cluster 'cl1' changed by: 'ambari-upgrade'; type='ranger-kms-audit' > tag='version2' from='version1' > 2017-11-20 13:09:45,367 INFO [main] upgrade.UpgradeCatalog260 > (UpgradeCatalog260.java:fixYarnHsiKerberosDescriptorAndSiteConfig(675)) - > Updating YARN's HSI Kerberos Descriptor .... > 2017-11-20 13:09:45,368 INFO [main] upgrade.UpgradeCatalog260 > (UpgradeCatalog260.java:fixYarnHsiKerberosDescriptorAndSiteConfig(687)) - > Retrieved HIVE->HIVE_SERVER kerberos descriptor. Name = hive_server_hive > 2017-11-20 13:09:45,368 INFO [main] upgrade.UpgradeCatalog260 > (UpgradeCatalog260.java:fixYarnHsiKerberosDescriptorAndSiteConfig(707)) - > Retrieved YARN->NODEMANAGER kerberos descriptor to be updated. Name = > llap_zk_hive > 2017-11-20 13:09:45,368 INFO [main] upgrade.UpgradeCatalog260 > (UpgradeCatalog260.java:fixYarnHsiKerberosDescriptorAndSiteConfig(712)) - > Updated 'llap_zk_hive' identity descriptor reference = > '/HIVE/HIVE_SERVER/hive_server_hive' > 2017-11-20 13:09:45,368 INFO [main] upgrade.UpgradeCatalog260 > (UpgradeCatalog260.java:fixYarnHsiKerberosDescriptorAndSiteConfig(715)) - > Updated 'llap_zk_hive' principal descriptor value = 'null' > 2017-11-20 13:09:45,369 INFO [main] upgrade.UpgradeCatalog260 > (UpgradeCatalog260.java:fixYarnHsiKerberosDescriptorAndSiteConfig(720)) - > Updated 'llap_zk_hive' keytab descriptor file = 'null' > 2017-11-20 13:09:45,369 INFO [main] upgrade.UpgradeCatalog260 > (UpgradeCatalog260.java:fixYarnHsiKerberosDescriptorAndSiteConfig(723)) - > Updated 'llap_zk_hive' keytab descriptor owner name = 'null' > 2017-11-20 13:09:45,369 INFO [main] upgrade.UpgradeCatalog260 > (UpgradeCatalog260.java:fixYarnHsiKerberosDescriptorAndSiteConfig(725)) - > Updated 'llap_zk_hive' keytab descriptor owner access = 'null' > 2017-11-20 13:09:45,369 INFO [main] upgrade.UpgradeCatalog260 > (UpgradeCatalog260.java:fixYarnHsiKerberosDescriptorAndSiteConfig(727)) - > Updated 'llap_zk_hive' keytab descriptor group name = 'null' > 2017-11-20 13:09:45,369 INFO [main] upgrade.UpgradeCatalog260 > (UpgradeCatalog260.java:fixYarnHsiKerberosDescriptorAndSiteConfig(729)) - > Updated 'llap_zk_hive' keytab descriptor group access = 'null' > 2017-11-20 13:09:45,369 INFO [main] upgrade.UpgradeCatalog260 > (UpgradeCatalog260.java:fixYarnHsiKerberosDescriptorAndSiteConfig(733)) - > Updated 'yarnKerberosDescUpdatedList' = [llap_zk_hive] > 2017-11-20 13:09:45,369 INFO [main] upgrade.UpgradeCatalog260 > (UpgradeCatalog260.java:fixYarnHsiKerberosDescriptorAndSiteConfig(707)) - > Retrieved YARN->NODEMANAGER kerberos descriptor to be updated. Name = > llap_task_hive > 2017-11-20 13:09:45,369 INFO [main] upgrade.UpgradeCatalog260 > (UpgradeCatalog260.java:fixYarnHsiKerberosDescriptorAndSiteConfig(712)) - > Updated 'llap_zk_hive' identity descriptor reference = > '/HIVE/HIVE_SERVER/hive_server_hive' > 2017-11-20 13:09:45,370 INFO [main] upgrade.UpgradeCatalog260 > (UpgradeCatalog260.java:fixYarnHsiKerberosDescriptorAndSiteConfig(715)) - > Updated 'llap_zk_hive' principal descriptor value = 'null' > 2017-11-20 13:09:45,370 INFO [main] upgrade.UpgradeCatalog260 > (UpgradeCatalog260.java:fixYarnHsiKerberosDescriptorAndSiteConfig(720)) - > Updated 'llap_zk_hive' keytab descriptor file = 'null' > 2017-11-20 13:09:45,370 INFO [main] upgrade.UpgradeCatalog260 > (UpgradeCatalog260.java:fixYarnHsiKerberosDescriptorAndSiteConfig(723)) - > Updated 'llap_zk_hive' keytab descriptor owner name = 'null' > 2017-11-20 13:09:45,370 INFO [main] upgrade.UpgradeCatalog260 > (UpgradeCatalog260.java:fixYarnHsiKerberosDescriptorAndSiteConfig(725)) - > Updated 'llap_zk_hive' keytab descriptor owner access = 'null' > 2017-11-20 13:09:45,370 INFO [main] upgrade.UpgradeCatalog260 > (UpgradeCatalog260.java:fixYarnHsiKerberosDescriptorAndSiteConfig(727)) - > Updated 'llap_zk_hive' keytab descriptor group name = 'null' > 2017-11-20 13:09:45,370 INFO [main] upgrade.UpgradeCatalog260 > (UpgradeCatalog260.java:fixYarnHsiKerberosDescriptorAndSiteConfig(729)) - > Updated 'llap_zk_hive' keytab descriptor group access = 'null' > 2017-11-20 13:09:45,370 INFO [main] upgrade.UpgradeCatalog260 > (UpgradeCatalog260.java:fixYarnHsiKerberosDescriptorAndSiteConfig(733)) - > [hive.llap.zk.sm.keytab.file, hive.llap.task.keytab.file] > > > **UpgradeCatalog260Test::testUpdateHiveConfigs()** > > (AbstractUpgradeCatalog.java:updateConfigurationPropertiesForCluster(573)) - > Applying configuration with tag 'version1511212185535' and configType > 'hive-interactive-site' to cluster 'null' > 2017-11-20 13:09:45,536 INFO [main] upgrade.UpgradeCatalog260 > (UpgradeCatalog260.java:updateHiveConfigs(778)) - Updated HSI config(s) : > [hive.llap.task.keytab.file, hive.llap.zk.sm.keytab.file] with values = > [/etc/security/keytabs/hive.service.keytab, > /etc/security/keytabs/hive.service.keytab] > > > Thanks, > > Swapan Shridhar > >
