> On Aug. 2, 2016, 12:34 a.m., Jayush Luniya wrote: > > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java, > > line 2273 > > <https://reviews.apache.org/r/50594/diff/3/?file=1458830#file1458830line2273> > > > > We need to replace hdp_full_version with full_stack_version. The SPARK > > code has been cleaned up to remove hdp hardcodings in Ambari 2.4. If you > > see params.py the variable hdp_full_version is not even defined. So when > > you upgrade ambari-server from 2.1.0 or 2.2.0 with my hotfix the config > > will remain -Dhdp.version={{hdp_full_version}}. > > Saisai Shao wrote: > From my understanding, you mean that in the params.py we should also take > care of {{hdp_full_version}} if amabri is upgraded from lower version. Can > you please explain more about it.
If you look at https://github.com/apache/ambari/blob/branch-2.1.2/ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py and https://github.com/apache/ambari/blob/branch-2.1.2/ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-defaults.xml#L128-L133 you will see that we use to set these properties as -Dhdp.version={{hdp_full_version}}. So when you upgrade Ambari 2.1.X -> 2.4.0, these config properties will continue to be -Dhdp.version={{hdp_full_version}}. However params.py doesnt have this variable defined anymore. It has been renamed to full_stack_version. So when we do an ambari-server upgrade we should update these config properties to find and replace hdp_full_version with full_stack_version. - Jayush ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50594/#review144418 ----------------------------------------------------------- On Aug. 1, 2016, 1:22 a.m., Saisai Shao wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/50594/ > ----------------------------------------------------------- > > (Updated Aug. 1, 2016, 1:22 a.m.) > > > Review request for Ambari, Jayush Luniya and Sumit Mohanty. > > > Bugs: AMBARI-17954 > https://issues.apache.org/jira/browse/AMBARI-17954 > > > Repository: ambari > > > Description > ------- > > Fix Spark hdp.version issues in upgrading and fresh install > > > Diffs > ----- > > > ambari-common/src/main/python/resource_management/libraries/functions/constants.py > 42ecf6a > > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java > 5495655 > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml > 646239e > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py > 86aeb70 > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py > 2b5d36b > > ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml > e27c986 > > ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json > 6d26052 > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml > b295cc9 > > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml > 5b20a72 > > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml > 5d2cc54 > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml > c9f9823 > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml > d0332ed > > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml > b3d05d8 > > ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java > 34ca199 > ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py > b2ed5df > ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea > ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py > a1abdfa > > Diff: https://reviews.apache.org/r/50594/diff/ > > > Testing > ------- > > Manual test with different scenarios: > > 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0 > 2. Upgrade for 2.3.6 to 2.5.0. > 3. Downgrade from 2.5.0 to 2.3.6. > > > Thanks, > > Saisai Shao > >
