----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48805/#review138033 -----------------------------------------------------------
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py (line 37) <https://reviews.apache.org/r/48805/#comment203245> Spark now needs to know where to write configs to. ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/spark_client.py (line 82) <https://reviews.apache.org/r/48805/#comment203246> Call configure on the newer version even though we haven't called hdp-select. ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py (line 44) <https://reviews.apache.org/r/48805/#comment203247> Tez now needs to know where to write configs to. ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py (line 90) <https://reviews.apache.org/r/48805/#comment203248> Call configure on the newer version even though we haven't called hdp-select. ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/mapreduce2_client.py (line 79) <https://reviews.apache.org/r/48805/#comment203249> Call configure on the newer version even though we haven't called hdp-select. ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py (line 98) <https://reviews.apache.org/r/48805/#comment203250> MR2 Client now needs to know where to write configs to. ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml (line 139) <https://reviews.apache.org/r/48805/#comment203251> We have to update these properties up front. ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml (line 153) <https://reviews.apache.org/r/48805/#comment203252> This gets called on every Tez Client host if ATS exists on the cluster. - Alejandro Fernandez On June 16, 2016, 5:47 p.m., Alejandro Fernandez wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48805/ > ----------------------------------------------------------- > > (Updated June 16, 2016, 5:47 p.m.) > > > Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, > Jonathan Hurley, Nate Cole, and Tim Thorpe. > > > Bugs: AMBARI-17280 > https://issues.apache.org/jira/browse/AMBARI-17280 > > > Repository: ambari > > > Description > ------- > > During RU, HiveServer2 is restarted but the newer tez configs have not yet > been saved, which is incorrect because Hive has a dependency on Tez. > This is important when configs change during a major stack upgrade, e.g., HDP > 2.4 -> 2.5. What happens today is, > > * Install packages generates /etc/tez/2.5.0.0-1/0 and copies the configs from > /etc/tez/2.4.0.0-1/0/ to the new folder > * If configs change during RU, then Hive is restarted and the classpath means > that it will pick up the older tez configs from the new /etc/tez/2.5.0.0-1/0 > folder > > > This problem exists for all of these components: > > * HiveServer: depends on Tez and MapReduce clients > * ATS: depends on Tez and Spark clients > * Oozie: depends on Tez, Spark, and MapReduce clients > > This problem only exists when configs change (so crossing major stack > version) and during RU (because it is allowed to change configs during the > middle of restarting services). > > > Diffs > ----- > > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java > fb3ae69 > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java > 80bb26c > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py > 63c72f7 > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/spark_client.py > ef41453 > > ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py > 44239c7 > > ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez.py > 67466e3 > > ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py > c79d63b > > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/mapreduce2_client.py > db22004 > > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py > 90f885a > > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py > d1ec15b > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml > 4187d64 > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml > 3461ad4 > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml > 426b452 > ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py e53eb4b > > Diff: https://reviews.apache.org/r/48805/diff/ > > > Testing > ------- > > Verified during RU from HDP 2.4 to 2.5 with ATS, Hive, Tez, Oozie, and Spark > > Python unit tests passed, > ---------------------------------------------------------------------- > Total run:1062 > Total errors:0 > Total failures:0 > OK > > > Thanks, > > Alejandro Fernandez > >
