----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/42610/#review115704 -----------------------------------------------------------
Ship it! Minor comments ambari-server/src/main/java/org/apache/ambari/server/checks/HardcodedStackVersionPropertiesCheck.java (lines 89 - 92) <https://reviews.apache.org/r/42610/#comment176722> The FailedOn list shouldn't apply to CLUSTER based checks. The fail reason should be sufficient. ambari-server/src/main/resources/common-services/OOZIE/4.2.0.2.3/configuration/oozie-env.xml (lines 102 - 104) <https://reviews.apache.org/r/42610/#comment176728> Are we sure about this? I could have sworn that other scripts were using $HDP_VERSION that may be passed in with python. - Nate Cole On Jan. 21, 2016, 2:26 p.m., Dmitro Lisnichenko wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/42610/ > ----------------------------------------------------------- > > (Updated Jan. 21, 2016, 2:26 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Dmytro Grinenko, Jonathan > Hurley, Jayush Luniya, and Nate Cole. > > > Bugs: AMBARI-14764 > https://issues.apache.org/jira/browse/AMBARI-14764 > > > Repository: ambari > > > Description > ------- > > # MapReduce jobs were failing after upgrading and restarting MapReduce2 > service (ClassNotFound exception). Also Oozie job were failing because of the > same reason. > # This was caused because mapreduce.application.classpath property had the > older version of the LZO codec > (/usr/hdp/current/hadoop/lib/hadoop-lzo-0.6.0.<OLD-STACK-VERSION>.jar). > # During upgrade the path was valid because we had not switched > /usr/hdp/current/hadoop symlink to new version. After upgrade, once the > symlink is updated to new stack version and after restarting MapReduce2 this > path becomes invalid. > # To fix this updated the LZO codec path to > /usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar in > mapreduce.application.classpath config property. Also in oozie-env property > added export HADOOP_OPTS="-Dhdp.version=$HDP_VERSION $HADOOP_OPTS” > > Fix: > 1. During stack upgrade, update lzo codec path in > mapreduce.application.classpath to > {code} > /usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar > {code} > > Also, looks like we also need to update following property for TEZ. > > {code} > <property> > <name>tez.cluster.additional.classpath.prefix</name> > <value>/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure</value> > <description></description> > </property> > {code} > > 2. Update oozie-env and add > {code} > export HADOOP_OPTS="-Dhdp.version=$HDP_VERSION $HADOOP_OPTS" > {code} > > 3. Add upgrade pre-checks to validate if any config properties have > old-stack-version hardcodings and report these config properties to be > updated. > (attached screenshot) > > > Diffs > ----- > > > ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatMonitor.java > 14f819e > > ambari-server/src/main/java/org/apache/ambari/server/checks/CheckDescription.java > d20203c > > ambari-server/src/main/java/org/apache/ambari/server/checks/HardcodedStackVersionPropertiesCheck.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/FixLzoCodecPath.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/OozieConfigCalculation.java > PRE-CREATION > > ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/configuration/oozie-env.xml > 1b1c8e5 > > ambari-server/src/main/resources/common-services/OOZIE/4.2.0.2.3/configuration/oozie-env.xml > 93742bb > > ambari-server/src/main/resources/stacks/HDP/2.1/upgrades/nonrolling-upgrade-2.3.xml > e581506 > > ambari-server/src/main/resources/stacks/HDP/2.2/services/OOZIE/configuration/oozie-env.xml > f0621cd > > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.2.xml > 098682b > > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.3.xml > fe9d97c > > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.4.xml > b756deb > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.2.xml > d2f6f48 > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml > c058536 > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.4.xml > d279ec1 > > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml > e3b8546 > > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml > 9f21067 > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.3.xml > 5851119 > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml > aef9c79 > > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml > 2600eae > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.4.xml > b91e27a > > ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/FixLzoCodecPathTest.java > PRE-CREATION > > ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/OozieConfigCalculationTest.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/42610/diff/ > > > Testing > ------- > > Things pending: > - E2E tests on live cluster (tested only parts as of now) > - probably add test for HardcodedStackVersionPropertiesCheck > - add LZO codec path modification action to all upgrade packs properly. It is > needed only for HDP 2.2+ upgrades, but all our upgrade packs fill into this > category > > > File Attachments > ---------------- > > screenshot of pre-upgrade check dialog > > https://reviews.apache.org/media/uploaded/files/2016/01/21/05890946-df65-4724-bcdd-3450b159b102__scr1.png > > > Thanks, > > Dmitro Lisnichenko > >
