----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61930/#review183898 -----------------------------------------------------------
Ship it! Ship It! - Mahadev Konar On Aug. 27, 2017, 2:38 a.m., Jonathan Hurley wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61930/ > ----------------------------------------------------------- > > (Updated Aug. 27, 2017, 2:38 a.m.) > > > Review request for Ambari and Mahadev Konar. > > > Bugs: AMBARI-21823 > https://issues.apache.org/jira/browse/AMBARI-21823 > > > Repository: ambari > > > Description > ------- > > Scenario: > * Upgrade ambari from Ambari 2.2.2.0 to 2.5.2.0 > * Upgrade BI 420 from HDP 2.6.2.0 > * > In this upgrade scenario, Spark1 was upgraded from BI->HDP > Spark2 was added after upgrade. > In this cluster , it has Spark and Spark2 both. > However, spark2 shuffle is not added to yarn.nodemanager.aux-services. > {code} > <property> > <name>yarn.nodemanager.aux-services</name> > <value>mapreduce_shuffle,spark_shuffle</value> > </property>{code} > > The migration from IOP to HDP is correct - we're setting the Spark1 > properties. Looks like when adding Spark2, the stack advisor is not > suggestion adding the spark2_shuffle. This goes back to the default behavior > of Ambari which is to always add both {{spark_shuffle}} and > {{spark2_shuffle}} properties when adding either Spark1 or Spark2. > > This was never an issue previously since we were always adding both > aux-services. We can fix this in one of two ways: > - On upgrade, just add both (like we're doing for the 4.2.5 migration) > - Fix stack advisor to correct these properties based on which Spark versions > are installed > > > Diffs > ----- > > > ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml > bdb9ed3b58 > > > Diff: https://reviews.apache.org/r/61930/diff/1/ > > > Testing > ------- > > IOP Upgrade With Spark1 to HDP 2.6 > > > Thanks, > > Jonathan Hurley > >
