> On Aug. 21, 2017, 7:09 p.m., Di Li wrote: > > ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml > > Lines 379 (patched) > > <https://reviews.apache.org/r/61786/diff/1/?file=1800806#file1800806line379> > > > > Spark2 config in IOP 4.2.5 also has these properties. They are defined > > under different config types though, i.e spark2-default...
Upgrade from 4.2.5 works fine. IOP 4.2.5's default values for both `spark.eventLog.dir` and `spark.history.fs.logDirectory` match those of HDP (`hdfs:///spark2-history/`), thus the problem does not exist there. - Attila ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61786/#review183349 ----------------------------------------------------------- On Aug. 21, 2017, 6:23 p.m., Attila Doroszlai wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61786/ > ----------------------------------------------------------- > > (Updated Aug. 21, 2017, 6:23 p.m.) > > > Review request for Ambari, Bikas Saha, Di Li, Jonathan Hurley, Sumit Mohanty, > Sid Wagle, and Tim Thorpe. > > > Bugs: AMBARI-21768 > https://issues.apache.org/jira/browse/AMBARI-21768 > > > Repository: ambari > > > Description > ------- > > * Get rid of unnecessary `-Dspark.history.fs.logDirectory`. The value is > taken from `spark-defaults` anyway. > * Tweak `spark.eventLog.dir` and `spark.history.fs.logDirectory` so that they > point to the cluster's existing Spark log directory. > > > Diffs > ----- > > > ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml > ad20bf9d9427b6891c4d5944fa3b3e347dbffe12 > > ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml > b8c23bbb5292d374d4fb53b9b43a97ebcb1853cd > > > Diff: https://reviews.apache.org/r/61786/diff/1/ > > > Testing > ------- > > Manual test according to steps in the bug. > Tested with both default and customized log directory location. > Verified that both pre-upgrade and post-upgrade jobs are shown in Spark > History Server UI. > > > Thanks, > > Attila Doroszlai > >
