> On Jan. 8, 2016, 1:49 p.m., Jonathan Hurley wrote: > > Why go through all of these code changes to accomplish this? It seems like > > you could just have removed the `parallel-scheduler` element from the > > upgrade XML: > > > > ``` > > <group xsi:type="restart" name="HIVE" title="Hive"> > > <service-check>false</service-check> > > <skippable>true</skippable> > > <supports-auto-skip-failure>false</supports-auto-skip-failure> > > <parallel-scheduler/> > > <service name="HIVE"> > > <component>HIVE_METASTORE</component> > > <component>HIVE_SERVER</component> > > <component>WEBHCAT_SERVER</component> > > <component>HIVE_CLIENT</component> > > <component>HCAT</component> > > </service> > > </group> > > ``` > > Nate Cole wrote: > +1 for this > > Alejandro Fernandez wrote: > I want the group to still be in parallel, but only Hive Server and > WebHCat server to be serial. Breaking it up into 2 separate groups seemed too > clunky. If the default behavior of parallel-scheduler changes, then I still > ensure that those 2 components run sequentially. > > Jonathan Hurley wrote: > That's what the groupings are for though. I'd expect this kind of change > to be an XML-only change since it's so simple. Seems like we're introducing a > lot of new code which overcomplicates the solution. > > Alejandro Fernandez wrote: > I thought about that, but didn't want the EU/RU wizard to then show > multiple groups for Hive. > > Jonathan Hurley wrote: > I think this is a matter of trading off between a cleaner XML with a > single group vs having additional complex code for calculating the maximum > degree of parallelism. My feeling is that even though it's a neat feature, > it's not something required to fix this problem. Fixing it in the XML seems > to be more intuitive since it will be represented to the user as 2 different > groups. > > If others feel strongly that we should keep Hive as a single group, then > I'm OK with it. I just didn't think this much code was necessary to fix the > problem. > > Nate Cole wrote: > I would prefer keeping the code simpler. The whole point of the upgrade > pack "verbosity" is to make it clear to someone viewing it what is going to > happen. That there's two hive groups being shown isn't a big deal. "Hive" > and "Hive Clients." > > Jayush Luniya wrote: > While it would have been better to have an ability to override, I agree > that it is less readable based on how the upgrade pack needs to be authored. > It would have been better if groups could have sub-groups and we can set > overrides in sub-groups. That way we wouldn't see separate groups but > internally override group-level properties. For now maybe we can just keep it > simple and leave it as separate groups. > > <group name="Hive"> > <parallel-scheduler> > <max-degree-of-parallelism>100</max-degree-of-parallelism> > </parallel-scheduler> > <sub-group name="Hive"> > <parallel-scheduler> > <max-degree-of-parallelism>1</max-degree-of-parallelism> > </parallel-scheduler> > <sub-group> > <sub-group name="Hive Client"> > </sub-group> > </group> > > Jayush Luniya wrote: > RE: E.g., Hive Server copies tarballs, which can only run one at a time. > Hive MetaStore updates the schema, and only one can run at a time. > Wouldnt this been an issue during clean installs too? (assuming blueprint > based HA deployments)
I wouldn't be surprised if this also has to happen for other masters like Oozie (which runs the prepare step in parallel). In my opinion, it makes the UI more cluttered to have separate groups, but I'm ok with that approach since it's simpler. - Alejandro ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/42038/#review113472 ----------------------------------------------------------- On Jan. 7, 2016, 10:18 p.m., Alejandro Fernandez wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/42038/ > ----------------------------------------------------------- > > (Updated Jan. 7, 2016, 10:18 p.m.) > > > Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jonathan > Hurley, Jayush Luniya, and Nate Cole. > > > Bugs: AMBARI-14581 > https://issues.apache.org/jira/browse/AMBARI-14581 > > > Repository: ambari > > > Description > ------- > > During an Express Upgrade, all of the tasks for Hive Server and Hive > MetaStore run in parallel, so if these components have multiple hosts, then > running in parallel can cause errors during the upgrade. > E.g., Hive Server copies tarballs, which can only run one at a time. Hive > MetaStore updates the schema, and only one can run at a time. > > > Diffs > ----- > > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java > aa852bd > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapperBuilder.java > b0df83f > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Task.java > 60091a6 > > ambari-server/src/main/resources/stacks/HDP/2.1/upgrades/nonrolling-upgrade-2.3.xml > a46a5cd > > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.2.xml > e57b13f > > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.3.xml > b1c3739 > > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml > ba2578b > > ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java > 7e61942 > > ambari-server/src/test/java/org/apache/ambari/server/state/stack/UpgradePackTest.java > 15be8b4 > > ambari-server/src/test/resources/stacks/HDP/2.0.7/services/HIVE/metainfo.xml > 35a4d05 > > ambari-server/src/test/resources/stacks/HDP/2.0.7/services/YARN/metainfo.xml > beee6a0 > > ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_test_nonrolling.xml > 9f57ca8 > > Diff: https://reviews.apache.org/r/42038/diff/ > > > Testing > ------- > > Ran unit tests in UpgradeHelperTest.java and UpgradePackTest.java > Verified with EU from HDP 2.2->2.3 with multiple Hive MetaStores. > > Waiting for unit test results. > > > Thanks, > > Alejandro Fernandez > >
