> On Dec. 19, 2015, 9:34 p.m., Sumit Mohanty wrote: > > ambari-server/src/main/resources/stacks/HDP/2.3/services/ZOOKEEPER/metainfo.xml, > > line 43 > > <https://reviews.apache.org/r/41542/diff/1/?file=1170791#file1170791line43> > > > > stack_version instead of hdp_version
Sumit you had a lot of very reasonable comments above. Most of that I didn't answer is where I agree with you, I will create a separate ticket and fix those. Thanks for this review you and other guys. Really helpful! - Andrew ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/41542/#review111382 ----------------------------------------------------------- On Dec. 18, 2015, 4:38 p.m., Andrew Onischuk wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/41542/ > ----------------------------------------------------------- > > (Updated Dec. 18, 2015, 4:38 p.m.) > > > Review request for Ambari, Jonathan Hurley, Jayush Luniya, Myroslav > Papirkovskyy, Nate Cole, and Sumit Mohanty. > > > Bugs: AMBARI-14430 > https://issues.apache.org/jira/browse/AMBARI-14430 > > > Repository: ambari > > > Description > ------- > > **Problem**: > Wildcards in packages bring and bunch of problems connected with RU and normal > installs. > a) If we install let's say hadoop_2_2_* and hadoop_2_2_0_1234-libhdfs get > installed, but hadoop_2_2_0_1234 fails, we can no longer install master > package. > b) Packages like hive_2_2-*-atlas-plugin, if for some reason they get > installed before hive_2_2_*, we get problems with inability to install. I've > bumped into a couple of different situations like this. (storm_2_2* and > storm_slider_2_2* etc. ; zookeeper-* and zookeeper-2_2*-server). That all 3 > are real bugs which we had to hack out > c) We install a lot of packages which are not needed by default like hadoop- > 2_2_...fuse, just by asking hadoop_2_2_*. > d) Situtations like 47931 can happen in RU. > e) The bug in RU where we add a service after install_packages was triggered. > > **Proposal:** > 1\. Replace * by ${hdp_version} in metainfo.xml. > E.g.: > > > > <osSpecific> > <osFamily>redhat7,redhat6,suse11</osFamily> > <packages> > <package> > <name>hbase_2_3_*</name> > </package> > <package> > <name>phoenix_2_3_*</name> > </package> > </packages> > </osSpecific> > > > -> > > > <osSpecific> > <osFamily>redhat7,redhat6,suse11</osFamily> > <packages> > <package> > <name>hbase_${hdp_version}</name> > </package> > <package> > <name>hbase_${hdp_version}-regionserver</name> > </package> > <package> > <name>phoenix_${hdp_version}</name> > </package> > </packages> > </osSpecific> > > > Note: this step will also let us not override packages in metainfo.xml with > every HDP version to just change e.g. hbase_2_2_* to hbase_2_3_*. > > 2\. During INSTALL action (script.py) or during install packages > (install_packages.py) get the version we are currently using from hdp-select. > > e.g. for centos: > > > > [root@andrew-1 ~]# yum info hdp-select | grep 'Version' | awk '{print $3}' > 2.3.4.0 > [root@andrew-1 ~]# yum info hdp-select | grep 'Release' | awk '{print $3}' > 3204 # a bit slow for every INSTALL task > > > Another better aproach is to install hdp-select and than use rpm/dpkg, which > is faster than yum/zypper/apt-get, considering we will do this on every > INSTALL task this is important. > > > > [root@andrew-1 ~]# rpm -q --queryformat '%{version}-%{release}' > hdp-select | sed -e 's/\.el[0-9]//g' # faster approach > 2.3.4.0-3204 > > > > u1201:~ # dpkg -s hdp-select | grep 'Version' | awk '{print $2}' > 2.3.4.0-3475 > > > Vs other approaches: > I think it's the only possible approach to get the version here, sending > version which is specified by user during creating of a version. Seems to be a > bad idea as the user can specify wrong version. Also we will need to use this > during fresh install, where we don't have it specified at all. > > 3\. Replace ${hdp_version} with the version we got in step 2. > > I also discussed this with dlysnichenko this seems to be feasible in terms of > RU/EU. > The overall idea was discussed in 47931 > cc smohanty > > > Also after talking to dlysnichenko he said we were excluding repos because of > issues with wildcard on suse when it is not excluded and mutptiple packages > is found for a wildcard. That's why when we stop using wildcards we can > remove those exclude repos, which as well caused a bunch of bugs. > > > Diffs > ----- > > > ambari-common/src/main/python/resource_management/core/providers/package/apt.py > 70a553d > > ambari-common/src/main/python/resource_management/core/providers/package/yumrpm.py > 87938ad > > ambari-common/src/main/python/resource_management/core/providers/package/zypper.py > 1fbc9bf > > ambari-common/src/main/python/resource_management/libraries/functions/get_lzo_packages.py > 6dbcef5 > > ambari-common/src/main/python/resource_management/libraries/functions/packages_analyzer.py > 7f34cc1 > > ambari-common/src/main/python/resource_management/libraries/script/script.py > c954a15 > > ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_service.py > 0d9687b > ambari-server/src/main/resources/custom_actions/scripts/install_packages.py > d39ccbf > > ambari-server/src/main/resources/stacks/HDP/2.2/services/FALCON/metainfo.xml > fe47276 > ambari-server/src/main/resources/stacks/HDP/2.2/services/FLUME/metainfo.xml > 4fa4ff7 > ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/metainfo.xml > 71297b1 > ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/metainfo.xml > 0366758 > ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/metainfo.xml > d59b89e > ambari-server/src/main/resources/stacks/HDP/2.2/services/KAFKA/metainfo.xml > aa164e3 > ambari-server/src/main/resources/stacks/HDP/2.2/services/KNOX/metainfo.xml > 481fca9 > ambari-server/src/main/resources/stacks/HDP/2.2/services/OOZIE/metainfo.xml > db7e025 > ambari-server/src/main/resources/stacks/HDP/2.2/services/PIG/metainfo.xml > 5472f43 > > ambari-server/src/main/resources/stacks/HDP/2.2/services/RANGER/metainfo.xml > 0c457f7 > > ambari-server/src/main/resources/stacks/HDP/2.2/services/SLIDER/metainfo.xml > 47bb207 > ambari-server/src/main/resources/stacks/HDP/2.2/services/SPARK/metainfo.xml > 5cb598e > ambari-server/src/main/resources/stacks/HDP/2.2/services/SQOOP/metainfo.xml > e7ff122 > ambari-server/src/main/resources/stacks/HDP/2.2/services/STORM/metainfo.xml > ccf765b > ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/metainfo.xml > b0ccddd > ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/metainfo.xml > 189defc > > ambari-server/src/main/resources/stacks/HDP/2.2/services/ZOOKEEPER/metainfo.xml > d0b37cf > > ambari-server/src/main/resources/stacks/HDP/2.3/services/ACCUMULO/metainfo.xml > b19aac1 > ambari-server/src/main/resources/stacks/HDP/2.3/services/ATLAS/metainfo.xml > c3a3c81 > > ambari-server/src/main/resources/stacks/HDP/2.3/services/FALCON/metainfo.xml > 6fdad8b > ambari-server/src/main/resources/stacks/HDP/2.3/services/FLUME/metainfo.xml > 5031f85 > ambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/metainfo.xml > a37378c > ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/metainfo.xml > d103caf > ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/metainfo.xml > b6b6bb8 > ambari-server/src/main/resources/stacks/HDP/2.3/services/KAFKA/metainfo.xml > 330b542 > ambari-server/src/main/resources/stacks/HDP/2.3/services/KNOX/metainfo.xml > d1172b1 > ambari-server/src/main/resources/stacks/HDP/2.3/services/OOZIE/metainfo.xml > 2fe0ed0 > ambari-server/src/main/resources/stacks/HDP/2.3/services/PIG/metainfo.xml > 8ba2426 > > ambari-server/src/main/resources/stacks/HDP/2.3/services/RANGER/metainfo.xml > bc52e85 > > ambari-server/src/main/resources/stacks/HDP/2.3/services/RANGER_KMS/metainfo.xml > b1a5584 > > ambari-server/src/main/resources/stacks/HDP/2.3/services/SLIDER/metainfo.xml > 8287b50 > ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/metainfo.xml > 2cf290b > ambari-server/src/main/resources/stacks/HDP/2.3/services/SQOOP/metainfo.xml > 1290cde > ambari-server/src/main/resources/stacks/HDP/2.3/services/STORM/metainfo.xml > 00af620 > ambari-server/src/main/resources/stacks/HDP/2.3/services/TEZ/metainfo.xml > 7720a3d > ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/metainfo.xml > 088bb16 > > ambari-server/src/main/resources/stacks/HDP/2.3/services/ZOOKEEPER/metainfo.xml > c2fe742 > > ambari-server/src/main/resources/stacks/HDP/2.4/services/ACCUMULO/metainfo.xml > 9e82a82 > ambari-server/src/main/resources/stacks/HDP/2.4/services/ATLAS/metainfo.xml > 158ef8c > > ambari-server/src/main/resources/stacks/HDP/2.4/services/FALCON/metainfo.xml > 183d55c > ambari-server/src/main/resources/stacks/HDP/2.4/services/FLUME/metainfo.xml > b7cd3d1 > ambari-server/src/main/resources/stacks/HDP/2.4/services/HBASE/metainfo.xml > a1c2bd0 > ambari-server/src/main/resources/stacks/HDP/2.4/services/HDFS/metainfo.xml > ce3fb95 > ambari-server/src/main/resources/stacks/HDP/2.4/services/HIVE/metainfo.xml > 5ad52b6 > ambari-server/src/main/resources/stacks/HDP/2.4/services/KAFKA/metainfo.xml > 90bc472 > ambari-server/src/main/resources/stacks/HDP/2.4/services/KNOX/metainfo.xml > e465d42 > ambari-server/src/main/resources/stacks/HDP/2.4/services/OOZIE/metainfo.xml > 2f09e88 > ambari-server/src/main/resources/stacks/HDP/2.4/services/PIG/metainfo.xml > 9ae1f42 > > ambari-server/src/main/resources/stacks/HDP/2.4/services/RANGER/metainfo.xml > d76ad5c > > ambari-server/src/main/resources/stacks/HDP/2.4/services/RANGER_KMS/metainfo.xml > a27bf31 > > ambari-server/src/main/resources/stacks/HDP/2.4/services/SLIDER/metainfo.xml > ff998f0 > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml > f4880e2 > ambari-server/src/main/resources/stacks/HDP/2.4/services/SQOOP/metainfo.xml > 721cc33 > ambari-server/src/main/resources/stacks/HDP/2.4/services/STORM/metainfo.xml > 36b01e4 > ambari-server/src/main/resources/stacks/HDP/2.4/services/TEZ/metainfo.xml > 356225e > ambari-server/src/main/resources/stacks/HDP/2.4/services/YARN/metainfo.xml > c61a270 > > ambari-server/src/main/resources/stacks/HDP/2.4/services/ZOOKEEPER/metainfo.xml > 4bf4198 > ambari-server/src/test/python/custom_actions/TestInstallPackages.py 6d8c2f0 > > ambari-server/src/test/python/custom_actions/configs/install_packages_config.json > 50e327a > ambari-server/src/test/python/stacks/2.0.6/HBASE/test_hbase_master.py > 5a95fe2 > > Diff: https://reviews.apache.org/r/41542/diff/ > > > Testing > ------- > > mvn clean test > - stack 2.3 and 2.4. - full stacks. > - Centos 6 and Ubuntu 12 - full stacks. > - EU full stack > - RU full stack > - downgrade EU > > > Thanks, > > Andrew Onischuk > >
