----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/41542/#review111386 -----------------------------------------------------------
Overall I'm ok with string-substituting the version, but should we query the repo right after we make the .repo file to check if it's an "HDP, versionable repo" instead of trying to see if hdp-select is in it? Can this be controlled (somehow) by the server by having something in cluster-env that would specify what to run to make this decision for us? As I look over this code, and issues come up, it will be extremely difficult to troubleshoot and track what's really happening. - Nate Cole On Dec. 18, 2015, 11:38 a.m., Andrew Onischuk wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/41542/ > ----------------------------------------------------------- > > (Updated Dec. 18, 2015, 11:38 a.m.) > > > Review request for Ambari, Jonathan Hurley, Jayush Luniya, Myroslav > Papirkovskyy, Nate Cole, and Sumit Mohanty. > > > Bugs: AMBARI-14430 > https://issues.apache.org/jira/browse/AMBARI-14430 > > > Repository: ambari > > > Description > ------- > > **Problem**: > Wildcards in packages bring and bunch of problems connected with RU and normal > installs. > a) If we install let's say hadoop_2_2_* and hadoop_2_2_0_1234-libhdfs get > installed, but hadoop_2_2_0_1234 fails, we can no longer install master > package. > b) Packages like hive_2_2-*-atlas-plugin, if for some reason they get > installed before hive_2_2_*, we get problems with inability to install. I've > bumped into a couple of different situations like this. (storm_2_2* and > storm_slider_2_2* etc. ; zookeeper-* and zookeeper-2_2*-server). That all 3 > are real bugs which we had to hack out > c) We install a lot of packages which are not needed by default like hadoop- > 2_2_...fuse, just by asking hadoop_2_2_*. > d) Situtations like 47931 can happen in RU. > e) The bug in RU where we add a service after install_packages was triggered. > > **Proposal:** > 1\. Replace * by ${hdp_version} in metainfo.xml. > E.g.: > > > > <osSpecific> > <osFamily>redhat7,redhat6,suse11</osFamily> > <packages> > <package> > <name>hbase_2_3_*</name> > </package> > <package> > <name>phoenix_2_3_*</name> > </package> > </packages> > </osSpecific> > > > -> > > > <osSpecific> > <osFamily>redhat7,redhat6,suse11</osFamily> > <packages> > <package> > <name>hbase_${hdp_version}</name> > </package> > <package> > <name>hbase_${hdp_version}-regionserver</name> > </package> > <package> > <name>phoenix_${hdp_version}</name> > </package> > </packages> > </osSpecific> > > > Note: this step will also let us not override packages in metainfo.xml with > every HDP version to just change e.g. hbase_2_2_* to hbase_2_3_*. > > 2\. During INSTALL action (script.py) or during install packages > (install_packages.py) get the version we are currently using from hdp-select. > > e.g. for centos: > > > > [root@andrew-1 ~]# yum info hdp-select | grep 'Version' | awk '{print $3}' > 2.3.4.0 > [root@andrew-1 ~]# yum info hdp-select | grep 'Release' | awk '{print $3}' > 3204 # a bit slow for every INSTALL task > > > Another better aproach is to install hdp-select and than use rpm/dpkg, which > is faster than yum/zypper/apt-get, considering we will do this on every > INSTALL task this is important. > > > > [root@andrew-1 ~]# rpm -q --queryformat '%{version}-%{release}' > hdp-select | sed -e 's/\.el[0-9]//g' # faster approach > 2.3.4.0-3204 > > > > u1201:~ # dpkg -s hdp-select | grep 'Version' | awk '{print $2}' > 2.3.4.0-3475 > > > Vs other approaches: > I think it's the only possible approach to get the version here, sending > version which is specified by user during creating of a version. Seems to be a > bad idea as the user can specify wrong version. Also we will need to use this > during fresh install, where we don't have it specified at all. > > 3\. Replace ${hdp_version} with the version we got in step 2. > > I also discussed this with dlysnichenko this seems to be feasible in terms of > RU/EU. > The overall idea was discussed in 47931 > cc smohanty > > > Also after talking to dlysnichenko he said we were excluding repos because of > issues with wildcard on suse when it is not excluded and mutptiple packages > is found for a wildcard. That's why when we stop using wildcards we can > remove those exclude repos, which as well caused a bunch of bugs. > > > Diffs > ----- > > > ambari-common/src/main/python/resource_management/core/providers/package/apt.py > 70a553d > > ambari-common/src/main/python/resource_management/core/providers/package/yumrpm.py > 87938ad > > ambari-common/src/main/python/resource_management/core/providers/package/zypper.py > 1fbc9bf > > ambari-common/src/main/python/resource_management/libraries/functions/get_lzo_packages.py > 6dbcef5 > > ambari-common/src/main/python/resource_management/libraries/functions/packages_analyzer.py > 7f34cc1 > > ambari-common/src/main/python/resource_management/libraries/script/script.py > c954a15 > > ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_service.py > 0d9687b > ambari-server/src/main/resources/custom_actions/scripts/install_packages.py > d39ccbf > > ambari-server/src/main/resources/stacks/HDP/2.2/services/FALCON/metainfo.xml > fe47276 > ambari-server/src/main/resources/stacks/HDP/2.2/services/FLUME/metainfo.xml > 4fa4ff7 > ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/metainfo.xml > 71297b1 > ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/metainfo.xml > 0366758 > ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/metainfo.xml > d59b89e > ambari-server/src/main/resources/stacks/HDP/2.2/services/KAFKA/metainfo.xml > aa164e3 > ambari-server/src/main/resources/stacks/HDP/2.2/services/KNOX/metainfo.xml > 481fca9 > ambari-server/src/main/resources/stacks/HDP/2.2/services/OOZIE/metainfo.xml > db7e025 > ambari-server/src/main/resources/stacks/HDP/2.2/services/PIG/metainfo.xml > 5472f43 > > ambari-server/src/main/resources/stacks/HDP/2.2/services/RANGER/metainfo.xml > 0c457f7 > > ambari-server/src/main/resources/stacks/HDP/2.2/services/SLIDER/metainfo.xml > 47bb207 > ambari-server/src/main/resources/stacks/HDP/2.2/services/SPARK/metainfo.xml > 5cb598e > ambari-server/src/main/resources/stacks/HDP/2.2/services/SQOOP/metainfo.xml > e7ff122 > ambari-server/src/main/resources/stacks/HDP/2.2/services/STORM/metainfo.xml > ccf765b > ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/metainfo.xml > b0ccddd > ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/metainfo.xml > 189defc > > ambari-server/src/main/resources/stacks/HDP/2.2/services/ZOOKEEPER/metainfo.xml > d0b37cf > > ambari-server/src/main/resources/stacks/HDP/2.3/services/ACCUMULO/metainfo.xml > b19aac1 > ambari-server/src/main/resources/stacks/HDP/2.3/services/ATLAS/metainfo.xml > c3a3c81 > > ambari-server/src/main/resources/stacks/HDP/2.3/services/FALCON/metainfo.xml > 6fdad8b > ambari-server/src/main/resources/stacks/HDP/2.3/services/FLUME/metainfo.xml > 5031f85 > ambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/metainfo.xml > a37378c > ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/metainfo.xml > d103caf > ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/metainfo.xml > b6b6bb8 > ambari-server/src/main/resources/stacks/HDP/2.3/services/KAFKA/metainfo.xml > 330b542 > ambari-server/src/main/resources/stacks/HDP/2.3/services/KNOX/metainfo.xml > d1172b1 > ambari-server/src/main/resources/stacks/HDP/2.3/services/OOZIE/metainfo.xml > 2fe0ed0 > ambari-server/src/main/resources/stacks/HDP/2.3/services/PIG/metainfo.xml > 8ba2426 > > ambari-server/src/main/resources/stacks/HDP/2.3/services/RANGER/metainfo.xml > bc52e85 > > ambari-server/src/main/resources/stacks/HDP/2.3/services/RANGER_KMS/metainfo.xml > b1a5584 > > ambari-server/src/main/resources/stacks/HDP/2.3/services/SLIDER/metainfo.xml > 8287b50 > ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/metainfo.xml > 2cf290b > ambari-server/src/main/resources/stacks/HDP/2.3/services/SQOOP/metainfo.xml > 1290cde > ambari-server/src/main/resources/stacks/HDP/2.3/services/STORM/metainfo.xml > 00af620 > ambari-server/src/main/resources/stacks/HDP/2.3/services/TEZ/metainfo.xml > 7720a3d > ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/metainfo.xml > 088bb16 > > ambari-server/src/main/resources/stacks/HDP/2.3/services/ZOOKEEPER/metainfo.xml > c2fe742 > > ambari-server/src/main/resources/stacks/HDP/2.4/services/ACCUMULO/metainfo.xml > 9e82a82 > ambari-server/src/main/resources/stacks/HDP/2.4/services/ATLAS/metainfo.xml > 158ef8c > > ambari-server/src/main/resources/stacks/HDP/2.4/services/FALCON/metainfo.xml > 183d55c > ambari-server/src/main/resources/stacks/HDP/2.4/services/FLUME/metainfo.xml > b7cd3d1 > ambari-server/src/main/resources/stacks/HDP/2.4/services/HBASE/metainfo.xml > a1c2bd0 > ambari-server/src/main/resources/stacks/HDP/2.4/services/HDFS/metainfo.xml > ce3fb95 > ambari-server/src/main/resources/stacks/HDP/2.4/services/HIVE/metainfo.xml > 5ad52b6 > ambari-server/src/main/resources/stacks/HDP/2.4/services/KAFKA/metainfo.xml > 90bc472 > ambari-server/src/main/resources/stacks/HDP/2.4/services/KNOX/metainfo.xml > e465d42 > ambari-server/src/main/resources/stacks/HDP/2.4/services/OOZIE/metainfo.xml > 2f09e88 > ambari-server/src/main/resources/stacks/HDP/2.4/services/PIG/metainfo.xml > 9ae1f42 > > ambari-server/src/main/resources/stacks/HDP/2.4/services/RANGER/metainfo.xml > d76ad5c > > ambari-server/src/main/resources/stacks/HDP/2.4/services/RANGER_KMS/metainfo.xml > a27bf31 > > ambari-server/src/main/resources/stacks/HDP/2.4/services/SLIDER/metainfo.xml > ff998f0 > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml > f4880e2 > ambari-server/src/main/resources/stacks/HDP/2.4/services/SQOOP/metainfo.xml > 721cc33 > ambari-server/src/main/resources/stacks/HDP/2.4/services/STORM/metainfo.xml > 36b01e4 > ambari-server/src/main/resources/stacks/HDP/2.4/services/TEZ/metainfo.xml > 356225e > ambari-server/src/main/resources/stacks/HDP/2.4/services/YARN/metainfo.xml > c61a270 > > ambari-server/src/main/resources/stacks/HDP/2.4/services/ZOOKEEPER/metainfo.xml > 4bf4198 > ambari-server/src/test/python/custom_actions/TestInstallPackages.py 6d8c2f0 > > ambari-server/src/test/python/custom_actions/configs/install_packages_config.json > 50e327a > ambari-server/src/test/python/stacks/2.0.6/HBASE/test_hbase_master.py > 5a95fe2 > > Diff: https://reviews.apache.org/r/41542/diff/ > > > Testing > ------- > > mvn clean test > - stack 2.3 and 2.4. - full stacks. > - Centos 6 and Ubuntu 12 - full stacks. > - EU full stack > - RU full stack > - downgrade EU > > > Thanks, > > Andrew Onischuk > >
