> On Dec. 19, 2015, 9:34 p.m., Sumit Mohanty wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/get_lzo_packages.py,
> > line 42
> > <https://reviews.apache.org/r/41542/diff/1/?file=1170750#file1170750line42>
> >
> > Why is these no longer needed?
It's a regexp.
hadoop_2_2_* matches hadoop_2_2_* regexp
now
hadoop_${hdp_version} matches hadoop_* regexp
- Andrew
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41542/#review111382
-----------------------------------------------------------
On Dec. 18, 2015, 4:38 p.m., Andrew Onischuk wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41542/
> -----------------------------------------------------------
>
> (Updated Dec. 18, 2015, 4:38 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Jayush Luniya, Myroslav
> Papirkovskyy, Nate Cole, and Sumit Mohanty.
>
>
> Bugs: AMBARI-14430
> https://issues.apache.org/jira/browse/AMBARI-14430
>
>
> Repository: ambari
>
>
> Description
> -------
>
> **Problem**:
> Wildcards in packages bring and bunch of problems connected with RU and normal
> installs.
> a) If we install let's say hadoop_2_2_* and hadoop_2_2_0_1234-libhdfs get
> installed, but hadoop_2_2_0_1234 fails, we can no longer install master
> package.
> b) Packages like hive_2_2-*-atlas-plugin, if for some reason they get
> installed before hive_2_2_*, we get problems with inability to install. I've
> bumped into a couple of different situations like this. (storm_2_2* and
> storm_slider_2_2* etc. ; zookeeper-* and zookeeper-2_2*-server). That all 3
> are real bugs which we had to hack out
> c) We install a lot of packages which are not needed by default like hadoop-
> 2_2_...fuse, just by asking hadoop_2_2_*.
> d) Situtations like 47931 can happen in RU.
> e) The bug in RU where we add a service after install_packages was triggered.
>
> **Proposal:**
> 1\. Replace * by ${hdp_version} in metainfo.xml.
> E.g.:
>
>
>
> <osSpecific>
> <osFamily>redhat7,redhat6,suse11</osFamily>
> <packages>
> <package>
> <name>hbase_2_3_*</name>
> </package>
> <package>
> <name>phoenix_2_3_*</name>
> </package>
> </packages>
> </osSpecific>
>
>
> ->
>
>
> <osSpecific>
> <osFamily>redhat7,redhat6,suse11</osFamily>
> <packages>
> <package>
> <name>hbase_${hdp_version}</name>
> </package>
> <package>
> <name>hbase_${hdp_version}-regionserver</name>
> </package>
> <package>
> <name>phoenix_${hdp_version}</name>
> </package>
> </packages>
> </osSpecific>
>
>
> Note: this step will also let us not override packages in metainfo.xml with
> every HDP version to just change e.g. hbase_2_2_* to hbase_2_3_*.
>
> 2\. During INSTALL action (script.py) or during install packages
> (install_packages.py) get the version we are currently using from hdp-select.
>
> e.g. for centos:
>
>
>
> [root@andrew-1 ~]# yum info hdp-select | grep 'Version' | awk '{print $3}'
> 2.3.4.0
> [root@andrew-1 ~]# yum info hdp-select | grep 'Release' | awk '{print $3}'
> 3204 # a bit slow for every INSTALL task
>
>
> Another better aproach is to install hdp-select and than use rpm/dpkg, which
> is faster than yum/zypper/apt-get, considering we will do this on every
> INSTALL task this is important.
>
>
>
> [root@andrew-1 ~]# rpm -q --queryformat '%{version}-%{release}'
> hdp-select | sed -e 's/\.el[0-9]//g' # faster approach
> 2.3.4.0-3204
>
>
>
> u1201:~ # dpkg -s hdp-select | grep 'Version' | awk '{print $2}'
> 2.3.4.0-3475
>
>
> Vs other approaches:
> I think it's the only possible approach to get the version here, sending
> version which is specified by user during creating of a version. Seems to be a
> bad idea as the user can specify wrong version. Also we will need to use this
> during fresh install, where we don't have it specified at all.
>
> 3\. Replace ${hdp_version} with the version we got in step 2.
>
> I also discussed this with dlysnichenko this seems to be feasible in terms of
> RU/EU.
> The overall idea was discussed in 47931
> cc smohanty
>
>
> Also after talking to dlysnichenko he said we were excluding repos because of
> issues with wildcard on suse when it is not excluded and mutptiple packages
> is found for a wildcard. That's why when we stop using wildcards we can
> remove those exclude repos, which as well caused a bunch of bugs.
>
>
> Diffs
> -----
>
>
> ambari-common/src/main/python/resource_management/core/providers/package/apt.py
> 70a553d
>
> ambari-common/src/main/python/resource_management/core/providers/package/yumrpm.py
> 87938ad
>
> ambari-common/src/main/python/resource_management/core/providers/package/zypper.py
> 1fbc9bf
>
> ambari-common/src/main/python/resource_management/libraries/functions/get_lzo_packages.py
> 6dbcef5
>
> ambari-common/src/main/python/resource_management/libraries/functions/packages_analyzer.py
> 7f34cc1
>
> ambari-common/src/main/python/resource_management/libraries/script/script.py
> c954a15
>
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_service.py
> 0d9687b
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py
> d39ccbf
>
> ambari-server/src/main/resources/stacks/HDP/2.2/services/FALCON/metainfo.xml
> fe47276
> ambari-server/src/main/resources/stacks/HDP/2.2/services/FLUME/metainfo.xml
> 4fa4ff7
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/metainfo.xml
> 71297b1
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/metainfo.xml
> 0366758
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/metainfo.xml
> d59b89e
> ambari-server/src/main/resources/stacks/HDP/2.2/services/KAFKA/metainfo.xml
> aa164e3
> ambari-server/src/main/resources/stacks/HDP/2.2/services/KNOX/metainfo.xml
> 481fca9
> ambari-server/src/main/resources/stacks/HDP/2.2/services/OOZIE/metainfo.xml
> db7e025
> ambari-server/src/main/resources/stacks/HDP/2.2/services/PIG/metainfo.xml
> 5472f43
>
> ambari-server/src/main/resources/stacks/HDP/2.2/services/RANGER/metainfo.xml
> 0c457f7
>
> ambari-server/src/main/resources/stacks/HDP/2.2/services/SLIDER/metainfo.xml
> 47bb207
> ambari-server/src/main/resources/stacks/HDP/2.2/services/SPARK/metainfo.xml
> 5cb598e
> ambari-server/src/main/resources/stacks/HDP/2.2/services/SQOOP/metainfo.xml
> e7ff122
> ambari-server/src/main/resources/stacks/HDP/2.2/services/STORM/metainfo.xml
> ccf765b
> ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/metainfo.xml
> b0ccddd
> ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/metainfo.xml
> 189defc
>
> ambari-server/src/main/resources/stacks/HDP/2.2/services/ZOOKEEPER/metainfo.xml
> d0b37cf
>
> ambari-server/src/main/resources/stacks/HDP/2.3/services/ACCUMULO/metainfo.xml
> b19aac1
> ambari-server/src/main/resources/stacks/HDP/2.3/services/ATLAS/metainfo.xml
> c3a3c81
>
> ambari-server/src/main/resources/stacks/HDP/2.3/services/FALCON/metainfo.xml
> 6fdad8b
> ambari-server/src/main/resources/stacks/HDP/2.3/services/FLUME/metainfo.xml
> 5031f85
> ambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/metainfo.xml
> a37378c
> ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/metainfo.xml
> d103caf
> ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/metainfo.xml
> b6b6bb8
> ambari-server/src/main/resources/stacks/HDP/2.3/services/KAFKA/metainfo.xml
> 330b542
> ambari-server/src/main/resources/stacks/HDP/2.3/services/KNOX/metainfo.xml
> d1172b1
> ambari-server/src/main/resources/stacks/HDP/2.3/services/OOZIE/metainfo.xml
> 2fe0ed0
> ambari-server/src/main/resources/stacks/HDP/2.3/services/PIG/metainfo.xml
> 8ba2426
>
> ambari-server/src/main/resources/stacks/HDP/2.3/services/RANGER/metainfo.xml
> bc52e85
>
> ambari-server/src/main/resources/stacks/HDP/2.3/services/RANGER_KMS/metainfo.xml
> b1a5584
>
> ambari-server/src/main/resources/stacks/HDP/2.3/services/SLIDER/metainfo.xml
> 8287b50
> ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/metainfo.xml
> 2cf290b
> ambari-server/src/main/resources/stacks/HDP/2.3/services/SQOOP/metainfo.xml
> 1290cde
> ambari-server/src/main/resources/stacks/HDP/2.3/services/STORM/metainfo.xml
> 00af620
> ambari-server/src/main/resources/stacks/HDP/2.3/services/TEZ/metainfo.xml
> 7720a3d
> ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/metainfo.xml
> 088bb16
>
> ambari-server/src/main/resources/stacks/HDP/2.3/services/ZOOKEEPER/metainfo.xml
> c2fe742
>
> ambari-server/src/main/resources/stacks/HDP/2.4/services/ACCUMULO/metainfo.xml
> 9e82a82
> ambari-server/src/main/resources/stacks/HDP/2.4/services/ATLAS/metainfo.xml
> 158ef8c
>
> ambari-server/src/main/resources/stacks/HDP/2.4/services/FALCON/metainfo.xml
> 183d55c
> ambari-server/src/main/resources/stacks/HDP/2.4/services/FLUME/metainfo.xml
> b7cd3d1
> ambari-server/src/main/resources/stacks/HDP/2.4/services/HBASE/metainfo.xml
> a1c2bd0
> ambari-server/src/main/resources/stacks/HDP/2.4/services/HDFS/metainfo.xml
> ce3fb95
> ambari-server/src/main/resources/stacks/HDP/2.4/services/HIVE/metainfo.xml
> 5ad52b6
> ambari-server/src/main/resources/stacks/HDP/2.4/services/KAFKA/metainfo.xml
> 90bc472
> ambari-server/src/main/resources/stacks/HDP/2.4/services/KNOX/metainfo.xml
> e465d42
> ambari-server/src/main/resources/stacks/HDP/2.4/services/OOZIE/metainfo.xml
> 2f09e88
> ambari-server/src/main/resources/stacks/HDP/2.4/services/PIG/metainfo.xml
> 9ae1f42
>
> ambari-server/src/main/resources/stacks/HDP/2.4/services/RANGER/metainfo.xml
> d76ad5c
>
> ambari-server/src/main/resources/stacks/HDP/2.4/services/RANGER_KMS/metainfo.xml
> a27bf31
>
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SLIDER/metainfo.xml
> ff998f0
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml
> f4880e2
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SQOOP/metainfo.xml
> 721cc33
> ambari-server/src/main/resources/stacks/HDP/2.4/services/STORM/metainfo.xml
> 36b01e4
> ambari-server/src/main/resources/stacks/HDP/2.4/services/TEZ/metainfo.xml
> 356225e
> ambari-server/src/main/resources/stacks/HDP/2.4/services/YARN/metainfo.xml
> c61a270
>
> ambari-server/src/main/resources/stacks/HDP/2.4/services/ZOOKEEPER/metainfo.xml
> 4bf4198
> ambari-server/src/test/python/custom_actions/TestInstallPackages.py 6d8c2f0
>
> ambari-server/src/test/python/custom_actions/configs/install_packages_config.json
> 50e327a
> ambari-server/src/test/python/stacks/2.0.6/HBASE/test_hbase_master.py
> 5a95fe2
>
> Diff: https://reviews.apache.org/r/41542/diff/
>
>
> Testing
> -------
>
> mvn clean test
> - stack 2.3 and 2.4. - full stacks.
> - Centos 6 and Ubuntu 12 - full stacks.
> - EU full stack
> - RU full stack
> - downgrade EU
>
>
> Thanks,
>
> Andrew Onischuk
>
>
