-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41542/
-----------------------------------------------------------
(Updated Dec. 18, 2015, 4:38 p.m.)
Review request for Ambari, Jonathan Hurley, Jayush Luniya, Myroslav
Papirkovskyy, Nate Cole, and Sumit Mohanty.
Bugs: AMBARI-14430
https://issues.apache.org/jira/browse/AMBARI-14430
Repository: ambari
Description
-------
**Problem**:
Wildcards in packages bring and bunch of problems connected with RU and normal
installs.
a) If we install let's say hadoop_2_2_* and hadoop_2_2_0_1234-libhdfs get
installed, but hadoop_2_2_0_1234 fails, we can no longer install master
package.
b) Packages like hive_2_2-*-atlas-plugin, if for some reason they get
installed before hive_2_2_*, we get problems with inability to install. I've
bumped into a couple of different situations like this. (storm_2_2* and
storm_slider_2_2* etc. ; zookeeper-* and zookeeper-2_2*-server). That all 3
are real bugs which we had to hack out
c) We install a lot of packages which are not needed by default like hadoop-
2_2_...fuse, just by asking hadoop_2_2_*.
d) Situtations like 47931 can happen in RU.
e) The bug in RU where we add a service after install_packages was triggered.
**Proposal:**
1\. Replace * by ${hdp_version} in metainfo.xml.
E.g.:
<osSpecific>
<osFamily>redhat7,redhat6,suse11</osFamily>
<packages>
<package>
<name>hbase_2_3_*</name>
</package>
<package>
<name>phoenix_2_3_*</name>
</package>
</packages>
</osSpecific>
->
<osSpecific>
<osFamily>redhat7,redhat6,suse11</osFamily>
<packages>
<package>
<name>hbase_${hdp_version}</name>
</package>
<package>
<name>hbase_${hdp_version}-regionserver</name>
</package>
<package>
<name>phoenix_${hdp_version}</name>
</package>
</packages>
</osSpecific>
Note: this step will also let us not override packages in metainfo.xml with
every HDP version to just change e.g. hbase_2_2_* to hbase_2_3_*.
2\. During INSTALL action (script.py) or during install packages
(install_packages.py) get the version we are currently using from hdp-select.
e.g. for centos:
[root@andrew-1 ~]# yum info hdp-select | grep 'Version' | awk '{print $3}'
2.3.4.0
[root@andrew-1 ~]# yum info hdp-select | grep 'Release' | awk '{print $3}'
3204 # a bit slow for every INSTALL task
Another better aproach is to install hdp-select and than use rpm/dpkg, which
is faster than yum/zypper/apt-get, considering we will do this on every
INSTALL task this is important.
[root@andrew-1 ~]# rpm -q --queryformat '%{version}-%{release}' hdp-select
| sed -e 's/\.el[0-9]//g' # faster approach
2.3.4.0-3204
u1201:~ # dpkg -s hdp-select | grep 'Version' | awk '{print $2}'
2.3.4.0-3475
Vs other approaches:
I think it's the only possible approach to get the version here, sending
version which is specified by user during creating of a version. Seems to be a
bad idea as the user can specify wrong version. Also we will need to use this
during fresh install, where we don't have it specified at all.
3\. Replace ${hdp_version} with the version we got in step 2.
I also discussed this with dlysnichenko this seems to be feasible in terms of
RU/EU.
The overall idea was discussed in 47931
cc smohanty
Also after talking to dlysnichenko he said we were excluding repos because of
issues with wildcard on suse when it is not excluded and mutptiple packages is
found for a wildcard. That's why when we stop using wildcards we can remove
those exclude repos, which as well caused a bunch of bugs.
Diffs
-----
ambari-common/src/main/python/resource_management/core/providers/package/apt.py
70a553d
ambari-common/src/main/python/resource_management/core/providers/package/yumrpm.py
87938ad
ambari-common/src/main/python/resource_management/core/providers/package/zypper.py
1fbc9bf
ambari-common/src/main/python/resource_management/libraries/functions/get_lzo_packages.py
6dbcef5
ambari-common/src/main/python/resource_management/libraries/functions/packages_analyzer.py
7f34cc1
ambari-common/src/main/python/resource_management/libraries/script/script.py
c954a15
ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_service.py
0d9687b
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py
d39ccbf
ambari-server/src/main/resources/stacks/HDP/2.2/services/FALCON/metainfo.xml
fe47276
ambari-server/src/main/resources/stacks/HDP/2.2/services/FLUME/metainfo.xml
4fa4ff7
ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/metainfo.xml
71297b1
ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/metainfo.xml
0366758
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/metainfo.xml
d59b89e
ambari-server/src/main/resources/stacks/HDP/2.2/services/KAFKA/metainfo.xml
aa164e3
ambari-server/src/main/resources/stacks/HDP/2.2/services/KNOX/metainfo.xml
481fca9
ambari-server/src/main/resources/stacks/HDP/2.2/services/OOZIE/metainfo.xml
db7e025
ambari-server/src/main/resources/stacks/HDP/2.2/services/PIG/metainfo.xml
5472f43
ambari-server/src/main/resources/stacks/HDP/2.2/services/RANGER/metainfo.xml
0c457f7
ambari-server/src/main/resources/stacks/HDP/2.2/services/SLIDER/metainfo.xml
47bb207
ambari-server/src/main/resources/stacks/HDP/2.2/services/SPARK/metainfo.xml
5cb598e
ambari-server/src/main/resources/stacks/HDP/2.2/services/SQOOP/metainfo.xml
e7ff122
ambari-server/src/main/resources/stacks/HDP/2.2/services/STORM/metainfo.xml
ccf765b
ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/metainfo.xml
b0ccddd
ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/metainfo.xml
189defc
ambari-server/src/main/resources/stacks/HDP/2.2/services/ZOOKEEPER/metainfo.xml
d0b37cf
ambari-server/src/main/resources/stacks/HDP/2.3/services/ACCUMULO/metainfo.xml
b19aac1
ambari-server/src/main/resources/stacks/HDP/2.3/services/ATLAS/metainfo.xml
c3a3c81
ambari-server/src/main/resources/stacks/HDP/2.3/services/FALCON/metainfo.xml
6fdad8b
ambari-server/src/main/resources/stacks/HDP/2.3/services/FLUME/metainfo.xml
5031f85
ambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/metainfo.xml
a37378c
ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/metainfo.xml
d103caf
ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/metainfo.xml
b6b6bb8
ambari-server/src/main/resources/stacks/HDP/2.3/services/KAFKA/metainfo.xml
330b542
ambari-server/src/main/resources/stacks/HDP/2.3/services/KNOX/metainfo.xml
d1172b1
ambari-server/src/main/resources/stacks/HDP/2.3/services/OOZIE/metainfo.xml
2fe0ed0
ambari-server/src/main/resources/stacks/HDP/2.3/services/PIG/metainfo.xml
8ba2426
ambari-server/src/main/resources/stacks/HDP/2.3/services/RANGER/metainfo.xml
bc52e85
ambari-server/src/main/resources/stacks/HDP/2.3/services/RANGER_KMS/metainfo.xml
b1a5584
ambari-server/src/main/resources/stacks/HDP/2.3/services/SLIDER/metainfo.xml
8287b50
ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/metainfo.xml
2cf290b
ambari-server/src/main/resources/stacks/HDP/2.3/services/SQOOP/metainfo.xml
1290cde
ambari-server/src/main/resources/stacks/HDP/2.3/services/STORM/metainfo.xml
00af620
ambari-server/src/main/resources/stacks/HDP/2.3/services/TEZ/metainfo.xml
7720a3d
ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/metainfo.xml
088bb16
ambari-server/src/main/resources/stacks/HDP/2.3/services/ZOOKEEPER/metainfo.xml
c2fe742
ambari-server/src/main/resources/stacks/HDP/2.4/services/ACCUMULO/metainfo.xml
9e82a82
ambari-server/src/main/resources/stacks/HDP/2.4/services/ATLAS/metainfo.xml
158ef8c
ambari-server/src/main/resources/stacks/HDP/2.4/services/FALCON/metainfo.xml
183d55c
ambari-server/src/main/resources/stacks/HDP/2.4/services/FLUME/metainfo.xml
b7cd3d1
ambari-server/src/main/resources/stacks/HDP/2.4/services/HBASE/metainfo.xml
a1c2bd0
ambari-server/src/main/resources/stacks/HDP/2.4/services/HDFS/metainfo.xml
ce3fb95
ambari-server/src/main/resources/stacks/HDP/2.4/services/HIVE/metainfo.xml
5ad52b6
ambari-server/src/main/resources/stacks/HDP/2.4/services/KAFKA/metainfo.xml
90bc472
ambari-server/src/main/resources/stacks/HDP/2.4/services/KNOX/metainfo.xml
e465d42
ambari-server/src/main/resources/stacks/HDP/2.4/services/OOZIE/metainfo.xml
2f09e88
ambari-server/src/main/resources/stacks/HDP/2.4/services/PIG/metainfo.xml
9ae1f42
ambari-server/src/main/resources/stacks/HDP/2.4/services/RANGER/metainfo.xml
d76ad5c
ambari-server/src/main/resources/stacks/HDP/2.4/services/RANGER_KMS/metainfo.xml
a27bf31
ambari-server/src/main/resources/stacks/HDP/2.4/services/SLIDER/metainfo.xml
ff998f0
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/metainfo.xml
f4880e2
ambari-server/src/main/resources/stacks/HDP/2.4/services/SQOOP/metainfo.xml
721cc33
ambari-server/src/main/resources/stacks/HDP/2.4/services/STORM/metainfo.xml
36b01e4
ambari-server/src/main/resources/stacks/HDP/2.4/services/TEZ/metainfo.xml
356225e
ambari-server/src/main/resources/stacks/HDP/2.4/services/YARN/metainfo.xml
c61a270
ambari-server/src/main/resources/stacks/HDP/2.4/services/ZOOKEEPER/metainfo.xml
4bf4198
ambari-server/src/test/python/custom_actions/TestInstallPackages.py 6d8c2f0
ambari-server/src/test/python/custom_actions/configs/install_packages_config.json
50e327a
ambari-server/src/test/python/stacks/2.0.6/HBASE/test_hbase_master.py 5a95fe2
Diff: https://reviews.apache.org/r/41542/diff/
Testing (updated)
-------
mvn clean test
- stack 2.3 and 2.4. - full stacks.
- Centos 6 and Ubuntu 12 - full stacks.
- EU full stack
- RU full stack
- downgrade EU
Thanks,
Andrew Onischuk
