Re: Review Request 64956: Update Hadoop RPC Encryption Properties During Kerberization and Upgrade
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64956/#review194855
---
Ship it!
Ship It!
- Dmitro Lisnichenko
On Jan. 4, 2018, 10:16 p.m., Jonathan Hurley wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64956/
> ---
>
> (Updated Jan. 4, 2018, 10:16 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-22725
> https://issues.apache.org/jira/browse/AMBARI-22725
>
>
> Repository: ambari
>
>
> Description
> ---
>
> Clients should have the ability to choose encrypted communication over RPC
> when talking to core hadoop components. Today, the properties that control
> this are:
>
> - {{core-site.xml : hadoop.rpc.protection = authentication}}
> - {{hdfs-site.xml : dfs.data.transfer.protection = authentication}}
>
> The new value of {{privacy}} enables clients to choose an encrypted means of
> communication. By keeping {{authentication}} first, it will be taken as the
> default mechanism so that wire encryption is not automatically enabled by
> accident.
>
> The following properties should be changed to add {{privacy}}:
>
> - {{core-site.xml : hadoop.rpc.protection = authentication,privacy}}
> - {{hdfs-site.xml : dfs.data.transfer.protection = authentication,privacy}}
>
> The following are cases when this needs to be performed:
> - During Kerberization, the above two properties should be automatically
> reconfigured.
> - During a stack upgrade to any version of HDP 2.6, they should be
> automatically merged
>
> Blueprint deployment is not a scenario being covered here.
>
>
> Diffs
> -
>
>
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/ConfigureAction.java
> fbcde51b32
>
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java
> c1a05c03b4
>
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/StageWrapperBuilder.java
> 7fd8938f7d
>
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Task.java
> 2167b7b464
> ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py
> 0b0c9c56af
> ambari-server/src/main/resources/stacks/HDP/2.6/services/HDFS/kerberos.json
> f8bdc5cc5c
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml
> 94787225f1
>
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml
> d506f1f16c
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml
> 17a63943ba
> ambari-server/src/main/resources/upgrade-pack.xsd 9e50a087b6
>
> ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java
> ac9be666c7
>
> ambari-server/src/test/resources/stacks/HDP/2.2.0/upgrades/upgrade_test_conditions.xml
> 61c891a4be
>
>
> Diff: https://reviews.apache.org/r/64956/diff/2/
>
>
> Testing
> ---
>
> --
> Total run:1201
> Total errors:0
> Total failures:0
> OK
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 01:01 min
> [INFO] Finished at: 2018-01-04T15:55:20-05:00
> [INFO] Final Memory: 21M/619M
> [INFO]
>
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 64950: Ambari Upgrade failed in case if database have no cluster_version table already. The issue with idempotence
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64950/
---
(Updated Jan. 4, 2018, 4:44 p.m.)
Review request for Ambari, Jonathan Hurley and Nate Cole.
Bugs: AMBARI-22723
https://issues.apache.org/jira/browse/AMBARI-22723
Repository: ambari
Description
---
AMBARI-21728 and AMBARI-22469 make the upgrade broken in case of several
upgrade tries, failing to follow the Idempotent upgrade paradigm.
Issue is with:
{code}
protected void executeDDLUpdates() throws AmbariException, SQLException {
Integer currentVersionID = getCurrentVersionID();<--
dropBrokenFK();
updateServiceComponentDesiredStateTable(currentVersionID);
<--
updateServiceDesiredStateTable(currentVersionID); <--
addSelectedCollumsToClusterconfigTable();
updateHostComponentDesiredStateTable();
updateHostComponentStateTable();
dropStaleTables(); <--
updateUpgradeTable();
createUpgradeHistoryTable();
updateRepositoryVersionTable();
renameServiceDeletedColumn();
addLegacyColumn();
expandUpgradeItemItemTextColumn();
addViewUrlPKConstraint();
removeStaleConstraints();
}
{code}
{{getCurrentVersionID()}} trying to query non-existing table and failing whole
upgrade
Diffs
-
ambari-server/src/main/java/org/apache/ambari/server/orm/DBAccessorImpl.java
391a6f567e
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
6ba1671cdf
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog260Test.java
e93cd1303a
Diff: https://reviews.apache.org/r/64950/diff/1/
Testing (updated)
---
mvn clean test
live cluster check
Thanks,
Dmitro Lisnichenko
Review Request 64950: Ambari Upgrade failed in case if database have no cluster_version table already. The issue with idempotence
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64950/
---
Review request for Ambari, Jonathan Hurley and Nate Cole.
Bugs: AMBARI-22723
https://issues.apache.org/jira/browse/AMBARI-22723
Repository: ambari
Description
---
AMBARI-21728 and AMBARI-22469 make the upgrade broken in case of several
upgrade tries, failing to follow the Idempotent upgrade paradigm.
Issue is with:
{code}
protected void executeDDLUpdates() throws AmbariException, SQLException {
Integer currentVersionID = getCurrentVersionID();<--
dropBrokenFK();
updateServiceComponentDesiredStateTable(currentVersionID);
<--
updateServiceDesiredStateTable(currentVersionID); <--
addSelectedCollumsToClusterconfigTable();
updateHostComponentDesiredStateTable();
updateHostComponentStateTable();
dropStaleTables(); <--
updateUpgradeTable();
createUpgradeHistoryTable();
updateRepositoryVersionTable();
renameServiceDeletedColumn();
addLegacyColumn();
expandUpgradeItemItemTextColumn();
addViewUrlPKConstraint();
removeStaleConstraints();
}
{code}
{{getCurrentVersionID()}} trying to query non-existing table and failing whole
upgrade
Diffs
-
ambari-server/src/main/java/org/apache/ambari/server/orm/DBAccessorImpl.java
391a6f567e
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
6ba1671cdf
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog260Test.java
e93cd1303a
Diff: https://reviews.apache.org/r/64950/diff/1/
Testing
---
mvn clean test
Thanks,
Dmitro Lisnichenko
Re: Review Request 64751: Fix Broken Symlinks on Stack Distribution
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64751/
---
(Updated Dec. 26, 2017, 7:10 p.m.)
Review request for Ambari, Jonathan Hurley and Nate Cole.
Changes
---
added Livy2
Bugs: AMBARI-22678
https://issues.apache.org/jira/browse/AMBARI-22678
Repository: ambari
Description
---
There are two scenarios to cover here:
# Ambari never conf-select'd a component (maybe because of a bug or because the
component didn't support it)
# The conf pointers of a component are broken
In either event, when distributing a new stack, the code detects this problem
(as it would on a first-time install) and tries to fix it:
{code}
/etc/component/conf (directory)
/usr/hdp/current/component -> /usr/hdp/v1/component
/usr/hdp/v1/component -> /etc/component/conf
{code}
The stack distribution thinks this is a first-time installed and tries to fix
the symlinks. We end up with:
{code}
/etc/component/conf -> /usr/hdp/current/component
/usr/hdp/current/component -> /usr/hdp/v1/component
/usr/hdp/v1/component -> /etc/component/conf
/usr/hdp/v2/component -> /etc/component/v2/0
{code}
Because we're only conf-selecting v2, v1 never gets corrected since it's
already installed. Thus, we have a circular symlink.
Most likely the proper fix will be:
- Iterate over the entire known conf-select structure
- Check to see the state /etc/component/conf - if it's bad, fix it to defaults
Chances are we can do this directly in
{{conf_select.convert_conf_directories_to_symlinks}}:
{code}
stack_name = Script.get_stack_name()
for directory_struct in dirs:
if not os.path.exists(directory_struct['conf_dir']):
Logger.info("Skipping the conf-select tool on {0} since {1} does not
exist.".format(
package, directory_struct['conf_dir']))
return
{code}
Diffs (updated)
-
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py
add5a7789e
ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_service.py
0180a315b6
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py
5cff342076
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
dc71b4da4f
Diff: https://reviews.apache.org/r/64751/diff/3/
Changes: https://reviews.apache.org/r/64751/diff/2-3/
Testing
---
mvn clean test
Live cluster check
Thanks,
Dmitro Lisnichenko
Re: Review Request 64667: Livy server fails to start during downgrade due to absence of 'conf' directory
> On Dec. 22, 2017, 5:13 p.m., Nate Cole wrote: > > What's the status of this? I thought we fixed all the outstanding Livy > > issues? It has been postponed to 2.6.2. Will open an another review request - Dmitro --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64667/#review194438 --- On Dec. 16, 2017, 5:53 p.m., Dmitro Lisnichenko wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64667/ > --- > > (Updated Dec. 16, 2017, 5:53 p.m.) > > > Review request for Ambari, Jonathan Hurley, Mahadev Konar, and Nate Cole. > > > Bugs: AMBARI-22522 > https://issues.apache.org/jira/browse/AMBARI-22522 > > > Repository: ambari > > > Description > --- > > Adding a small, livy-only workaround into install_packages.py > > > Diffs > - > > ambari-server/src/main/resources/custom_actions/scripts/install_packages.py > c8497cd > > > Diff: https://reviews.apache.org/r/64667/diff/1/ > > > Testing > --- > > mvn clean test, > live cluster check > > > Thanks, > > Dmitro Lisnichenko > >
Re: Review Request 64751: Fix Broken Symlinks on Stack Distribution
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64751/
---
(Updated Dec. 21, 2017, 3:25 p.m.)
Review request for Ambari, Jonathan Hurley and Nate Cole.
Changes
---
Addressed comments
Bugs: AMBARI-22678
https://issues.apache.org/jira/browse/AMBARI-22678
Repository: ambari
Description
---
There are two scenarios to cover here:
# Ambari never conf-select'd a component (maybe because of a bug or because the
component didn't support it)
# The conf pointers of a component are broken
In either event, when distributing a new stack, the code detects this problem
(as it would on a first-time install) and tries to fix it:
{code}
/etc/component/conf (directory)
/usr/hdp/current/component -> /usr/hdp/v1/component
/usr/hdp/v1/component -> /etc/component/conf
{code}
The stack distribution thinks this is a first-time installed and tries to fix
the symlinks. We end up with:
{code}
/etc/component/conf -> /usr/hdp/current/component
/usr/hdp/current/component -> /usr/hdp/v1/component
/usr/hdp/v1/component -> /etc/component/conf
/usr/hdp/v2/component -> /etc/component/v2/0
{code}
Because we're only conf-selecting v2, v1 never gets corrected since it's
already installed. Thus, we have a circular symlink.
Most likely the proper fix will be:
- Iterate over the entire known conf-select structure
- Check to see the state /etc/component/conf - if it's bad, fix it to defaults
Chances are we can do this directly in
{{conf_select.convert_conf_directories_to_symlinks}}:
{code}
stack_name = Script.get_stack_name()
for directory_struct in dirs:
if not os.path.exists(directory_struct['conf_dir']):
Logger.info("Skipping the conf-select tool on {0} since {1} does not
exist.".format(
package, directory_struct['conf_dir']))
return
{code}
Diffs (updated)
-
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py
cce2148ab6
ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_service.py
dfadd84613
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py
c8497cd2cb
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
62a46b91bd
Diff: https://reviews.apache.org/r/64751/diff/2/
Changes: https://reviews.apache.org/r/64751/diff/1-2/
Testing
---
mvn clean test
Live cluster check
Thanks,
Dmitro Lisnichenko
Re: Review Request 64751: Fix Broken Symlinks on Stack Distribution
> On Dec. 20, 2017, 5:03 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/custom_actions/scripts/install_packages.py
> > Lines 199-201 (patched)
> > <https://reviews.apache.org/r/64751/diff/1/?file=1925645#file1925645line200>
> >
> > I'd say just return earlier like this:
> >
> > if not stack_version:
> > Logger.warn("Unable to fix {0} since there is no known installed
> > version for this component".format(package_name))
Stack is being checked for every package in a loop, so return here would break
the execution flow
- Dmitro
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64751/#review194249
-----------
On Dec. 20, 2017, 4:33 p.m., Dmitro Lisnichenko wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64751/
> ---
>
> (Updated Dec. 20, 2017, 4:33 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley and Nate Cole.
>
>
> Bugs: AMBARI-22678
> https://issues.apache.org/jira/browse/AMBARI-22678
>
>
> Repository: ambari
>
>
> Description
> ---
>
> There are two scenarios to cover here:
>
> # Ambari never conf-select'd a component (maybe because of a bug or because
> the component didn't support it)
> # The conf pointers of a component are broken
>
> In either event, when distributing a new stack, the code detects this problem
> (as it would on a first-time install) and tries to fix it:
> {code}
> /etc/component/conf (directory)
> /usr/hdp/current/component -> /usr/hdp/v1/component
> /usr/hdp/v1/component -> /etc/component/conf
> {code}
>
> The stack distribution thinks this is a first-time installed and tries to fix
> the symlinks. We end up with:
> {code}
> /etc/component/conf -> /usr/hdp/current/component
> /usr/hdp/current/component -> /usr/hdp/v1/component
> /usr/hdp/v1/component -> /etc/component/conf
> /usr/hdp/v2/component -> /etc/component/v2/0
> {code}
>
> Because we're only conf-selecting v2, v1 never gets corrected since it's
> already installed. Thus, we have a circular symlink.
>
> Most likely the proper fix will be:
> - Iterate over the entire known conf-select structure
> - Check to see the state /etc/component/conf - if it's bad, fix it to defaults
>
> Chances are we can do this directly in
> {{conf_select.convert_conf_directories_to_symlinks}}:
> {code}
> stack_name = Script.get_stack_name()
> for directory_struct in dirs:
> if not os.path.exists(directory_struct['conf_dir']):
> Logger.info("Skipping the conf-select tool on {0} since {1} does not
> exist.".format(
> package, directory_struct['conf_dir']))
>
> return
> {code}
>
>
> Diffs
> -
>
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py
> cce2148
>
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_service.py
> dfadd84
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py
> c8497cd
>
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
> 62a46b9
>
>
> Diff: https://reviews.apache.org/r/64751/diff/1/
>
>
> Testing
> ---
>
> mvn clean test
>
> Live cluster check
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 64722: [Patch Hive]webhcat: test_sqoop fails with hdfs:///hdp/apps/2.6.*/sqoop/sqoop.tar.gz does not exist
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64722/#review194243 --- Ship it! Ship It! - Dmitro Lisnichenko On Dec. 19, 2017, 10:58 p.m., Nate Cole wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64722/ > --- > > (Updated Dec. 19, 2017, 10:58 p.m.) > > > Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Jonathan > Hurley. > > > Bugs: AMBARI-22676 > https://issues.apache.org/jira/browse/AMBARI-22676 > > > Repository: ambari > > > Description > --- > > Add SQOOP to the list of services that should go with HIVE > > > Diffs > - > > > ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json > 62a46b91bd > > > Diff: https://reviews.apache.org/r/64722/diff/1/ > > > Testing > --- > > No new tests for json change. > > > Thanks, > > Nate Cole > >
Review Request 64751: Fix Broken Symlinks on Stack Distribution
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64751/
---
Review request for Ambari, Jonathan Hurley and Nate Cole.
Bugs: AMBARI-22678
https://issues.apache.org/jira/browse/AMBARI-22678
Repository: ambari
Description
---
There are two scenarios to cover here:
# Ambari never conf-select'd a component (maybe because of a bug or because the
component didn't support it)
# The conf pointers of a component are broken
In either event, when distributing a new stack, the code detects this problem
(as it would on a first-time install) and tries to fix it:
{code}
/etc/component/conf (directory)
/usr/hdp/current/component -> /usr/hdp/v1/component
/usr/hdp/v1/component -> /etc/component/conf
{code}
The stack distribution thinks this is a first-time installed and tries to fix
the symlinks. We end up with:
{code}
/etc/component/conf -> /usr/hdp/current/component
/usr/hdp/current/component -> /usr/hdp/v1/component
/usr/hdp/v1/component -> /etc/component/conf
/usr/hdp/v2/component -> /etc/component/v2/0
{code}
Because we're only conf-selecting v2, v1 never gets corrected since it's
already installed. Thus, we have a circular symlink.
Most likely the proper fix will be:
- Iterate over the entire known conf-select structure
- Check to see the state /etc/component/conf - if it's bad, fix it to defaults
Chances are we can do this directly in
{{conf_select.convert_conf_directories_to_symlinks}}:
{code}
stack_name = Script.get_stack_name()
for directory_struct in dirs:
if not os.path.exists(directory_struct['conf_dir']):
Logger.info("Skipping the conf-select tool on {0} since {1} does not
exist.".format(
package, directory_struct['conf_dir']))
return
{code}
Diffs
-
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py
cce2148
ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_service.py
dfadd84
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py
c8497cd
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
62a46b9
Diff: https://reviews.apache.org/r/64751/diff/1/
Testing
---
mvn clean test
Live cluster check
Thanks,
Dmitro Lisnichenko
Re: Review Request 64667: Livy server fails to start during downgrade due to absence of 'conf' directory
> On Dec. 17, 2017, 9:14 p.m., Jonathan Hurley wrote: > > ambari-server/src/main/resources/custom_actions/scripts/install_packages.py > > Lines 155 (patched) > > <https://reviews.apache.org/r/64667/diff/1/?file=1919066#file1919066line156> > > > > OK - let's just make this a little less hard-coded ... > > > > Can we pass in the paramters of livy/livy-server to this method and > > call it something like "fix_default_links" > > > > We should also have some documentation as to why we need this method ... > > Jonathan Hurley wrote: > Also, i think livy2 also had this > > Dmitro Lisnichenko wrote: > This fix will not work for Livy2, because it is also missing from > stack_packages.json also, conf-select has not idea about Livy2 /usr/bin/conf-select dry-run-create --package livy2 --stack-version 2.6.3.0-235 --conf-version 0 livy2 not installed or incorrect package name - Dmitro --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64667/#review194033 --- On Dec. 16, 2017, 5:53 p.m., Dmitro Lisnichenko wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64667/ > --- > > (Updated Dec. 16, 2017, 5:53 p.m.) > > > Review request for Ambari, Jonathan Hurley, Mahadev Konar, and Nate Cole. > > > Bugs: AMBARI-22522 > https://issues.apache.org/jira/browse/AMBARI-22522 > > > Repository: ambari > > > Description > --- > > Adding a small, livy-only workaround into install_packages.py > > > Diffs > - > > ambari-server/src/main/resources/custom_actions/scripts/install_packages.py > c8497cd > > > Diff: https://reviews.apache.org/r/64667/diff/1/ > > > Testing > --- > > mvn clean test, > live cluster check > > > Thanks, > > Dmitro Lisnichenko > >
Re: Review Request 64667: Livy server fails to start during downgrade due to absence of 'conf' directory
> On Dec. 17, 2017, 9:14 p.m., Jonathan Hurley wrote: > > ambari-server/src/main/resources/custom_actions/scripts/install_packages.py > > Lines 155 (patched) > > <https://reviews.apache.org/r/64667/diff/1/?file=1919066#file1919066line156> > > > > OK - let's just make this a little less hard-coded ... > > > > Can we pass in the paramters of livy/livy-server to this method and > > call it something like "fix_default_links" > > > > We should also have some documentation as to why we need this method ... > > Jonathan Hurley wrote: > Also, i think livy2 also had this This fix will not work for Livy2, because it is also missing from stack_packages.json - Dmitro --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64667/#review194033 ----------- On Dec. 16, 2017, 5:53 p.m., Dmitro Lisnichenko wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64667/ > --- > > (Updated Dec. 16, 2017, 5:53 p.m.) > > > Review request for Ambari, Jonathan Hurley, Mahadev Konar, and Nate Cole. > > > Bugs: AMBARI-22522 > https://issues.apache.org/jira/browse/AMBARI-22522 > > > Repository: ambari > > > Description > --- > > Adding a small, livy-only workaround into install_packages.py > > > Diffs > - > > ambari-server/src/main/resources/custom_actions/scripts/install_packages.py > c8497cd > > > Diff: https://reviews.apache.org/r/64667/diff/1/ > > > Testing > --- > > mvn clean test, > live cluster check > > > Thanks, > > Dmitro Lisnichenko > >
Re: Review Request 64667: Livy server fails to start during downgrade due to absence of 'conf' directory
> On Dec. 16, 2017, 10:35 p.m., Jonathan Hurley wrote: > > ambari-server/src/main/resources/custom_actions/scripts/install_packages.py > > Lines 160-164 (patched) > > <https://reviews.apache.org/r/64667/diff/1/?file=1919066#file1919066line161> > > > > We should not hard code this into install_packages. Can we just expose > > this method for the Livy Python files to call? > > Dmitro Lisnichenko wrote: > In this case, this code should run before install_packages.py . > Otherwise, symlinks will be badly broken, and my code will not be able to fix > that. I can move the code to restart() of Livy, but we would have to require > users (via release notes) to restart Livy before installing packages for a > new stack. > Should I move this code to restart? Or should leave it as is, intending > to refactor/generalize it at 2.6.2? > > Jonathan Hurley wrote: > I don't see how they will be badly broken - we're updating the conf > pointers after the packages are installed anyway. Let's say that we have this > problem in the system: > > /etc/livy/conf -> /usr/hdp/current/livy (broken) > /usr/hdp/current/livy-server/conf -> /etc/component/conf (wrong) > > During Livy's configure() or pre_upgrade_restart(), can we specifically > invoke conf_select.convert_conf_directories_to_symlinks() for the Livy > package? > > ``` > old_conf = directory_struct['conf_dir'] > current_dir = directory_struct['current_dir'] > if os.path.islink(old_conf): > # it's already a link; make sure it's a link to where we want it > if os.readlink(old_conf) != current_dir: > ``` > > Seems like this would allow us to detect the broken link in > /etc/livy/conf and fix it. > > Jonathan Hurley wrote: > Ah, I dove deeper into this problem. To summarize: > > /usr/hdp/current/livy-client/conf -> /etc/livy/conf exists on Ambari 2.5. > > After an upgrade to 2.6, we distribute packages for the new stack. The > code detects this problem (as it would on a first-time install) and tries to > fix it: > > /etc/livy/conf -> /usr/hdp/current/livy-client/conf > /usr/hdp/v1/livy/conf -> /etc/livy/conf > /usr/hdp/v2/livy/conf -> /etc/livy/v2/0 > > Because we're only conf-selecting v2, v1 never gets correct since it's > already installed... exactly - Dmitro --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64667/#review194019 --- On Dec. 16, 2017, 5:53 p.m., Dmitro Lisnichenko wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64667/ > --- > > (Updated Dec. 16, 2017, 5:53 p.m.) > > > Review request for Ambari, Jonathan Hurley, Mahadev Konar, and Nate Cole. > > > Bugs: AMBARI-22522 > https://issues.apache.org/jira/browse/AMBARI-22522 > > > Repository: ambari > > > Description > --- > > Adding a small, livy-only workaround into install_packages.py > > > Diffs > - > > ambari-server/src/main/resources/custom_actions/scripts/install_packages.py > c8497cd > > > Diff: https://reviews.apache.org/r/64667/diff/1/ > > > Testing > --- > > mvn clean test, > live cluster check > > > Thanks, > > Dmitro Lisnichenko > >
Re: Review Request 64667: Livy server fails to start during downgrade due to absence of 'conf' directory
> On Dec. 16, 2017, 10:35 p.m., Jonathan Hurley wrote: > > ambari-server/src/main/resources/custom_actions/scripts/install_packages.py > > Lines 160-164 (patched) > > <https://reviews.apache.org/r/64667/diff/1/?file=1919066#file1919066line161> > > > > We should not hard code this into install_packages. Can we just expose > > this method for the Livy Python files to call? In this case, this code should run before install_packages.py . Otherwise, symlinks will be badly broken, and my code will not be able to fix that. I can move the code to restart() of Livy, but we would have to require users (via release notes) to restart Livy before installing packages for a new stack. Should I move this code to restart? Or should leave it as is, intending to refactor/generalize it at 2.6.2? - Dmitro --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64667/#review194019 ------- On Dec. 16, 2017, 5:53 p.m., Dmitro Lisnichenko wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64667/ > --- > > (Updated Dec. 16, 2017, 5:53 p.m.) > > > Review request for Ambari, Jonathan Hurley, Mahadev Konar, and Nate Cole. > > > Bugs: AMBARI-22522 > https://issues.apache.org/jira/browse/AMBARI-22522 > > > Repository: ambari > > > Description > --- > > Adding a small, livy-only workaround into install_packages.py > > > Diffs > - > > ambari-server/src/main/resources/custom_actions/scripts/install_packages.py > c8497cd > > > Diff: https://reviews.apache.org/r/64667/diff/1/ > > > Testing > --- > > mvn clean test, > live cluster check > > > Thanks, > > Dmitro Lisnichenko > >
Re: Review Request 64637: Livy/Livy2 Unable To Start Due to Address Already In Use
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64637/#review194016
---
Ship it!
Ship It!
- Dmitro Lisnichenko
On Dec. 15, 2017, 5:38 a.m., Jonathan Hurley wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64637/
> ---
>
> (Updated Dec. 15, 2017, 5:38 a.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-22655
> https://issues.apache.org/jira/browse/AMBARI-22655
>
>
> Repository: ambari
>
>
> Description
> ---
>
> While restarting Livy and Livy2 on a non-root cluster, the following is seen:
>
> ```
> 17/12/14 14:36:23 WARN LivyConf: The configuration key
> livy.repl.enableHiveContext has been deprecated as of Livy 0.4 and may be
> removed in the future. Please use the new key livy.repl.enable-hive-context
> instead.
> 17/12/14 14:36:23 WARN LivyConf: The configuration key
> livy.server.csrf_protection.enabled has been deprecated as of Livy 0.4 and
> may be removed in the future. Please use the new key
> livy.server.csrf-protection.enabled instead.
> 17/12/14 14:36:23 INFO AccessManager: AccessControlManager acls
> disabled;users with view permission: ;users with modify permission: ;users
> with super permission: cstm-zeppelin;other allowed users: *
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Welcome to
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: __
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: / __/__ ___ _/
> /__
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: _\ \/ _ \/ _ `/ __/
> '_/
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout:/___/ .__/_,_/_/ /_/_\
> version 2.2.0.2.6.4.0-73
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: /_/
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout:
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Using Scala version
> 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_131
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Branch HEAD
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Compiled by user jenkins
> on 2017-12-13T19:08:32Z
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Revision
> a24017869f5450397136ee8b11be818e7cd3facb
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Url
> g...@github.com:hortonworks/spark2.git
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Type --help for more
> information.
> 17/12/14 14:36:29 WARN NativeCodeLoader: Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 17/12/14 14:36:31 INFO AHSProxy: Connecting to Application History server at
> nat-yc-r7-ovvs-ambari-autostart-4-re-2.openstacklocal/172.22.121.144:10200
> 17/12/14 14:36:32 WARN DomainSocketFactory: The short-circuit local reads
> feature cannot be used because libhadoop cannot be loaded.
> 17/12/14 14:36:33 INFO StateStore$: Using FileSystemStateStore for recovery.
> 17/12/14 14:36:33 INFO BatchSessionManager: Recovered 0 batch sessions. Next
> session id: 0
> 17/12/14 14:36:33 INFO InteractiveSessionManager: Recovered 0 interactive
> sessions. Next session id: 0
> 17/12/14 14:36:33 INFO InteractiveSessionManager: Heartbeat watchdog thread
> started.
> 17/12/14 14:36:33 INFO LivyServer: SPNEGO auth enabled (principal =
> HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklo...@example.com)
> 17/12/14 14:36:33 INFO LivyServer: CSRF protection is enabled.
> 17/12/14 14:36:34 INFO KerberosAuthenticationHandler: Login using keytab
> /etc/security/keytabs/spnego.service.keytab, for principal
> HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklo...@example.com
> 17/12/14 14:36:34 INFO KerberosAuthenticationHandler: Map server:
> nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal to principal:
> [HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklo...@example.com],
> added = true
> 17/12/14 14:36:34 WARN AbstractLifeCycle: FAILED
> ServerConnector@df1cff6{SSL-http/1.1}{0.0.0.0:8999}: java.net.BindException:
> Address already in use
> java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> ```
>
> This occurs because the PID file cannot be accessed by the non-root agent and
> returns an exit code of 1:
> {code}
> call returned (1, 'cat: /var/run/livy/livy-cstm-livy-server.pid: Permission
> denied')
> {code}
>
> This tricks out PID detection into thi
Review Request 64667: Livy server fails to start during downgrade due to absence of 'conf' directory
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64667/ --- Review request for Ambari, Jonathan Hurley, Mahadev Konar, and Nate Cole. Bugs: AMBARI-22522 https://issues.apache.org/jira/browse/AMBARI-22522 Repository: ambari Description --- Adding a small, livy-only workaround into install_packages.py Diffs - ambari-server/src/main/resources/custom_actions/scripts/install_packages.py c8497cd Diff: https://reviews.apache.org/r/64667/diff/1/ Testing --- mvn clean test, live cluster check Thanks, Dmitro Lisnichenko
Re: Review Request 64652: unable to proceed with cluster install after component install fails.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64652/#review193934
---
Ship it!
Ship It!
- Dmitro Lisnichenko
On Dec. 15, 2017, 7 p.m., Myroslav Papirkovskyy wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64652/
> ---
>
> (Updated Dec. 15, 2017, 7 p.m.)
>
>
> Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, and Nate Cole.
>
>
> Bugs: AMBARI-22659
> https://issues.apache.org/jira/browse/AMBARI-22659
>
>
> Repository: ambari
>
>
> Description
> ---
>
> Unable go back in install wizard if install was finished at one of hosts.
>
> ```
> Request
> URL:http://ambari:8080/api/v1/stacks/HDP/versions/2.6/repository_versions/1
>
> {
> "status" : 500,
> "message" : "org.apache.ambari.server.controller.spi.SystemException:
> Repository version can't be deleted as it is used by the following hosts:
> CURRENT on sanjay-divgi-test-2.openstacklocal,
> sanjay-divgi-test-5.openstacklocal, sanjay-divgi-test-1.openstacklocal,
> sanjay-divgi-test-4.openstacklocal"
> }
> ```
>
>
> Diffs
> -
>
>
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostVersionDAO.java
> 158370f3528f796e30a46862e8eb42882001ca02
>
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostVersionEntity.java
> 4a030af122dbe3ceb3c3592c000b126c864dc626
>
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
> c23b971b85a843433f230b5a94c66ac46c5b1914
>
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/HostVersionDAOTest.java
> bb077d6667138acdaeaafa8b07f188b2ce3dc2af
>
>
> Diff: https://reviews.apache.org/r/64652/diff/1/
>
>
> Testing
> ---
>
> mvn clean test
> manual check
>
>
> Thanks,
>
> Myroslav Papirkovskyy
>
>
Re: Review Request 64554: HBase Cannot Find LZO Classes After Being Patched
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64554/#review193705 --- Ship it! Ship It! - Dmitro Lisnichenko On Dec. 13, 2017, 5:46 p.m., Jonathan Hurley wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64554/ > --- > > (Updated Dec. 13, 2017, 5:46 p.m.) > > > Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole. > > > Bugs: AMBARI-22640 > https://issues.apache.org/jira/browse/AMBARI-22640 > > > Repository: ambari > > > Description > --- > > After patching HBase where LZO compression is being used, the following is > seen: > > ``` > 2017-12-10 22:31:09,244|INFO|MainThread|machine.py:164 - > run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|Exception in thread "main" > java.lang.RuntimeException: java.lang.ClassNotFoundException: > com.hadoop.compression.lzo.LzoCodec > 2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - > run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at > org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.buildCodec(Compression.java:130) > 2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - > run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at > org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.getCodec(Compression.java:116) > 2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - > run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at > org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:301) > 2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - > run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at > org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultEncodingContext.2017-12-10 > 22:31:09,267|INFO|MainThread|machine.py:164 - > run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > 2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - > run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at > java.util.concurrent.FutureTask.run(FutureTask.java:266) > 2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - > run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > 2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - > run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > 2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - > run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at > java.lang.Thread.run(Thread.java:745) > 2017-12-10 22:31:09,741|INFO|MainThread|machine.py:189 - > run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|Exit Code: 1 > ``` > > > Diffs > - > > > ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase.py > ac71ce4b36 > > > Diff: https://reviews.apache.org/r/64554/diff/2/ > > > Testing > --- > > Manual testing on Hbase cluster. > > > Thanks, > > Jonathan Hurley > >
Re: Review Request 64502: YARN Shuffle Service Can't Be Found On Client-Only Nodes After New Cluster Install
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64502/#review193410
---
Ship it!
Ship It!
- Dmitro Lisnichenko
On Dec. 11, 2017, 7:04 p.m., Jonathan Hurley wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64502/
> ---
>
> (Updated Dec. 11, 2017, 7:04 p.m.)
>
>
> Review request for Ambari, Dmitro Lisnichenko and Nate Cole.
>
>
> Bugs: AMBARI-22628
> https://issues.apache.org/jira/browse/AMBARI-22628
>
>
> Repository: ambari
>
>
> Description
> ---
>
> Installing a new cluster can create values in yarn-site.xml which have
> {{None}} specified in the classpath for Spark
>
> ```
>
> yarn.nodemanager.aux-services.spark2_shuffle.classpath
> /usr/hdp/None/spark2/aux/*
>
>
>
> yarn.nodemanager.aux-services.spark_shuffle.classpath
> /usr/hdp/None/spark/aux/*
>
>
>
>
> yarn.timeline-service.entity-group-fs-store.group-id-plugin-classpath
> /usr/hdp/None/spark/hdpLib/*
>
> ```
>
> The cause for this is that YARN Clients on hosts without daemons never get a
> restart command after the initial {{yarn-site.xml}}, and can never fill in
> the correct values. This causes problems when jobs are run on these nodes:
>
> ```
> 2017-12-04 10:16:41,789 INFO service.AbstractService
> (AbstractService.java:noteFailure(272)) - Service
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices failed
> in state INITED; cause: java.lang.ClassNotFoundException:
> org.apache.spark.network.yarn.YarnShuffleService
> java.lang.ClassNotFoundException:
> org.apache.spark.network.yarn.YarnShuffleService
> ```
>
>
> Diffs
> -
>
>
> ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
> 9d5e29ee8a
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProvider.java
> a7c712bd1a
> ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java
> 15efcd2173
>
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
> ce328f91ff
>
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
> 98141456c7
>
>
> Diff: https://reviews.apache.org/r/64502/diff/3/
>
>
> Testing
> ---
>
> Manual install via UI and Blueprint
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 64419: Hive Queries Failing with Missing SnappyCodec Compression Type Due to Hive's Use of MapReduce Admin Env Property
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64419/#review193236
---
Ship it!
Ship It!
- Dmitro Lisnichenko
On Dec. 7, 2017, 10:53 p.m., Jonathan Hurley wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64419/
> ---
>
> (Updated Dec. 7, 2017, 10:53 p.m.)
>
>
> Review request for Ambari, Dmitro Lisnichenko and Nate Cole.
>
>
> Bugs: AMBARI-22613
> https://issues.apache.org/jira/browse/AMBARI-22613
>
>
> Repository: ambari
>
>
> Description
> ---
>
> Many queries, similar to the below {{InsertOverwrite}}, are failing after
> performing an HDP->HDP upgrade. The cause is that Hive does not localize the
> Tez TAR file, but instead uses the MapReduce2 property
> {{mapreduce.admin.user.env}}
>
> ```
> >>> analyze table studenttab10k compute statistics for columns age ;
> INFO : Session is already open
> INFO : Dag name: analyze table studenttab10k compute st...age(Stage-0)
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1512568339263_0003)
>
> [2K
> [2K[36;1mVERTICES STATUS TOTAL COMPLETED RUNNING PENDING
> FAILED KILLED
> [22;0m[2K
> [2KMap 1RUNNING 1 010 2
> 0
> Reducer 2 INITED 1 001 0
> 0
> [2K
> [2K[31;1mVERTICES: 00/02 [>>--] 0%ELAPSED
> TIME: 9.38 s
> [22;0m[2K
> [8A[2K
> [2K[36;1mVERTICES STATUS TOTAL COMPLETED RUNNING PENDING
> FAILED KILLED
> [22;0m[2K
> [2KMap 1RUNNING 1 001 4
> 0
> Reducer 2 INITED 1 001 0
> 0
> [2K
> [2K[31;1mVERTICES: 00/02 [>>--] 0%ELAPSED
> TIME: 14.37 s
> [22;0m[2K
> ERROR : Status: Failed
> ERROR : Vertex failed, vertexName=Map 1,
> vertexId=vertex_1512568339263_0003_2_00, diagnostics=[Task failed,
> taskId=task_1512568339263_0003_2_00_00, diagnostics=[TaskAttempt 0
> failed, info=[Error: Failure while running task:java.lang.RuntimeException:
> java.io.IOException: Unable to get CompressorType for codec
> (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to
> missing native libraries for the codec.
> Caused by: java.lang.RuntimeException: native snappy library not available:
> this version of libhadoop was built without snappy support.
> at
> org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65)
> at
> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134)
> at
> org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:208)
> ... 18 more
> ], TaskAttempt 1 failed, info=[Error: Failure while running
> task:java.lang.RuntimeException: java.io.IOException: Unable to get
> CompressorType for codec (org.apache.hadoop.io.compress.SnappyCodec). This is
> most likely due to missing native libraries for the codec.
> ```
>
>
> Diffs
> -
>
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
> 18e297865e
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml
> 6e1c81968d
>
>
> Diff: https://reviews.apache.org/r/64419/diff/2/
>
>
> Testing
> ---
>
> Manual upgrade testing.
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 64348: Pig service check failed after PU with LzoCodec CNF
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64348/#review192974 --- Ship it! Ship It! - Dmitro Lisnichenko On Dec. 5, 2017, 8:33 p.m., Jonathan Hurley wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64348/ > --- > > (Updated Dec. 5, 2017, 8:33 p.m.) > > > Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole. > > > Bugs: AMBARI-22598 > https://issues.apache.org/jira/browse/AMBARI-22598 > > > Repository: ambari > > > Description > --- > > When Pig (and other hadoop clients) are upgraded by themselves, they need to > create jobs with versions of the LZO libraries which match _their_ versions. > > *STR* > # Deployed cluster with Ambari version: 2.5.1.0-159 and HDP version: > 2.6.1.0-129 (HDFS has lzo enabled) > # Upgrade Ambari to Target Version: 2.6.1.0-84 > # Perform full stack upgrade to 2.6.4.0-55 > # Perform 4th digit PU of some services in batches (see attached for list) > # Run Pig service check > > *Result* > ``` > Caused by: java.lang.IllegalArgumentException: Compression codec > com.hadoop.compression.lzo.LzoCodec not found. > at > org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:139) > at > org.apache.hadoop.io.compress.CompressionCodecFactory.(CompressionCodecFactory.java:180) > at > org.apache.hadoop.mapreduce.lib.input.TextInputFormat.isSplitable(TextInputFormat.java:59) > at > org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:399) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:265) > ... 28 more > Caused by: java.lang.ClassNotFoundException: Class > com.hadoop.compression.lzo.LzoCodec not found > at > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2147) > at > org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:132) > ... 32 more > 2017-12-04 12:07:30,322 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR > 2118: Compression codec com.hadoop.compression.lzo.LzoCodec not found. > Details at logfile: /home/ambari-qa/pig_1512389247042.log > 2017-12-04 12:07:30,338 [main] INFO org.apache.pig.Main - Pig script > completed in 3 seconds and 457 milliseconds (3457 ms) > 2017-12-04 12:07:30,338 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher - Shutting down > thread pool > ``` > > > Diffs > - > > > ambari-server/src/main/resources/common-services/MAHOUT/1.0.0.2.3/package/scripts/mahout.py > f2c3c18077 > > ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/pig.py > 83f70488a3 > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py > 426509cb0e > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/setup_spark.py > 82b6c63d6d > > > Diff: https://reviews.apache.org/r/64348/diff/1/ > > > Testing > --- > > OK > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 01:29 min > [INFO] Finished at: 2017-12-05T13:27:57-05:00 > [INFO] Final Memory: 21M/619M > [INFO] > > > > Thanks, > > Jonathan Hurley > >
Re: Review Request 64340: Livy server start fails during EU with 'Address already in use' error
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64340/
---
(Updated Dec. 5, 2017, 7:29 p.m.)
Review request for Ambari, Dmytro Grinenko, Jonathan Hurley, and Nate Cole.
Changes
---
tested changes
Summary (updated)
-
Livy server start fails during EU with 'Address already in use' error
Bugs: AMBARI-22594
https://issues.apache.org/jira/browse/AMBARI-22594
Repository: ambari
Description
---
Observed this issue quite consistently in Ambari-2.6.1 Upgrade ST runs
*STR*
# Deployed cluster with Ambari version: 2.5.1.0-159 and HDP version: 2.6.1.0-129
# Upgrade Ambari to Target Version: 2.6.1.0-43 | Hash:
acbce28fdd119c72625c6beff63fc169de58ba22
# Regenerate keytabs post Ambari upgrade and this step will restart all
services. Here Livy server is operational and gets restarted fine (at
timestamp: 09:29)
# Now register HDP-2.6.4.0-36 version and perform EU. During EU 'Restart Livy
server' task happens and reports success (at timestamp: 10:26)
# However when checking the livy logs - Livy restart reported below exception
as the previous process was not killed/stopped
{code}
17/11/21 10:26:22 WARN AbstractLifeCycle: FAILED
org.eclipse.jetty.server.Server@3bc735b3: java.net.BindException: Address
already in use
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321)
at org.apache.livy.server.LivyServer.main(LivyServer.scala)
Exception in thread "main" java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
{code}
- Post Upgrade, I tried to stop/start Spark as well and Livy still gave same
exception; although web Ui reports operation as success (at timestamp: 11:37)
- Finally the web UI shows Livy as down, even though the process is running
from the initial step (at timestamp: 09:29)
Diffs (updated)
-
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py
a78f50c077
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
9b813a13f0
ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_service.py
0d60cf41ad
ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py
1968b0e8fa
Diff: https://reviews.apache.org/r/64340/diff/2/
Changes: https://reviews.apache.org/r/64340/diff/1-2/
Testing (updated)
---
mvn clean test
check on live cluster
Thanks,
Dmitro Lisnichenko
Review Request 64340: PREVIEW - Livy server start fails during EU with 'Address already in use' error
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64340/
---
Review request for Ambari, Dmytro Grinenko, Jonathan Hurley, and Nate Cole.
Bugs: AMBARI-22594
https://issues.apache.org/jira/browse/AMBARI-22594
Repository: ambari
Description
---
Observed this issue quite consistently in Ambari-2.6.1 Upgrade ST runs
*STR*
# Deployed cluster with Ambari version: 2.5.1.0-159 and HDP version: 2.6.1.0-129
# Upgrade Ambari to Target Version: 2.6.1.0-43 | Hash:
acbce28fdd119c72625c6beff63fc169de58ba22
# Regenerate keytabs post Ambari upgrade and this step will restart all
services. Here Livy server is operational and gets restarted fine (at
timestamp: 09:29)
# Now register HDP-2.6.4.0-36 version and perform EU. During EU 'Restart Livy
server' task happens and reports success (at timestamp: 10:26)
# However when checking the livy logs - Livy restart reported below exception
as the previous process was not killed/stopped
{code}
17/11/21 10:26:22 WARN AbstractLifeCycle: FAILED
org.eclipse.jetty.server.Server@3bc735b3: java.net.BindException: Address
already in use
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321)
at org.apache.livy.server.LivyServer.main(LivyServer.scala)
Exception in thread "main" java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
{code}
- Post Upgrade, I tried to stop/start Spark as well and Livy still gave same
exception; although web Ui reports operation as success (at timestamp: 11:37)
- Finally the web UI shows Livy as down, even though the process is running
from the initial step (at timestamp: 09:29)
Diffs
-
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py
a78f50c077
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
9b813a13f0
Diff: https://reviews.apache.org/r/64340/diff/1/
Testing
---
waiting for feedback from reviewers to apply this change to all Livy and Livy2
service scripts
Thanks,
Dmitro Lisnichenko
Re: Review Request 64316: Messages for some services during PU package installation indicate circular dependency
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64316/#review192837 --- Ship it! Ship It! - Dmitro Lisnichenko On Dec. 5, 2017, 12:13 a.m., Jonathan Hurley wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64316/ > --- > > (Updated Dec. 5, 2017, 12:13 a.m.) > > > Review request for Ambari, Dmitro Lisnichenko and Nate Cole. > > > Bugs: AMBARI-22590 > https://issues.apache.org/jira/browse/AMBARI-22590 > > > Repository: ambari > > > Description > --- > > Written this as 'Blocker' severity, please triage once and decide > > This issue could affect user experience for Patch Upgrades > > *Case 1:* Register Patch VDF for Mapreduce service > Try to install the packages > Result: Error for missing Hive service in the VDF (see MR-only.png) > > *Case 2:* Add Hive to the patch VDF, os that it now includes both MR and Hive > Try to install the packages > Result: Error for missing Tez service in the VDF (see MR-and-Hive) > > This is now confusing as the dependency looks cyclic, we asked to add Hive in > first case and now asking to add MR again > > *Case 3:* Add a new VDF for only Mahout or Oozie. This would prompt for MR > due to dependency. Now you need to add MR and Hive and then Tez too (see > Mahout-only.png) > > *Expected result:* The message for dependent services should recursively > decide the services and print them all at once. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/checks/RequiredServicesInRepositoryCheck.java > d9114113b2 > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProvider.java > d155d46a58 > > ambari-server/src/main/java/org/apache/ambari/server/state/repository/VersionDefinitionXml.java > 69e22b37e0 > > ambari-server/src/test/java/org/apache/ambari/server/checks/RequiredServicesInRepositoryCheckTest.java > 59848719ad > > ambari-server/src/test/java/org/apache/ambari/server/state/repository/VersionDefinitionTest.java > 9fe6146a9c > > > Diff: https://reviews.apache.org/r/64316/diff/2/ > > > Testing > --- > > PENDING > > > Thanks, > > Jonathan Hurley > >
Re: Review Request 63516: Remove properties.json And Switch To Adding Properties to ResourceProviders Dynamically
/internal/HostComponentResourceProviderTest.java 7e32c15991 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/HostResourceProviderTest.java 5e6201bcd1 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/HostStackVersionResourceProviderTest.java 6c95c53e5b ambari-server/src/test/java/org/apache/ambari/server/controller/internal/InstanceResourceProviderTest.java ba6793989e ambari-server/src/test/java/org/apache/ambari/server/controller/internal/JMXHostProviderTest.java e19ad29453 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/JobResourceProviderTest.java 6fd1a238ae ambari-server/src/test/java/org/apache/ambari/server/controller/internal/KerberosDescriptorResourceProviderTest.java ca8127e42f ambari-server/src/test/java/org/apache/ambari/server/controller/internal/MemberResourceProviderTest.java 39ba4b9fe3 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RequestResourceProviderTest.java c0695b12d6 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RequestScheduleResourceProviderTest.java 6d4f72d9ad ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RootServiceComponentResourceProviderTest.java 4316647e8e ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RootServiceHostComponentResourceProviderTest.java ce3656c4ab ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RootServiceResourceProviderTest.java 96bc7c9674 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackAdvisorResourceProviderTest.java 05232eac49 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackArtifactResourceProviderTest.java 21df88d545 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackConfigurationDependencyResourceProviderTest.java 7c88852f48 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackConfigurationResourceProviderTest.java 9c48ad0d23 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackDependencyResourceProviderTest.java f35015348f ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackLevelConfigurationResourceProviderTest.java e9742bdfb1 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackResourceProviderTest.java 4281e3372e ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackServiceResourceProviderTest.java 2a2ef0b0b7 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/TargetClusterResourceProviderTest.java 492c012c7e ambari-server/src/test/java/org/apache/ambari/server/controller/internal/TaskAttemptResourceProviderTest.java 992bce0936 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/TaskResourceProviderTest.java 365d408ff6 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/TestIvoryProviderModule.java 90f10118d4 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java ec240df800 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UserResourceProviderDBTest.java c4f0f349fb ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UserResourceProviderTest.java 9efad8a54d ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ValidationResourceProviderTest.java 42d4770a25 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/WorkflowResourceProviderTest.java 337dc466da Diff: https://reviews.apache.org/r/63516/diff/5/ Changes: https://reviews.apache.org/r/63516/diff/4-5/ Testing (updated) --- tests pass checked live cluster deploy and EU Thanks, Dmitro Lisnichenko
Re: Review Request 64223: Oozie Fails To Restart During Upgrade Because of Missing ExtJS Library
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64223/#review192471
---
Ship it!
Ship It!
- Dmitro Lisnichenko
On Nov. 30, 2017, 11:18 p.m., Jonathan Hurley wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64223/
> ---
>
> (Updated Nov. 30, 2017, 11:18 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-22568
> https://issues.apache.org/jira/browse/AMBARI-22568
>
>
> Repository: ambari
>
>
> Description
> ---
>
> 1) Deployed cluster with Ambari version: 2.6.1.0-65 and HDP version:
> 2.6.3.0-215
> 2) Express upgrade to target version 2.6.4.0-52
>
> Oozie Server Restart fails with below error:
>
> ```
> Traceback (most recent call last):
> ...
> File
> "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py",
> line 123, in pre_upgrade_restart
> OozieUpgrade.prepare_libext_directory(upgrade_type=upgrade_type)
> File
> "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py",
> line 113, in prepare_libext_directory
> raise Fail("Unable to find any Oozie source extension files from the
> following paths {0}".format(source_ext_zip_paths))
> resource_management.core.exceptions.Fail: Unable to find any Oozie source
> extension files from the following paths ['/usr/share/HDP-oozie/ext-2.2.zip',
> '/var/lib/oozie/ext-2.2.zip']
> ```
>
> This is because we have removed the requirement of having ExtJS in the
> cluster.
>
>
> Diffs
> -
>
>
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
> b81186144c
>
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py
> eb57c22834
>
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
> 2109a5d5e9
> ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py
> 4d94723bbf
>
>
> Diff: https://reviews.apache.org/r/64223/diff/2/
>
>
> Testing
> ---
>
> Upgraded Oozie
>
> --
> Total run:1196
> Total errors:0
> Total failures:0
> OK
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 58.251 s
> [INFO] Finished at: 2017-11-30T15:16:50-05:00
> [INFO] Final Memory: 21M/619M
> [INFO]
>
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 63516: [Preview] Remove properties.json And Switch To Adding Properties to ResourceProviders Dynamically
/HostResourceProviderTest.java fd28081944 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/HostStackVersionResourceProviderTest.java 6c95c53e5b ambari-server/src/test/java/org/apache/ambari/server/controller/internal/InstanceResourceProviderTest.java ba6793989e ambari-server/src/test/java/org/apache/ambari/server/controller/internal/JMXHostProviderTest.java e19ad29453 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/JobResourceProviderTest.java 6fd1a238ae ambari-server/src/test/java/org/apache/ambari/server/controller/internal/KerberosDescriptorResourceProviderTest.java ca8127e42f ambari-server/src/test/java/org/apache/ambari/server/controller/internal/MemberResourceProviderTest.java 39ba4b9fe3 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RequestResourceProviderTest.java c0695b12d6 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RequestScheduleResourceProviderTest.java 6d4f72d9ad ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RootServiceComponentResourceProviderTest.java 4316647e8e ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RootServiceHostComponentResourceProviderTest.java ce3656c4ab ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RootServiceResourceProviderTest.java 96bc7c9674 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackAdvisorResourceProviderTest.java 05232eac49 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackArtifactResourceProviderTest.java 21df88d545 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackConfigurationDependencyResourceProviderTest.java 7c88852f48 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackConfigurationResourceProviderTest.java 9c48ad0d23 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackDependencyResourceProviderTest.java f35015348f ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackLevelConfigurationResourceProviderTest.java e9742bdfb1 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackResourceProviderTest.java 4281e3372e ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackServiceResourceProviderTest.java 2a2ef0b0b7 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/TargetClusterResourceProviderTest.java 492c012c7e ambari-server/src/test/java/org/apache/ambari/server/controller/internal/TaskAttemptResourceProviderTest.java 992bce0936 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/TaskResourceProviderTest.java 365d408ff6 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/TestIvoryProviderModule.java 90f10118d4 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java ec240df800 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UserResourceProviderDBTest.java c4f0f349fb ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UserResourceProviderTest.java 9efad8a54d ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ValidationResourceProviderTest.java c639d1fcdc ambari-server/src/test/java/org/apache/ambari/server/controller/internal/WorkflowResourceProviderTest.java 337dc466da Diff: https://reviews.apache.org/r/63516/diff/4/ Changes: https://reviews.apache.org/r/63516/diff/3-4/ Testing --- tests do not pass yet, also on deploy get errors in some places. Fixing mistakes Thanks, Dmitro Lisnichenko
Re: Review Request 64205: Snapshot HBase task failed during IOP migration with TypeError
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64205/
---
(Updated Nov. 30, 2017, 11:20 p.m.)
Review request for Ambari, Dmytro Grinenko, Jonathan Hurley, and Nate Cole.
Changes
---
removed * import, checked on live IOP cluster
Bugs: AMBARI-22558
https://issues.apache.org/jira/browse/AMBARI-22558
Repository: ambari
Description
---
*STR*
# Deployed cluster with Ambari version: 2.2.0 and IOP version: 4.2.0.0
# Upgrade Ambari to Target Version: 2.5.2.0-298 | Hash:
2453e16418fd964042452b649153dbe45f3c6009
# Upgrade Ambari to Target Version: 2.6.1.0-64 | Hash:
cd4db8e9ac0ea7ce14fc1253959a121688f34952
# Register HDP-2.6.4.0-51 and call remove iop-select
# Install the new HDP version bits and start Express Upgrade
*Result*
Snapshot HBase task failed with below error:
{code}
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/custom_actions/scripts/ru_execute_tasks.py",
line 157, in
ExecuteUpgradeTasks().execute()
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 367, in execute
method(env)
File "/var/lib/ambari-agent/cache/custom_actions/scripts/ru_execute_tasks.py",
line 153, in actionexecute
shell.checked_call(task.command, logoutput=True, quiet=True)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line
72, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line
102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line
150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line
303, in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'source
/var/lib/ambari-agent/ambari-env.sh ; /usr/bin/ambari-python-wrap
/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py
take_snapshot /var/lib/ambari-agent/data/command-404.json
/var/lib/ambari-agent/cache/custom_actions
/var/lib/ambari-agent/data/structured-out-404.json INFO
/var/lib/ambari-agent/tmp' returned 1. 2017-11-29 07:01:50,049 - Stack Feature
Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=4.2.0.0,
Upgrade Direction=upgrade -> 4.2.0.0
2017-11-29 07:01:50,096 - Using hadoop conf dir:
/usr/hdp/current/hadoop-client/conf
Traceback (most recent call last):
File
"/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py",
line 37, in
HbaseMasterUpgrade().execute()
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 367, in execute
method(env)
File
"/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py",
line 28, in take_snapshot
import params
File
"/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py",
line 107, in
regionserver_xmn_percent =
expect("/configurations/hbase-env/hbase_regionserver_xmn_ratio", float)
#AMBARI-15614
TypeError: 'module' object is not callable
{code}
Upon checking the code
[here|https://github.com/hortonworks/ambari/blob/AMBARI-2.6.1.0/ambari-server/src/main/resources/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py#L30],
the issue seems to be with import of 'expect' module
I tried the following changes in params.py:
{code}
L30: from resource_management.libraries.functions import expect
L107: regionserver_xmn_percent =
expect.expect("/configurations/hbase-env/hbase_regionserver_xmn_ratio", float)
#AMBARI-15614
{code}
Now the snapshot command ran fine:
{code}
2017-11-29 11:14:15,472 - Stack Feature Version Info: Cluster Stack=2.6,
Command Stack=None, Command Version=4.2.0.0, Upgrade Direction=upgrade ->
4.2.0.0
2017-11-29 11:14:15,474 - Using hadoop conf dir:
/usr/hdp/current/hadoop-client/conf
2017-11-29 11:14:15,476 - checked_call['hostid'] {}
2017-11-29 11:14:15,494 - checked_call returned (0, '16ace057')
2017-11-29 11:14:15,495 - Execute[' echo 'snapshot_all' |
/usr/iop/current/hbase-client/bin/hbase shell'] {'user': 'hbase'}
{code}
Diffs (updated)
-
ambari-server/src/main/resources/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py
c0c314d
Diff: https://reviews.apache.org/r/64205/diff/3/
Changes: https://reviews.apache.org/r/64205/diff/2-3/
Testing
---
mvn clean test
Thanks,
Dmitro Lisnichenko
Re: Review Request 64205: Snapshot HBase task failed during IOP migration with TypeError
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64205/
---
(Updated Nov. 30, 2017, 5:04 p.m.)
Review request for Ambari, Dmytro Grinenko, Jonathan Hurley, and Nate Cole.
Changes
---
another approach
Bugs: AMBARI-22558
https://issues.apache.org/jira/browse/AMBARI-22558
Repository: ambari
Description
---
*STR*
# Deployed cluster with Ambari version: 2.2.0 and IOP version: 4.2.0.0
# Upgrade Ambari to Target Version: 2.5.2.0-298 | Hash:
2453e16418fd964042452b649153dbe45f3c6009
# Upgrade Ambari to Target Version: 2.6.1.0-64 | Hash:
cd4db8e9ac0ea7ce14fc1253959a121688f34952
# Register HDP-2.6.4.0-51 and call remove iop-select
# Install the new HDP version bits and start Express Upgrade
*Result*
Snapshot HBase task failed with below error:
{code}
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/custom_actions/scripts/ru_execute_tasks.py",
line 157, in
ExecuteUpgradeTasks().execute()
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 367, in execute
method(env)
File "/var/lib/ambari-agent/cache/custom_actions/scripts/ru_execute_tasks.py",
line 153, in actionexecute
shell.checked_call(task.command, logoutput=True, quiet=True)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line
72, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line
102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line
150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line
303, in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'source
/var/lib/ambari-agent/ambari-env.sh ; /usr/bin/ambari-python-wrap
/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py
take_snapshot /var/lib/ambari-agent/data/command-404.json
/var/lib/ambari-agent/cache/custom_actions
/var/lib/ambari-agent/data/structured-out-404.json INFO
/var/lib/ambari-agent/tmp' returned 1. 2017-11-29 07:01:50,049 - Stack Feature
Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=4.2.0.0,
Upgrade Direction=upgrade -> 4.2.0.0
2017-11-29 07:01:50,096 - Using hadoop conf dir:
/usr/hdp/current/hadoop-client/conf
Traceback (most recent call last):
File
"/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py",
line 37, in
HbaseMasterUpgrade().execute()
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 367, in execute
method(env)
File
"/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py",
line 28, in take_snapshot
import params
File
"/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py",
line 107, in
regionserver_xmn_percent =
expect("/configurations/hbase-env/hbase_regionserver_xmn_ratio", float)
#AMBARI-15614
TypeError: 'module' object is not callable
{code}
Upon checking the code
[here|https://github.com/hortonworks/ambari/blob/AMBARI-2.6.1.0/ambari-server/src/main/resources/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py#L30],
the issue seems to be with import of 'expect' module
I tried the following changes in params.py:
{code}
L30: from resource_management.libraries.functions import expect
L107: regionserver_xmn_percent =
expect.expect("/configurations/hbase-env/hbase_regionserver_xmn_ratio", float)
#AMBARI-15614
{code}
Now the snapshot command ran fine:
{code}
2017-11-29 11:14:15,472 - Stack Feature Version Info: Cluster Stack=2.6,
Command Stack=None, Command Version=4.2.0.0, Upgrade Direction=upgrade ->
4.2.0.0
2017-11-29 11:14:15,474 - Using hadoop conf dir:
/usr/hdp/current/hadoop-client/conf
2017-11-29 11:14:15,476 - checked_call['hostid'] {}
2017-11-29 11:14:15,494 - checked_call returned (0, '16ace057')
2017-11-29 11:14:15,495 - Execute[' echo 'snapshot_all' |
/usr/iop/current/hbase-client/bin/hbase shell'] {'user': 'hbase'}
{code}
Diffs (updated)
-
ambari-server/src/main/resources/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py
c0c314dcdf
Diff: https://reviews.apache.org/r/64205/diff/2/
Changes: https://reviews.apache.org/r/64205/diff/1-2/
Testing
---
mvn clean test
Thanks,
Dmitro Lisnichenko
Review Request 64205: Snapshot HBase task failed during IOP migration with TypeError
/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py b8e709a6db ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py 46f715ecde ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py 5e1f4ac5cb ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py 18e297865e ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/params.py 4eb5b02c3f ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params.py f39d6326de ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py 8f38843364 ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py eaf1ee4940 ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/params_linux.py d9fae763ad ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5/package/scripts/params_linux.py 34c6b9ab96 ambari-server/src/main/resources/stacks/BigInsights/4.0/services/HBASE/package/scripts/params.py 97657ad7f4 ambari-server/src/main/resources/stacks/BigInsights/4.2.5/hooks/before-ANY/scripts/params.py 1ed6d4d748 ambari-server/src/main/resources/stacks/BigInsights/4.2.5/hooks/before-INSTALL/scripts/params.py 6193c11c22 ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/HBASE/package/scripts/params_linux.py bff022d1f7 ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/HDFS/package/scripts/params_linux.py a77ec82cdc ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/OOZIE/package/scripts/params.py f39d6326de ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/OOZIE/package/scripts/params_linux.py 5868f5e604 ambari-server/src/main/resources/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py c0c314dcdf ambari-server/src/main/resources/stacks/BigInsights/4.2/services/KAFKA/package/scripts/params.py 07e7ad19d1 ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py 8ad0d51376 ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/params.py da75a8ee89 ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-ANY/scripts/params.py 2c2c90113a ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/params.py 4eb5b02c3f Diff: https://reviews.apache.org/r/64205/diff/1/ Testing --- mvn clean test Thanks, Dmitro Lisnichenko
Re: Review Request 64090: ATS start failed during patch upgrade due to CNF SparkATSPlugin
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64090/#review192016
---
Ship it!
Ship It!
- Dmitro Lisnichenko
On Nov. 27, 2017, 9:01 p.m., Jonathan Hurley wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64090/
> ---
>
> (Updated Nov. 27, 2017, 9:01 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-22525
> https://issues.apache.org/jira/browse/AMBARI-22525
>
>
> Repository: ambari
>
>
> Description
> ---
>
> After patching only YARN, ATS will fail to start with the following because
> it cannot find the correct Spark JARS:
> ```
> 2017-11-20 11:36:45,457 FATAL
> applicationhistoryservice.ApplicationHistoryServer
> (ApplicationHistoryServer.java:launchAppHistoryServer(177)) - Error starting
> ApplicationHistoryServer
> java.lang.RuntimeException: No class defined for
> org.apache.spark.deploy.history.yarn.plugin.SparkATSPlugin
> ```
>
> This is because Spark is configured via properties which currently use
> ${hdp.version} which will match the version of YARN and not Spark.
>
>
> Diffs
> -
>
>
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
> 4079f1866e
>
> ambari-server/src/main/resources/stacks/BigInsights/4.0/stack-advisor/stack_advisor_25.py
> 7e77382e2b
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/yarn-site.xml
> b6fadcbec5
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py
> a3b8263454
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml
> 45380728e3
>
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml
> f3cea4875d
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml
> 3bb6322fcb
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py
> bb11969a98
>
>
> Diff: https://reviews.apache.org/r/64090/diff/1/
>
>
> Testing
> ---
>
> Manual install & upgrade
>
> --
> Total run:1195
> Total errors:0
> Total failures:0
> OK
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 59.118 s
> [INFO] Finished at: 2017-11-27T13:47:56-05:00
> [INFO] Final Memory: 21M/619M
> [INFO]
>
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 64030: Modify AMBARI-22387 to Check for LZO + No Opt-in
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64030/ --- (Updated Nov. 22, 2017, 5:43 p.m.) Review request for Ambari, Jonathan Hurley and Nate Cole. Changes --- Fix order of imports Bugs: AMBARI-22500 https://issues.apache.org/jira/browse/AMBARI-22500 Repository: ambari Description --- The work for AMBARI-22387 was initially created to warn the user before performing an upgrade that LZO was enabled and would need to be managed on their own. However, with AMBARI-22457, we now support LZO when a user opts-in to installed GPL-licensed code. The warning in AMBARI-22387 should be changed to check for LZO being enabled but the user _NOT_ opt-ing in Diffs (updated) - ambari-server/src/main/java/org/apache/ambari/server/checks/LZOCheck.java 9c0286b ambari-server/src/test/java/org/apache/ambari/server/checks/LZOCheckTest.java e50e936 Diff: https://reviews.apache.org/r/64030/diff/2/ Changes: https://reviews.apache.org/r/64030/diff/1-2/ Testing --- mvn clean test and check on live cluster Thanks, Dmitro Lisnichenko
Review Request 64030: Modify AMBARI-22387 to Check for LZO + No Opt-in
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64030/ --- Review request for Ambari, Jonathan Hurley and Nate Cole. Bugs: AMBARI-22500 https://issues.apache.org/jira/browse/AMBARI-22500 Repository: ambari Description --- The work for AMBARI-22387 was initially created to warn the user before performing an upgrade that LZO was enabled and would need to be managed on their own. However, with AMBARI-22457, we now support LZO when a user opts-in to installed GPL-licensed code. The warning in AMBARI-22387 should be changed to check for LZO being enabled but the user _NOT_ opt-ing in Diffs - ambari-server/src/main/java/org/apache/ambari/server/checks/LZOCheck.java 9c0286b ambari-server/src/test/java/org/apache/ambari/server/checks/LZOCheckTest.java e50e936 Diff: https://reviews.apache.org/r/64030/diff/1/ Testing --- mvn clean test and check on live cluster Thanks, Dmitro Lisnichenko
Re: Review Request 63516: [Preview] Remove properties.json And Switch To Adding Properties to ResourceProviders Dynamically
/apache/ambari/server/controller/internal/RequestScheduleResourceProviderTest.java 6d4f72d9ad ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RootServiceComponentResourceProviderTest.java 222340b4e7 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RootServiceHostComponentResourceProviderTest.java ce3656c4ab ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RootServiceResourceProviderTest.java 96bc7c9674 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackAdvisorResourceProviderTest.java 05232eac49 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackArtifactResourceProviderTest.java 21df88d545 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackConfigurationDependencyResourceProviderTest.java 7c88852f48 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackConfigurationResourceProviderTest.java 9c48ad0d23 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackDependencyResourceProviderTest.java f35015348f ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackLevelConfigurationResourceProviderTest.java e9742bdfb1 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackResourceProviderTest.java 4281e3372e ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackServiceResourceProviderTest.java 2a2ef0b0b7 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/TargetClusterResourceProviderTest.java 492c012c7e ambari-server/src/test/java/org/apache/ambari/server/controller/internal/TaskAttemptResourceProviderTest.java 992bce0936 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/TaskResourceProviderTest.java 365d408ff6 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/TestIvoryProviderModule.java 90f10118d4 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java 64d416f252 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UserResourceProviderDBTest.java c4f0f349fb ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UserResourceProviderTest.java 9efad8a54d ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ValidationResourceProviderTest.java c639d1fcdc ambari-server/src/test/java/org/apache/ambari/server/controller/internal/WorkflowResourceProviderTest.java 337dc466da Diff: https://reviews.apache.org/r/63516/diff/3/ Changes: https://reviews.apache.org/r/63516/diff/2-3/ Testing --- tests do not pass yet, also on deploy get errors in some places. Fixing mistakes Thanks, Dmitro Lisnichenko
Re: Review Request 63961: Ambari upgrade failed
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63961/#review191606
---
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
Line 365 (original), 368 (patched)
<https://reviews.apache.org/r/63961/#comment269450>
currentRepoID null value as default column value for non-nullable column
works as a safeguard here. If table is not empty, the statement would fail.
Nate proposed to use -1 if not current cluster version exists. But if we
happen to have entries at SERVICE_DESIRED_STATE_TABLE or
SERVICE_COMPONENT_DESIRED_STATE_TABLE, that would populate their current
cluster version id with -1
Open for discussion
- Dmitro Lisnichenko
On Nov. 21, 2017, 7:56 p.m., Dmitro Lisnichenko wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63961/
> ---
>
> (Updated Nov. 21, 2017, 7:56 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley and Nate Cole.
>
>
> Bugs: AMBARI-22469
> https://issues.apache.org/jira/browse/AMBARI-22469
>
>
> Repository: ambari
>
>
> Description
> ---
>
> Ambari upgrade would fail for all Ambari view servers.
> Steps to reproduce:
> 1. Install Ambari 2.5.2 and setup it as view server. (it you don't set up it
> up as view server also it fails)
> 2. now install 2.6.0
> 3. run ambari-server upgrade
> it fails out with below exception.
>
> {noformat}
> ERROR: Error executing schema upgrade, please check the server logs.
> ERROR: Error output from schema upgrade command:
> ERROR: Exception in thread "main" org.apache.ambari.server.AmbariException:
> Unable to find any CURRENT repositories.
> at
> org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:203)
> at
> org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:418)
> Caused by: org.apache.ambari.server.AmbariException: Unable to find any
> CURRENT repositories.
> at
> org.apache.ambari.server.upgrade.UpgradeCatalog260.getCurrentVersionID(UpgradeCatalog260.java:510)
> at
> org.apache.ambari.server.upgrade.UpgradeCatalog260.executeDDLUpdates(UpgradeCatalog260.java:194)
> at
> org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:923)
> at
> org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:200)
> ... 1 more
>
>
> ERROR: Ambari server upgrade failed. Please look at
> /var/log/ambari-server/ambari-server.log, for more details.
> ERROR: Exiting with exit code 11.
> REASON: Schema upgrade failed.
> {noformat}
>
> For some reason we are checking cluster_version table entries and throwing up
> above exception.
>
>
> {noformat}
> In UpgradeCatalog260.java
> public int getCurrentVersionID() throws AmbariException, SQLException {
> List currentVersionList =
> dbAccessor.getIntColumnValues(CLUSTER_VERSION_TABLE, REPO_VERSION_ID_COLUMN,
> new String[]{STATE_COLUMN}, new String[]{CURRENT}, false);
> if (currentVersionList.isEmpty()) {
> throw new AmbariException("Unable to find any CURRENT repositories.");
> } else if (currentVersionList.size() != 1) {
> throw new AmbariException("The following repositories were found to be
> CURRENT: ".concat(StringUtils.join(currentVersionList, ",")));
> }
> return currentVersionList.get(0);
> }
> {noformat}
>
>
> Diffs
> -
>
>
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
> 96ce807d5f
>
>
> Diff: https://reviews.apache.org/r/63961/diff/2/
>
>
> Testing
> ---
>
> check on live cluster
> tests passed
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 63961: Ambari upgrade failed
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63961/
---
(Updated Nov. 21, 2017, 7:56 p.m.)
Review request for Ambari, Jonathan Hurley and Nate Cole.
Bugs: AMBARI-22469
https://issues.apache.org/jira/browse/AMBARI-22469
Repository: ambari
Description
---
Ambari upgrade would fail for all Ambari view servers.
Steps to reproduce:
1. Install Ambari 2.5.2 and setup it as view server. (it you don't set up it up
as view server also it fails)
2. now install 2.6.0
3. run ambari-server upgrade
it fails out with below exception.
{noformat}
ERROR: Error executing schema upgrade, please check the server logs.
ERROR: Error output from schema upgrade command:
ERROR: Exception in thread "main" org.apache.ambari.server.AmbariException:
Unable to find any CURRENT repositories.
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:203)
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:418)
Caused by: org.apache.ambari.server.AmbariException: Unable to find any CURRENT
repositories.
at
org.apache.ambari.server.upgrade.UpgradeCatalog260.getCurrentVersionID(UpgradeCatalog260.java:510)
at
org.apache.ambari.server.upgrade.UpgradeCatalog260.executeDDLUpdates(UpgradeCatalog260.java:194)
at
org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:923)
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:200)
... 1 more
ERROR: Ambari server upgrade failed. Please look at
/var/log/ambari-server/ambari-server.log, for more details.
ERROR: Exiting with exit code 11.
REASON: Schema upgrade failed.
{noformat}
For some reason we are checking cluster_version table entries and throwing up
above exception.
{noformat}
In UpgradeCatalog260.java
public int getCurrentVersionID() throws AmbariException, SQLException {
List currentVersionList =
dbAccessor.getIntColumnValues(CLUSTER_VERSION_TABLE, REPO_VERSION_ID_COLUMN,
new String[]{STATE_COLUMN}, new String[]{CURRENT}, false);
if (currentVersionList.isEmpty()) {
throw new AmbariException("Unable to find any CURRENT repositories.");
} else if (currentVersionList.size() != 1) {
throw new AmbariException("The following repositories were found to be
CURRENT: ".concat(StringUtils.join(currentVersionList, ",")));
}
return currentVersionList.get(0);
}
{noformat}
Diffs (updated)
-
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
96ce807d5f
Diff: https://reviews.apache.org/r/63961/diff/2/
Changes: https://reviews.apache.org/r/63961/diff/1-2/
Testing (updated)
---
check on live cluster
tests passed
Thanks,
Dmitro Lisnichenko
Re: Review Request 63961: Ambari upgrade failed
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63961/#review191536
---
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
Line 510 (original), 510 (patched)
<https://reviews.apache.org/r/63961/#comment269377>
I'm also considering changing this to
if (allVersionsList.isEmpty()) {
return null;
} else if (currentVersionList.isEmpty()) {
throw new AmbariException("Unable to find any CURRENT repositories.");
} else if (currentVersionList.size() != 1) {
throw new AmbariException("The following repositories were found to be
CURRENT: ".concat(StringUtils.join(currentVersionList, ",")));
}
- Dmitro Lisnichenko
On Nov. 20, 2017, 9:52 p.m., Dmitro Lisnichenko wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63961/
> ---
>
> (Updated Nov. 20, 2017, 9:52 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley and Nate Cole.
>
>
> Bugs: AMBARI-22469
> https://issues.apache.org/jira/browse/AMBARI-22469
>
>
> Repository: ambari
>
>
> Description
> ---
>
> Ambari upgrade would fail for all Ambari view servers.
> Steps to reproduce:
> 1. Install Ambari 2.5.2 and setup it as view server. (it you don't set up it
> up as view server also it fails)
> 2. now install 2.6.0
> 3. run ambari-server upgrade
> it fails out with below exception.
>
> {noformat}
> ERROR: Error executing schema upgrade, please check the server logs.
> ERROR: Error output from schema upgrade command:
> ERROR: Exception in thread "main" org.apache.ambari.server.AmbariException:
> Unable to find any CURRENT repositories.
> at
> org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:203)
> at
> org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:418)
> Caused by: org.apache.ambari.server.AmbariException: Unable to find any
> CURRENT repositories.
> at
> org.apache.ambari.server.upgrade.UpgradeCatalog260.getCurrentVersionID(UpgradeCatalog260.java:510)
> at
> org.apache.ambari.server.upgrade.UpgradeCatalog260.executeDDLUpdates(UpgradeCatalog260.java:194)
> at
> org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:923)
> at
> org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:200)
> ... 1 more
>
>
> ERROR: Ambari server upgrade failed. Please look at
> /var/log/ambari-server/ambari-server.log, for more details.
> ERROR: Exiting with exit code 11.
> REASON: Schema upgrade failed.
> {noformat}
>
> For some reason we are checking cluster_version table entries and throwing up
> above exception.
>
>
> {noformat}
> In UpgradeCatalog260.java
> public int getCurrentVersionID() throws AmbariException, SQLException {
> List currentVersionList =
> dbAccessor.getIntColumnValues(CLUSTER_VERSION_TABLE, REPO_VERSION_ID_COLUMN,
> new String[]{STATE_COLUMN}, new String[]{CURRENT}, false);
> if (currentVersionList.isEmpty()) {
> throw new AmbariException("Unable to find any CURRENT repositories.");
> } else if (currentVersionList.size() != 1) {
> throw new AmbariException("The following repositories were found to be
> CURRENT: ".concat(StringUtils.join(currentVersionList, ",")));
> }
> return currentVersionList.get(0);
> }
> {noformat}
>
>
> Diffs
> -
>
>
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
> 96ce807d5f
>
>
> Diff: https://reviews.apache.org/r/63961/diff/1/
>
>
> Testing
> ---
>
> pending check on live cluster
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Review Request 63961: Ambari upgrade failed
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63961/
---
Review request for Ambari, Jonathan Hurley and Nate Cole.
Bugs: AMBARI-22469
https://issues.apache.org/jira/browse/AMBARI-22469
Repository: ambari
Description
---
Ambari upgrade would fail for all Ambari view servers.
Steps to reproduce:
1. Install Ambari 2.5.2 and setup it as view server. (it you don't set up it up
as view server also it fails)
2. now install 2.6.0
3. run ambari-server upgrade
it fails out with below exception.
{noformat}
ERROR: Error executing schema upgrade, please check the server logs.
ERROR: Error output from schema upgrade command:
ERROR: Exception in thread "main" org.apache.ambari.server.AmbariException:
Unable to find any CURRENT repositories.
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:203)
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:418)
Caused by: org.apache.ambari.server.AmbariException: Unable to find any CURRENT
repositories.
at
org.apache.ambari.server.upgrade.UpgradeCatalog260.getCurrentVersionID(UpgradeCatalog260.java:510)
at
org.apache.ambari.server.upgrade.UpgradeCatalog260.executeDDLUpdates(UpgradeCatalog260.java:194)
at
org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:923)
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:200)
... 1 more
ERROR: Ambari server upgrade failed. Please look at
/var/log/ambari-server/ambari-server.log, for more details.
ERROR: Exiting with exit code 11.
REASON: Schema upgrade failed.
{noformat}
For some reason we are checking cluster_version table entries and throwing up
above exception.
{noformat}
In UpgradeCatalog260.java
public int getCurrentVersionID() throws AmbariException, SQLException {
List currentVersionList =
dbAccessor.getIntColumnValues(CLUSTER_VERSION_TABLE, REPO_VERSION_ID_COLUMN,
new String[]{STATE_COLUMN}, new String[]{CURRENT}, false);
if (currentVersionList.isEmpty()) {
throw new AmbariException("Unable to find any CURRENT repositories.");
} else if (currentVersionList.size() != 1) {
throw new AmbariException("The following repositories were found to be
CURRENT: ".concat(StringUtils.join(currentVersionList, ",")));
}
return currentVersionList.get(0);
}
{noformat}
Diffs
-
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
96ce807d5f
Diff: https://reviews.apache.org/r/63961/diff/1/
Testing
---
pending check on live cluster
Thanks,
Dmitro Lisnichenko
Re: Review Request 63880: VDF defined HDP-GPL repo should be tagged appropriately
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63880/#review191321 --- Ship it! Ship It! - Dmitro Lisnichenko On Nov. 16, 2017, 5:45 p.m., Nate Cole wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/63880/ > --- > > (Updated Nov. 16, 2017, 5:45 p.m.) > > > Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, and Jonathan > Hurley. > > > Bugs: AMBARI-22461 > https://issues.apache.org/jira/browse/AMBARI-22461 > > > Repository: ambari > > > Description > --- > > This adds tags to a repository for use by Ambari > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java > a6315f8350 > > ambari-server/src/main/java/org/apache/ambari/server/controller/RepositoryResponse.java > 5ec46f044f > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RepositoryResourceProvider.java > 3c10e43343 > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/VersionDefinitionResourceProvider.java > 3ca81d7648 > > ambari-server/src/main/java/org/apache/ambari/server/orm/entities/RepositoryEntity.java > 034f3bce6e > ambari-server/src/main/java/org/apache/ambari/server/stack/RepoUtil.java > 81b4da949f > > ambari-server/src/main/java/org/apache/ambari/server/state/RepositoryInfo.java > bd53c11bd7 > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/RepoTag.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/RepositoryXml.java > 79ace89380 > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/RepositoryVersionHelper.java > a77caab32d > ambari-server/src/main/resources/version_definition.xsd eaed31d582 > > ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RepositoryResourceProviderTest.java > c46cce708e > > ambari-server/src/test/java/org/apache/ambari/server/state/repository/VersionDefinitionTest.java > 6f33c7e8c1 > > ambari-server/src/test/java/org/apache/ambari/server/state/stack/upgrade/RepositoryVersionHelperTest.java > 70ff801eb3 > ambari-server/src/test/resources/version_definition_with_tags.xml > PRE-CREATION > contrib/version-builder/version_builder.py 5bf54303ed > > > Diff: https://reviews.apache.org/r/63880/diff/1/ > > > Testing > --- > > Manual. Unit tests: > > [INFO] Results: > [INFO] > [WARNING] Tests run: 4925, Failures: 0, Errors: 0, Skipped: 34 > [INFO] > [INFO] > > > Thanks, > > Nate Cole > >
Re: Review Request 63807: Add Native Libraries To Tez Tarball
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63807/#review191046
---
Ship it!
Ship It!
- Dmitro Lisnichenko
On Nov. 14, 2017, 11:21 p.m., Jonathan Hurley wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63807/
> ---
>
> (Updated Nov. 14, 2017, 11:21 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-22444
> https://issues.apache.org/jira/browse/AMBARI-22444
>
>
> Repository: ambari
>
>
> Description
> ---
>
> As part of the investigation for getting patch upgrades working in Ambari
> 2.6.1, it was determined that the Tez tarball will need to have the hadoop
> native libraries added to it so that they can be detected from the tarball.
>
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
>
> When Hive commands run, they will attempt to load the native snappy libraries
> from the Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec
> (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to
> missing native libraries for the codec.
> at
> org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:217)
> {code}
>
> In order to fix this, the Tez tarball should include the native hadoop
> libraries as well:
> {code}
> ??? tez
> ?  ??? lib
> ?  ?  ??? native
> ?  ?  ?  ??? libhadoop.a
> ?  ?  ?  ??? libhadoop.so -> libhadoop.so.1.0.0
> ?  ?  ?  ??? libhadoop.so.1.0.0
> ?  ?  ?  ??? libhadooppipes.a
> ?  ?  ?  ??? libhadooputils.a
> ?  ?  ?  ??? libhdfs.a
> ?  ?  ?  ??? libhdfs.so -> libhdfs.so.0.0.0
> ?  ?  ?  ??? libhdfs.so.0.0.0
> ?  ?  ?  ??? libsnappy.so -> libsnappy.so.1.1.4
> ?  ?  ?  ??? libsnappy.so.1 -> libsnappy.so.1.1.4
> ?  ?  ?  ??? libsnappy.so.1.1.4
> {code}
>
>
> Diffs
> -
>
>
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
> 03b6213ac4
>
> ambari-common/src/main/python/resource_management/libraries/functions/tar_archive.py
> c682c3e24a
>
>
> Diff: https://reviews.apache.org/r/63807/diff/1/
>
>
> Testing
> ---
>
> Manual testing to ensure the file is created and uploaded.
>
> UNIT TESTS PENDING...
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 63698: Implement many-to-many relation between keytabs and principals
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63698/#review190824 --- Ship it! Ship It! - Dmitro Lisnichenko On Nov. 13, 2017, 5:03 p.m., Eugene Chekanskiy wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/63698/ > --- > > (Updated Nov. 13, 2017, 5:03 p.m.) > > > Review request for Ambari, Attila Magyar, Dmitro Lisnichenko, and Robert > Levas. > > > Bugs: AMBARI-22390 > https://issues.apache.org/jira/browse/AMBARI-22390 > > > Repository: ambari > > > Description > --- > > Now it is possible to put mulitple different principals to same keytab: > > * copy keytap entry from existant identity: > 1. define principal with new unique name(identity1) and reference to > principal that you want to update(identity0) > 2. redefine principal record of identity > 3. Good luck, now principals from identity1 and identity0 will be located > in keytab file from identity0 > * just define new keytab entry in identity with same keytab file. If owners > are different for same keytab in different identities warning will be > printed, if owners and goups are different, or group does not have "r" > permission for file, error will be printed, so make sure that users that need > this keytab are in group that can access it > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java > f91383117f > > ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/AbstractPrepareKerberosServerAction.java > 1dc8ca8ec7 > > ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/CreatePrincipalsServerAction.java > 59d532753d > > ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/KerberosServerAction.java > 3491f18931 > > ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/stageutils/ResolvedKerberosKeytab.java > f66d273665 > > > Diff: https://reviews.apache.org/r/63698/diff/3/ > > > Testing > --- > > mvn clean test, cluster deploy > > > Thanks, > > Eugene Chekanskiy > >
Re: Review Request 63695: Create a Pre-Upgrade Check Warning About LZO
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63695/
---
(Updated Nov. 10, 2017, 7:47 p.m.)
Review request for Ambari, Jonathan Hurley and Nate Cole.
Changes
---
fixed review comments
Bugs: AMBARI-22387
https://issues.apache.org/jira/browse/AMBARI-22387
Repository: ambari
Description
---
Ambari has removed its native support of distributing and installing LZO when
the LZO codecs are enabled in {{core-site}}. For existing clusters where LZO is
enabled, this means that performing an upgrade will now require manual user
intervention to get the LZO packages installed.
A pre-upgrade check should be created which checks to see if LZO is enabled in
the cluster and then produces a {{WARNING}} to the user letting them know that
before upgrading, they'd need to distribute the appropriate LZO packages.
Diffs (updated)
-
ambari-server/src/main/java/org/apache/ambari/server/checks/CheckDescription.java
f5e4ff9061
ambari-server/src/main/java/org/apache/ambari/server/checks/LZOCheck.java
PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.6.xml
3d0d906856
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.6.xml
3e044ec02e
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.6.xml
c698d5e244
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.6.xml
08725f04c4
ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml
48be5d1971
ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml
cbe809a410
ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml
b4e3745099
ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml
98223904c8
ambari-server/src/test/java/org/apache/ambari/server/checks/LZOCheckTest.java
PRE-CREATION
Diff: https://reviews.apache.org/r/63695/diff/3/
Changes: https://reviews.apache.org/r/63695/diff/2-3/
Testing (updated)
---
mvn clean test
live cluster check
Thanks,
Dmitro Lisnichenko
Re: Review Request 63729: Blueprint deploys failing with missing smoke user keytab file
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63729/#review190722 --- Ship it! Ship It! - Dmitro Lisnichenko On Nov. 10, 2017, 5:21 p.m., Eugene Chekanskiy wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/63729/ > --- > > (Updated Nov. 10, 2017, 5:21 p.m.) > > > Review request for Ambari, Dmitro Lisnichenko and Robert Levas. > > > Bugs: AMBARI-22415 > https://issues.apache.org/jira/browse/AMBARI-22415 > > > Repository: ambari > > > Description > --- > > cache files respected by blueprint install > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/CreateKeytabFilesServerAction.java > aa65e61c67 > > > Diff: https://reviews.apache.org/r/63729/diff/1/ > > > Testing > --- > > mvn clean test > > > Thanks, > > Eugene Chekanskiy > >
Re: Review Request 63713: Restart all required services failed when running cluster with several patch upgrades applied
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63713/#review190701 --- Ship it! Ship It! - Dmitro Lisnichenko On Nov. 10, 2017, 3:43 a.m., Nate Cole wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/63713/ > --- > > (Updated Nov. 10, 2017, 3:43 a.m.) > > > Review request for Ambari, Dmitro Lisnichenko and Jonathan Hurley. > > > Bugs: AMBARI-22411 > https://issues.apache.org/jira/browse/AMBARI-22411 > > > Repository: ambari > > > Description > --- > > Small refactor to pass StackID instead of RepositoryVersionEntity, negating > the check that made this call fail. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java > d8712f2c75 > > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java > 5ea4bbdf3d > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java > 1ec790f7cc > > ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json > 23c6d32dff > > ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerImplTest.java > c80620f6a3 > > > Diff: https://reviews.apache.org/r/63713/diff/1/ > > > Testing > --- > > Manual. > Unit tests pending. Mostly verify that existing tests continue to work > without error. > > > Thanks, > > Nate Cole > >
Re: Review Request 63698: Implement many-to-many relation between keytabs and principals
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63698/#review190582 --- Ship it! Ship It! - Dmitro Lisnichenko On Nov. 9, 2017, 2:53 p.m., Eugene Chekanskiy wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/63698/ > --- > > (Updated Nov. 9, 2017, 2:53 p.m.) > > > Review request for Ambari, Attila Magyar, Dmitro Lisnichenko, and Robert > Levas. > > > Bugs: AMBARI-22390 > https://issues.apache.org/jira/browse/AMBARI-22390 > > > Repository: ambari > > > Description > --- > > Now it is possible to put mulitple different principals to same keytab: > > * copy keytap entry from existant identity: > 1. define principal with new unique name(identity1) and reference to > principal that you want to update(identity0) > 2. redefine principal record of identity > 3. Good luck, now principals from identity1 and identity0 will be located > in keytab file from identity0 > * just define new keytab entry in identity with same keytab file. If owners > are different for same keytab in different identities warning will be > printed, if owners and goups are different, or group does not have "r" > permission for file, error will be printed, so make sure that users that need > this keytab are in group that can access it > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java > f91383117f > > ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/AbstractPrepareKerberosServerAction.java > 1dc8ca8ec7 > > ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/CreatePrincipalsServerAction.java > 59d532753d > > ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/KerberosServerAction.java > 3491f18931 > > ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/stageutils/ResolvedKerberosKeytab.java > f66d273665 > ambari-server/src/main/resources/common-services/SPARK/1.2.1/kerberos.json > 166adbd7d0 > ambari-server/src/main/resources/common-services/SPARK/1.4.1/kerberos.json > f2dd9e7e3d > ambari-server/src/main/resources/common-services/SPARK/2.2.0/kerberos.json > bf763de6d9 > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/kerberos.json > 95d735b972 > > ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json > b4e93ddc77 > > ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json > 575b9fa42f > > ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK2/kerberos.json > 89f19d4927 > > > Diff: https://reviews.apache.org/r/63698/diff/1/ > > > Testing > --- > > mvn clean test, cluster deploy > > > Thanks, > > Eugene Chekanskiy > >
Re: Review Request 63516: [Preview] Remove properties.json And Switch To Adding Properties to ResourceProviders Dynamically
> On Nov. 3, 2017, 1:42 a.m., Nate Cole wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ActionResourceProvider.java
> > Lines 69-72 (patched)
> > <https://reviews.apache.org/r/63516/diff/1/?file=1879216#file1879216line71>
> >
> > I don't think we ever change this map do we? Can use
> > ImmutableMap.builder().put()...build()
> >
> > Or, simple ones like this can be Collections.singletonMap(key, value)
these changes would require a big patch. Should I fix all providers, or only
ones affected by a current patch?
> On Nov. 3, 2017, 1:42 a.m., Nate Cole wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ActionResourceProvider.java
> > Lines 77-87 (patched)
> > <https://reviews.apache.org/r/63516/diff/1/?file=1879216#file1879216line79>
> >
> > Recommend Sets.newHashSet(String...) - it's much cleaner looking and
> > avoids an added static{} block.
these changes would require a big patch. Should I fix all providers, or only
ones affected by a current patch?
- Dmitro
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63516/#review189988
---
On Nov. 9, 2017, 11:58 a.m., Dmitro Lisnichenko wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63516/
> ---
>
> (Updated Nov. 9, 2017, 11:58 a.m.)
>
>
> Review request for Ambari, Jonathan Hurley and Nate Cole.
>
>
> Bugs: AMBARI-22353
> https://issues.apache.org/jira/browse/AMBARI-22353
>
>
> Repository: ambari
>
>
> Description
> ---
>
> Legacy/ancient ResourceProviders use the {{properties.json}} file to govern
> which properties can be used with the provider. This seems like excessive
> decoupling without any benefit and usually leads to runtime errors when new
> or removed properties are forgotten.
>
> This file should be removed and the providers should be registering the known
> properties on their own.
>
>
> Diffs
> -
>
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/ResourceProviderFactory.java
> 711ae10f7e
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AbstractAuthorizedResourceProvider.java
> a24c6394ac
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AbstractControllerResourceProvider.java
> 1dc0841d19
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AbstractDRResourceProvider.java
> 953a2a7c77
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ActionResourceProvider.java
> 4e618e7ff6
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ActiveWidgetLayoutResourceProvider.java
> 389f0b2bf2
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AlertDefinitionResourceProvider.java
> 32f1fd8f4f
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AlertGroupResourceProvider.java
> cf4e7475b3
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AlertHistoryResourceProvider.java
> 8d716eabe0
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AlertNoticeResourceProvider.java
> 8ffd596cfa
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AlertResourceProvider.java
> c71651e5ff
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AlertTargetResourceProvider.java
> 43ee7fe7c7
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AmbariConfigurationResourceProvider.java
> 4f4cc7070d
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintResourceProvider.java
> 73d24908ea
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProvider.java
> a2a49d7cb5
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterKerberosDescriptorResourceProvider.java
> 2678d55975
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterResourceProvider.java
> 54c83600b7
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProvider.java
> 1a755590b7
>
> ambari-server/src/main/java/org/apache/a
Re: Review Request 63695: Create a Pre-Upgrade Check Warning About LZO
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63695/
---
(Updated Nov. 9, 2017, 12:15 p.m.)
Review request for Ambari, Jonathan Hurley and Nate Cole.
Bugs: AMBARI-22387
https://issues.apache.org/jira/browse/AMBARI-22387
Repository: ambari
Description
---
Ambari has removed its native support of distributing and installing LZO when
the LZO codecs are enabled in {{core-site}}. For existing clusters where LZO is
enabled, this means that performing an upgrade will now require manual user
intervention to get the LZO packages installed.
A pre-upgrade check should be created which checks to see if LZO is enabled in
the cluster and then produces a {{WARNING}} to the user letting them know that
before upgrading, they'd need to distribute the appropriate LZO packages.
Diffs (updated)
-
ambari-server/src/main/java/org/apache/ambari/server/checks/CheckDescription.java
f5e4ff9061
ambari-server/src/main/java/org/apache/ambari/server/checks/LZOCheck.java
PRE-CREATION
Diff: https://reviews.apache.org/r/63695/diff/2/
Changes: https://reviews.apache.org/r/63695/diff/1-2/
Testing
---
checking
Thanks,
Dmitro Lisnichenko
Review Request 63695: Create a Pre-Upgrade Check Warning About LZO
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63695/
---
Review request for Ambari, Jonathan Hurley and Nate Cole.
Bugs: AMBARI-22387
https://issues.apache.org/jira/browse/AMBARI-22387
Repository: ambari
Description
---
Ambari has removed its native support of distributing and installing LZO when
the LZO codecs are enabled in {{core-site}}. For existing clusters where LZO is
enabled, this means that performing an upgrade will now require manual user
intervention to get the LZO packages installed.
A pre-upgrade check should be created which checks to see if LZO is enabled in
the cluster and then produces a {{WARNING}} to the user letting them know that
before upgrading, they'd need to distribute the appropriate LZO packages.
Diffs
-
ambari-server/src/main/java/org/apache/ambari/server/checks/CheckDescription.java
f5e4ff9061
ambari-server/src/main/java/org/apache/ambari/server/checks/LZOCheck.java
PRE-CREATION
Diff: https://reviews.apache.org/r/63695/diff/1/
Testing
---
checking
Thanks,
Dmitro Lisnichenko
Re: Review Request 63516: [Preview] Remove properties.json And Switch To Adding Properties to ResourceProviders Dynamically
/java/org/apache/ambari/server/controller/internal/RootServiceHostComponentResourceProviderTest.java ce3656c4ab ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RootServiceResourceProviderTest.java 96bc7c9674 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackAdvisorResourceProviderTest.java 05232eac49 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackArtifactResourceProviderTest.java 21df88d545 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackConfigurationDependencyResourceProviderTest.java 7c88852f48 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackConfigurationResourceProviderTest.java 9c48ad0d23 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackDependencyResourceProviderTest.java f35015348f ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackLevelConfigurationResourceProviderTest.java e9742bdfb1 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackResourceProviderTest.java 4281e3372e ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackServiceResourceProviderTest.java 2a2ef0b0b7 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/TargetClusterResourceProviderTest.java 492c012c7e ambari-server/src/test/java/org/apache/ambari/server/controller/internal/TaskResourceProviderTest.java 365d408ff6 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/TestIvoryProviderModule.java 90f10118d4 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java d6b1ab3766 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UserResourceProviderDBTest.java c4f0f349fb ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UserResourceProviderTest.java 9efad8a54d ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ValidationResourceProviderTest.java c639d1fcdc ambari-server/src/test/java/org/apache/ambari/server/controller/internal/WorkflowResourceProviderTest.java 337dc466da Diff: https://reviews.apache.org/r/63516/diff/2/ Changes: https://reviews.apache.org/r/63516/diff/1-2/ Testing --- tests do not pass yet, also on deploy get errors in some places. Fixing mistakes Thanks, Dmitro Lisnichenko
Re: Review Request 63684: Patch Upgrades Broken For Clients Due To Versioned LD Library
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63684/#review190553
---
Ship it!
Ship It!
- Dmitro Lisnichenko
On Nov. 8, 2017, 11:22 p.m., Jonathan Hurley wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63684/
> ---
>
> (Updated Nov. 8, 2017, 11:22 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-22386
> https://issues.apache.org/jira/browse/AMBARI-22386
>
>
> Repository: ambari
>
>
> Description
> ---
>
> When services are patched to different versions, jobs being run in containers
> are unable to pickup the correct JARs from the {{LD_LIBRARY_PATH}} passed in
> via the job.
>
> Consider the following situation:
> - Base cluster installed at 2.6.3.0-1
> - Enable some sort of codec, like snappy
> - YARN patched to 2.6.3.1-1
> - Hive patched to 2.6.3.2-1
>
> When Hive goes to run jobs on Tez or when jobs are run on MapR, whatever is
> sending the job is resolving the {{hdp.version}} to its own version. The box
> that the job runs on, however, might not have that specific versioned
> directory.
>
> The problem can only be solved externally by providing a way to ensure that
> the native libraries are on every machine which could potentially run jobs.
> We could install bits on every host, but that would defeat the space-saving
> purposes of patch upgrades.
>
> Another solution is to ensure that the Tez and MapR tarballs have the
> required libraries and that those libraries are added to the classpath of
> launched jobs.
>
>
> Diffs
> -
>
>
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/configuration-mapred/mapred-site.xml
> a7d8cd67bd
>
> ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml
> 1427a6faae
>
> ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/configuration-mapred/mapred-site.xml
> 4ad08cebf9
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml
> 0df0334c53
>
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml
> b4e3745099
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml
> 98223904c8
>
>
> Diff: https://reviews.apache.org/r/63684/diff/2/
>
>
> Testing
> ---
>
> Manual upgrade on patched cluster.
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 63671: Remove Auto-Installation of LZO Libraries
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63671/#review190464 --- Ship it! Ship It! - Dmitro Lisnichenko On Nov. 8, 2017, 7:08 p.m., Nate Cole wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/63671/ > --- > > (Updated Nov. 8, 2017, 7:08 p.m.) > > > Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, and Sid Wagle. > > > Bugs: AMBARI-22383 > https://issues.apache.org/jira/browse/AMBARI-22383 > > > Repository: ambari > > > Description > --- > > The LZO library licenses are not compatible with Ambari's license and should > not be assumed to be available in Apache-based repositories. > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/__init__.py > f144b2da32 > > ambari-common/src/main/python/resource_management/libraries/functions/get_lzo_packages.py > cfbb7d81e8 > > ambari-common/src/main/python/resource_management/libraries/functions/package_conditions.py > 31e78b9e60 > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml > 2df1b9f647 > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py > 8bfa7e63e9 > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs.py > 6f702d3616 > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_client.py > 9c4f9d4493 > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/install_params.py > fe488c3004 > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/journalnode.py > 25534e9ff5 > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py > 89719cff2c > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py > 5ca012b796 > > ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py > 7d22603184 > > ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py > eb57c22834 > > ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py > cd1f7ad36f > > ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py > ed3f5fd387 > ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py > 0be9a34ab0 > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/metainfo.xml > 8500839692 > > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/package/scripts/params.py > 3a711c721a > > ambari-server/src/main/resources/stacks/BigInsights/4.0/services/HDFS/metainfo.xml > 918cdb32fa > > ambari-server/src/main/resources/stacks/BigInsights/4.0/services/HDFS/package/scripts/params.py > 1f76f8033a > > ambari-server/src/main/resources/stacks/BigInsights/4.0/services/OOZIE/package/scripts/oozie.py > 6cd2b71f8e > > ambari-server/src/main/resources/stacks/BigInsights/4.0/services/OOZIE/package/scripts/params.py > febd3e6580 > > ambari-server/src/main/resources/stacks/BigInsights/4.1/services/HDFS/metainfo.xml > 4c00b4f3f8 > > ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/HDFS/metainfo.xml > fc7c324ff6 > > ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/HDFS/package/scripts/hdfs.py > 1264284cd4 > > ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/HDFS/package/scripts/install_params.py > da8de46165 > > ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/HDFS/package/scripts/params_linux.py > a77ec82cdc > > ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/OOZIE/package/scripts/oozie.py > 33a309fa2e > > ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/OOZIE/package/scripts/params_linux.py > 0e6179b8a7 > > ambari-server/src/main/resources/stacks/BigInsights/4.2/services/HDFS/metainfo.xml > 72ce2be363 > > ambari-server/src/main/resources/stacks/BigInsights/4.2/services/HDFS/package/scripts/hdfs.py > 2b158bd128 > > ambari-server/src/main/resources/stacks/BigInsights/4.2/services/HDFS/package/scripts/install_params.py
Re: Review Request 63595: Remove HADOOP_HOME From Environment For Daemons
> On Nov. 7, 2017, 1:10 p.m., Dmitro Lisnichenko wrote:
> > ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
> > Line 119 (original), 119 (patched)
> > <https://reviews.apache.org/r/63595/diff/1/?file=1882235#file1882235line119>
> >
> >
will it be a case during upgrade/downgrade: hadoop_conf_dir =
format('{stack_root}/current/hadoop-client/conf')?
- Dmitro
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63595/#review190288
---
On Nov. 6, 2017, 11:17 p.m., Jonathan Hurley wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63595/
> ---
>
> (Updated Nov. 6, 2017, 11:17 p.m.)
>
>
> Review request for Ambari, Dmitro Lisnichenko and Nate Cole.
>
>
> Bugs: AMBARI-22370
> https://issues.apache.org/jira/browse/AMBARI-22370
>
>
> Repository: ambari
>
>
> Description
> ---
>
> The wrapper scripts for services should be setting {{HADOOP_HOME}} correctly
> regardless of whether or not an upgrade is taking place. There are many spots
> which we current override this value and pass it in:
>
> - Either via the Python files and setting the {{environment}}
> - Or by setting it in configuration files
>
> This should be removed since we must rely on correct {{HADOOP_HOME}} to be
> set by the wrapper scripts.
>
>
> Diffs
> -
>
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
> d8f3d374f5
>
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
> f6f678fa89
>
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon.py
> 933515b01b
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service_interactive.py
> eaf95ad5db
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/webhcat_service.py
> 646090ac27
>
> ambari-server/src/main/resources/common-services/MAHOUT/1.0.0.2.3/package/scripts/service_check.py
> a79b593ce8
>
> ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py
> 400c87c828
>
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py
> 4d636855c3
>
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
> 12edc6924f
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_webhcat_server.py
> f9480eeaff
> ambari-server/src/test/python/stacks/2.1/FALCON/test_falcon_server.py
> 7f2ed464dc
> ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py fad99f6496
>
> ambari-server/src/test/python/stacks/2.3/MAHOUT/test_mahout_service_check.py
> b80476cab2
> ambari-server/src/test/python/stacks/2.5/HIVE/test_hive_server_int.py
> 4951c7e2ec
>
>
> Diff: https://reviews.apache.org/r/63595/diff/1/
>
>
> Testing
> ---
>
> OK
> --
> Total run:1193
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 63595: Remove HADOOP_HOME From Environment For Daemons
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63595/#review190288
---
Fix it, then Ship it!
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
Line 119 (original), 119 (patched)
<https://reviews.apache.org/r/63595/#comment267580>
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/webhcat_service.py
Line 107 (original)
<https://reviews.apache.org/r/63595/#comment267581>
is it intended to remove env passed here?
- Dmitro Lisnichenko
On Nov. 6, 2017, 11:17 p.m., Jonathan Hurley wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63595/
> ---
>
> (Updated Nov. 6, 2017, 11:17 p.m.)
>
>
> Review request for Ambari, Dmitro Lisnichenko and Nate Cole.
>
>
> Bugs: AMBARI-22370
> https://issues.apache.org/jira/browse/AMBARI-22370
>
>
> Repository: ambari
>
>
> Description
> ---
>
> The wrapper scripts for services should be setting {{HADOOP_HOME}} correctly
> regardless of whether or not an upgrade is taking place. There are many spots
> which we current override this value and pass it in:
>
> - Either via the Python files and setting the {{environment}}
> - Or by setting it in configuration files
>
> This should be removed since we must rely on correct {{HADOOP_HOME}} to be
> set by the wrapper scripts.
>
>
> Diffs
> -
>
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
> d8f3d374f5
>
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
> f6f678fa89
>
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon.py
> 933515b01b
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service_interactive.py
> eaf95ad5db
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/webhcat_service.py
> 646090ac27
>
> ambari-server/src/main/resources/common-services/MAHOUT/1.0.0.2.3/package/scripts/service_check.py
> a79b593ce8
>
> ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py
> 400c87c828
>
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py
> 4d636855c3
>
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
> 12edc6924f
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_webhcat_server.py
> f9480eeaff
> ambari-server/src/test/python/stacks/2.1/FALCON/test_falcon_server.py
> 7f2ed464dc
> ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py fad99f6496
>
> ambari-server/src/test/python/stacks/2.3/MAHOUT/test_mahout_service_check.py
> b80476cab2
> ambari-server/src/test/python/stacks/2.5/HIVE/test_hive_server_int.py
> 4951c7e2ec
>
>
> Diff: https://reviews.apache.org/r/63595/diff/1/
>
>
> Testing
> ---
>
> OK
> --
> Total run:1193
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 63558: AMBARI-22361: Fix bug in base_alert when matching hostnames.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63558/#review190149
---
Ship it!
Ship It!
- Dmitro Lisnichenko
On Nov. 4, 2017, 11:42 a.m., stephane san wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63558/
> ---
>
> (Updated Nov. 4, 2017, 11:42 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk and Dmitro Lisnichenko.
>
>
> Bugs: https://issues.apache.org/jira/browse/AMBARI-22361
>
> https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/AMBARI-22361
>
>
> Repository: ambari
>
>
> Description
> ---
>
> In the ambari console, a critical alert gets reported for the NameNode Web UI
> due to mixed cases in hostname
>
> Error at hand:
> Connection failed to http://vz-sl-upupup-8724-hadoop-mgr-1:0 ( [Errno 111] Connection refused>)
>
> Issue:
> While in HA mode the hdfs-site property
> dfs.namenode.http-address.{{ha-nameservice}}.{{nn_id}}
> may have upper case parts, which prevents matching the hostname.
> The matching should be case insensitive and so both will be
> converted to lower case before the matching is performed.
>
> Fix:
> lower case before doing the comparison
> (samilar resolution as in https://issues.apache.org/jira/browse/AMBARI-19282)
>
>
> Diffs
> -
>
> ambari-agent/src/main/python/ambari_agent/alerts/base_alert.py 05f80234d0
>
>
> Diff: https://reviews.apache.org/r/63558/diff/1/
>
>
> Testing
> ---
>
> Verified the ambari console doesn't report the critical alert any more.
> (the ambari console reports http return code 200 when testing the NameNode
> Web UI access).
>
>
> Thanks,
>
> stephane san
>
>
Re: Review Request 63535: Send ClusterSummary to install_packages and conf-select exclusion
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63535/#review190052
---
Ship it!
Ship It!
- Dmitro Lisnichenko
On Nov. 3, 2017, 4:59 a.m., Nate Cole wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63535/
> ---
>
> (Updated Nov. 3, 2017, 4:59 a.m.)
>
>
> Review request for Ambari, Dmitro Lisnichenko and Jonathan Hurley.
>
>
> Bugs: AMBARI-22360
> https://issues.apache.org/jira/browse/AMBARI-22360
>
>
> Repository: ambari
>
>
> Description
> ---
>
> Pass the ClusterVersionSummary object with install_packages.
>
> - {{conf-select}} should be called as normal for {{STANDARD}} upgrades and
> component installations / host installs
> - {{conf-select}} should only be invoked for the services involved in a
> {{PATCH}}/{{MAINT}} upgrade
> -- The exception here is that hadoop shared components (HDFS, YARN, MapR) can
> never be {{conf-select}}'d in this case.
>
>
> Diffs
> -
>
>
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
> c89e7675a4
>
> ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
> b3f91304c8
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProvider.java
> a20658304f
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostStackVersionResourceProvider.java
> b68dc22093
>
> ambari-server/src/main/java/org/apache/ambari/server/state/repository/ClusterVersionSummary.java
> e9d99208e5
>
> ambari-server/src/main/java/org/apache/ambari/server/state/repository/ServiceVersionSummary.java
> 29505c832e
>
> ambari-server/src/main/java/org/apache/ambari/server/state/repository/VersionDefinitionXml.java
> 7944de8e8c
>
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/RepositoryVersionHelper.java
> 19019502dd
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py
> 8ba22d263b
>
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
> 20b12a97e5
>
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProviderTest.java
> 51979f6dee
> ambari-server/src/test/python/stacks/2.2/common/test_conf_select.py
> 92dd634cca
>
>
> Diff: https://reviews.apache.org/r/63535/diff/1/
>
>
> Testing
> ---
>
> Manual. Unit tests:
>
> [INFO] Results:
> [INFO]
> [WARNING] Tests run: 4922, Failures: 0, Errors: 0, Skipped: 34
>
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 24:46.046s
> [INFO] Finished at: Thu Nov 02 23:22:42 EDT 2017
> [INFO] Final Memory: 85M/1940M
> [INFO]
>
>
>
> Thanks,
>
> Nate Cole
>
>
Re: Review Request 63537: Specify the Correct HIVE_BIN In Hive Scripts
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63537/#review190035
---
Ship it!
Ship It!
- Dmitro Lisnichenko
On Nov. 3, 2017, 4:10 p.m., Jonathan Hurley wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63537/
> ---
>
> (Updated Nov. 3, 2017, 4:10 p.m.)
>
>
> Review request for Ambari, Dmitro Lisnichenko and Nate Cole.
>
>
> Bugs: AMBARI-22362
> https://issues.apache.org/jira/browse/AMBARI-22362
>
>
> Repository: ambari
>
>
> Description
> ---
>
> The Hive scripts pass in a parameter called {{HIVE_BIN}}. Hive uses this to
> determine where {{hdp.version}} and {{HADOOP_HOME}} should be sourced from.
> In many cases, the binary chosen is:
>
> {{/usr/bin/hive -> /usr/hdp/current/hive-client}}
>
> This can lead to problems during an upgrade scenario where {{hive-client}}
> has not yet moved forward. It should be chosen based on the component being
> restarted...
>
>
> Diffs
> -
>
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/files/startMetastore.sh
> 86541f07e1
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py
> 3c4255975b
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
> 9c6db006c8
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/service_check.py
> db253d3eed
>
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/HIVE/package/files/startMetastore.sh
> 86541f07e1
>
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/HIVE/package/scripts/hive_service.py
> c3bf30c9ab
>
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/HIVE/package/scripts/params.py
> 8f2132aa76
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/HIVE/package/files/startMetastore.sh
> 86541f07e1
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/HIVE/package/scripts/hive_service.py
> b5640572b3
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/HIVE/package/scripts/params_linux.py
> 528760a3b5
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_metastore.py
> 452c0b61ae
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py
> fd92277c3b
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_service_check.py
> 3421189f83
> ambari-server/src/test/python/stacks/2.1/HIVE/test_hive_metastore.py
> e497f3358c
>
>
> Diff: https://reviews.apache.org/r/63537/diff/1/
>
>
> Testing
> ---
>
> Manual deploy of Hive to verify Metastore starts using the right component
>
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 59.066 s
> [INFO] Finished at: 2017-11-03T10:09:14-04:00
> [INFO] Final Memory: 21M/619M
> [INFO]
>
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Review Request 63516: [Preview] Remove properties.json And Switch To Adding Properties to ResourceProviders Dynamically
/ambari/server/controller/internal/StackAdvisorResourceProviderTest.java 05232eac49 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackArtifactResourceProviderTest.java 21df88d545 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackConfigurationDependencyResourceProviderTest.java 7c88852f48 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackConfigurationResourceProviderTest.java 9c48ad0d23 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackDependencyResourceProviderTest.java f35015348f ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackLevelConfigurationResourceProviderTest.java e9742bdfb1 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackResourceProviderTest.java 4281e3372e ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackServiceResourceProviderTest.java 2a2ef0b0b7 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/TargetClusterResourceProviderTest.java 492c012c7e ambari-server/src/test/java/org/apache/ambari/server/controller/internal/TaskResourceProviderTest.java 365d408ff6 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/TestIvoryProviderModule.java 90f10118d4 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java d6b1ab3766 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UserResourceProviderDBTest.java c4f0f349fb ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UserResourceProviderTest.java 9efad8a54d ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ValidationResourceProviderTest.java c639d1fcdc Diff: https://reviews.apache.org/r/63516/diff/1/ Testing --- tests do not pass yet, also on deploy get errors in some places. Fixing mistakes Thanks, Dmitro Lisnichenko
Re: Review Request 63150: Ambari Schema Upgrade Failed during Ambari Upgrade (configuration: 2.5.2.0 -> 2.6.0.0 upgrade)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63150/ --- (Updated Oct. 19, 2017, 6:57 p.m.) Review request for Ambari, Dmytro Grinenko, Jonathan Hurley, and Nate Cole. Summary (updated) - Ambari Schema Upgrade Failed during Ambari Upgrade (configuration: 2.5.2.0 -> 2.6.0.0 upgrade) Bugs: AMBARI-22216 https://issues.apache.org/jira/browse/AMBARI-22216 Repository: ambari Description --- Yet another failure when upgrading 2.5.2.0 to 2.6.0.0 ERROR: Exception in thread "main" org.apache.ambari.server.AmbariException: Cannot drop index 'FK_upgrade_from_repo_version_id': needed in a foreign key constraint at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:203) at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:418) Caused by: java.sql.SQLException: Cannot drop index 'FK_upgrade_from_repo_version_id': needed in a foreign key constraint at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:996) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2526) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2484) at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:848) at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:742) at org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:877) at org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:869) at org.apache.ambari.server.orm.DBAccessorImpl.dropColumn(DBAccessorImpl.java:974) at org.apache.ambari.server.upgrade.UpgradeCatalog260.updateUpgradeTable(UpgradeCatalog260.java:333) at org.apache.ambari.server.upgrade.UpgradeCatalog260.executeDDLUpdates(UpgradeCatalog260.java:200) at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:923) at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:200) ... 1 more Diffs - ambari-server/src/main/java/org/apache/ambari/server/orm/DBAccessorImpl.java 09316be8b3 ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java 6f379858c0 Diff: https://reviews.apache.org/r/63150/diff/1/ Testing --- Live cluster check passed Tests passed Thanks, Dmitro Lisnichenko
Re: Review Request 63150: Ambari Schema Upgrade Failed during Ambari Upgrade
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63150/ --- (Updated Oct. 19, 2017, 6:38 p.m.) Review request for Ambari, Jonathan Hurley and Nate Cole. Bugs: AMBARI-22216 https://issues.apache.org/jira/browse/AMBARI-22216 Repository: ambari Description --- Yet another failure when upgrading 2.5.2.0 to 2.6.0.0 ERROR: Exception in thread "main" org.apache.ambari.server.AmbariException: Cannot drop index 'FK_upgrade_from_repo_version_id': needed in a foreign key constraint at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:203) at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:418) Caused by: java.sql.SQLException: Cannot drop index 'FK_upgrade_from_repo_version_id': needed in a foreign key constraint at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:996) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2526) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2484) at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:848) at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:742) at org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:877) at org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:869) at org.apache.ambari.server.orm.DBAccessorImpl.dropColumn(DBAccessorImpl.java:974) at org.apache.ambari.server.upgrade.UpgradeCatalog260.updateUpgradeTable(UpgradeCatalog260.java:333) at org.apache.ambari.server.upgrade.UpgradeCatalog260.executeDDLUpdates(UpgradeCatalog260.java:200) at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:923) at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:200) ... 1 more Diffs - ambari-server/src/main/java/org/apache/ambari/server/orm/DBAccessorImpl.java 09316be8b3 ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java 6f379858c0 Diff: https://reviews.apache.org/r/63150/diff/1/ Testing (updated) --- Live cluster check passed Tests passed Thanks, Dmitro Lisnichenko
Re: Review Request 63150: Ambari Schema Upgrade Failed during Ambari Upgrade
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63150/ --- (Updated Oct. 19, 2017, 6:10 p.m.) Review request for Ambari, Jonathan Hurley and Nate Cole. Bugs: AMBARI-22216 https://issues.apache.org/jira/browse/AMBARI-22216 Repository: ambari Description --- Yet another failure when upgrading 2.5.2.0 to 2.6.0.0 ERROR: Exception in thread "main" org.apache.ambari.server.AmbariException: Cannot drop index 'FK_upgrade_from_repo_version_id': needed in a foreign key constraint at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:203) at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:418) Caused by: java.sql.SQLException: Cannot drop index 'FK_upgrade_from_repo_version_id': needed in a foreign key constraint at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:996) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2526) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2484) at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:848) at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:742) at org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:877) at org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:869) at org.apache.ambari.server.orm.DBAccessorImpl.dropColumn(DBAccessorImpl.java:974) at org.apache.ambari.server.upgrade.UpgradeCatalog260.updateUpgradeTable(UpgradeCatalog260.java:333) at org.apache.ambari.server.upgrade.UpgradeCatalog260.executeDDLUpdates(UpgradeCatalog260.java:200) at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:923) at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:200) ... 1 more Diffs - ambari-server/src/main/java/org/apache/ambari/server/orm/DBAccessorImpl.java 09316be8b3 ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java 6f379858c0 Diff: https://reviews.apache.org/r/63150/diff/1/ Testing (updated) --- Live cluster check passed Tests are running Thanks, Dmitro Lisnichenko
Review Request 63150: Ambari Schema Upgrade Failed during Ambari Upgrade
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63150/ --- Review request for Ambari, Jonathan Hurley and Nate Cole. Bugs: AMBARI-22216 https://issues.apache.org/jira/browse/AMBARI-22216 Repository: ambari Description --- Yet another failure when upgrading 2.5.2.0 to 2.6.0.0 ERROR: Exception in thread "main" org.apache.ambari.server.AmbariException: Cannot drop index 'FK_upgrade_from_repo_version_id': needed in a foreign key constraint at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:203) at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:418) Caused by: java.sql.SQLException: Cannot drop index 'FK_upgrade_from_repo_version_id': needed in a foreign key constraint at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:996) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2526) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2484) at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:848) at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:742) at org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:877) at org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:869) at org.apache.ambari.server.orm.DBAccessorImpl.dropColumn(DBAccessorImpl.java:974) at org.apache.ambari.server.upgrade.UpgradeCatalog260.updateUpgradeTable(UpgradeCatalog260.java:333) at org.apache.ambari.server.upgrade.UpgradeCatalog260.executeDDLUpdates(UpgradeCatalog260.java:200) at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:923) at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:200) ... 1 more Diffs - ambari-server/src/main/java/org/apache/ambari/server/orm/DBAccessorImpl.java 09316be8b3 ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java 6f379858c0 Diff: https://reviews.apache.org/r/63150/diff/1/ Testing --- Tests are running Thanks, Dmitro Lisnichenko
Re: Review Request 63075: RU: RU failed on "Updating configuration sqoop-atlas-application.properties"
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63075/#review188330 --- Ship it! Ship It! - Dmitro Lisnichenko On Oct. 17, 2017, 5:22 p.m., Nate Cole wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/63075/ > --- > > (Updated Oct. 17, 2017, 5:22 p.m.) > > > Review request for Ambari, Dmitro Lisnichenko and Jonathan Hurley. > > > Bugs: AMBARI-22254 > https://issues.apache.org/jira/browse/AMBARI-22254 > > > Repository: ambari > > > Description > --- > > Config types are associated with a service. As a fallback, use the enclosed > config-task's service as the source of the service name. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/ConfigureAction.java > 5c65911a9e > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/UpgradePack.java > d8676bdb58 > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java > 7ad02579e8 > > ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ConfigureTask.java > 9f4440f055 > > ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java > d789ab45d9 > > ambari-server/src/test/java/org/apache/ambari/server/state/stack/ConfigUpgradeValidityTest.java > 44f72bd72a > > > Diff: https://reviews.apache.org/r/63075/diff/1/ > > > Testing > --- > > Manual. Unit tests: > > [WARNING] Tests run: 4921, Failures: 0, Errors: 0, Skipped: 34 > [INFO] > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 24:01.536s > [INFO] Finished at: Tue Oct 17 10:47:00 EDT 2017 > [INFO] Final Memory: 99M/1912M > [INFO] > > > > Thanks, > > Nate Cole > >
Re: Review Request 63032: YARN Service Checks Fails Because of Old hadoop-client Classpath Entry
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63032/#review188151
---
Ship it!
Ship It!
- Dmitro Lisnichenko
On Oct. 16, 2017, 7:22 p.m., Jonathan Hurley wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63032/
> ---
>
> (Updated Oct. 16, 2017, 7:22 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-22245
> https://issues.apache.org/jira/browse/AMBARI-22245
>
>
> Repository: ambari
>
>
> Description
> ---
>
> STR:
> - Install basic YARN on HDP 2.6.0.0 (no Tez)
> - Upgrade just YARN to HDP 2.6.0.2
>
> Service checks will fail with the following:
> ```
> 17/10/12 17:55:20 FATAL distributedshell.ApplicationMaster: Error running
> ApplicationMaster
> java.lang.NoSuchMethodError:
> org.apache.hadoop.io.retry.RetryPolicies.retryForeverWithFixedSleep(JLjava/util/concurrent/TimeUnit;)Lorg/apache/hadoop/io/retry/RetryPolicy;
> at
> org.apache.hadoop.yarn.client.RMProxy.createRetryPolicy(RMProxy.java:280)
> ```
>
> - All YARN daemons and clients are reporting 2.6.0.2
> - All YARN daemons have loaded JARs for 2.6.0.2
>
> What is happening here is that the applications being run on YARN are picking
> up the older hadoop-common JAR file. The method
> {{retryForeverWithFixedSleep}} did not exist in HDP 2.6.0.0.
>
> It is picking up the older JARs for running applications because of the
> {{yarn-site.xml}} property:
> ```
> yarn-site.xml- yarn.application.classpath
> yarn-site.xml:
> /etc/hadoop/conf,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*,/usr/hdp/current/ext/hadoop/*
> ```
>
> Ambari should be parameterizing the {{/usr/hdp/current/hadoop-client}} paths
> here.
>
>
> Diffs
> -
>
>
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
> f21719261a
>
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/status_params.py
> c2e9d92a04
>
> ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/configuration/yarn-site.xml
> d0b4bb1f0f
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml
> 5dbcf2da1b
>
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml
> b7548eab30
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml
> d3a096a4fa
>
>
> Diff: https://reviews.apache.org/r/63032/diff/1/
>
>
> Testing
> ---
>
> Manual
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 62996: Restart of random service could fail during express downgrade
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62996/#review188111 --- Ship it! Ship It! - Dmitro Lisnichenko On Oct. 16, 2017, 4:53 a.m., Nate Cole wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62996/ > --- > > (Updated Oct. 16, 2017, 4:53 a.m.) > > > Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Jonathan > Hurley. > > > Bugs: AMBARI-22240 > https://issues.apache.org/jira/browse/AMBARI-22240 > > > Repository: ambari > > > Description > --- > > Turns out when we did the import for PU, we neglected to bring over the > per-service config changes that are required to support it. This didn't show > during all our development because we were only checking on the same stack. > > This jira brings in those changes such that we are correctly creating and > reverting configs when crossing stack versions. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java > 91bfe09deb > > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java > 18659d44a1 > > ambari-server/src/main/java/org/apache/ambari/server/state/ConfigFactory.java > d6cd99786c > > ambari-server/src/main/java/org/apache/ambari/server/state/ConfigHelper.java > 987ff3818d > ambari-server/src/main/java/org/apache/ambari/server/state/ConfigImpl.java > 0a861d80c9 > > ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java > 77e2d47d09 > > ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java > 6bde42c10b > > ambari-server/src/main/java/org/apache/ambari/server/state/configgroup/ConfigGroupImpl.java > 1a8b5b61f2 > > ambari-server/src/main/java/org/apache/ambari/server/upgrade/AbstractUpgradeCatalog.java > d7cd087001 > > ambari-server/src/test/java/org/apache/ambari/server/actionmanager/ExecutionCommandWrapperTest.java > 87721e4ccd > > ambari-server/src/test/java/org/apache/ambari/server/agent/HeartbeatTestHelper.java > 76f4bb30d9 > > ambari-server/src/test/java/org/apache/ambari/server/agent/TestHeartbeatMonitor.java > 088ec8ddd2 > > ambari-server/src/test/java/org/apache/ambari/server/upgrade/AbstractUpgradeCatalogTest.java > 96ccc57f39 > > ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog250Test.java > 2590d1c995 > > ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog252Test.java > e6dbb7c0cf > > ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog260Test.java > 23962b7d79 > > > Diff: https://reviews.apache.org/r/62996/diff/2/ > > > Testing > --- > > Manual IN PROGRESS: > RU Full through Finalize (DONE) > RU Full but Downgrade (DONE) > EU Full through Finalize (DONE) > EU Full but Downgrade (DONE) > > EU Patch but Downgrade (DONE) > EU Patch then Finalize (DONE) > EU Patch then Finalize then Revert (DONE) > > PENDING Unit tests: > Verify no tests are broken with this change: > > > Thanks, > > Nate Cole > >
Re: Review Request 62949: RU: Downgrade is failed ZKFC
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62949/#review187941 --- Ship it! Ship It! - Dmitro Lisnichenko On Oct. 12, 2017, 11:46 p.m., Nate Cole wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62949/ > --- > > (Updated Oct. 12, 2017, 11:46 p.m.) > > > Review request for Ambari, Dmitro Lisnichenko and Jonathan Hurley. > > > Bugs: AMBARI-6 > https://issues.apache.org/jira/browse/AMBARI-6 > > > Repository: ambari > > > Description > --- > > The code to check versions was only looking at the upgrade context repo > version, not the target service repo > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/serveraction/ServerActionExecutor.java > f80d06accc > > ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/FinalizeUpgradeAction.java > 0ce5d30b0b > > > Diff: https://reviews.apache.org/r/62949/diff/1/ > > > Testing > --- > > No new tests, just verify existing tests pass. > Manual. Unit tests: > > [INFO] > [INFO] Results: > [INFO] > [WARNING] Tests run: 4919, Failures: 0, Errors: 0, Skipped: 34 > [INFO] > > > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 26:49.685s > [INFO] Finished at: Thu Oct 12 17:00:33 EDT 2017 > [INFO] Final Memory: 108M/1943M > [INFO] > > > > Thanks, > > Nate Cole > >
Re: Review Request 62931: Ambari Schema Upgrade Failed during Ambari Upgrade (patch with workaround)
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62931/
---
(Updated Oct. 12, 2017, 6:36 p.m.)
Review request for Ambari, Jonathan Hurley and Nate Cole.
Bugs: AMBARI-22216
https://issues.apache.org/jira/browse/AMBARI-22216
Repository: ambari
Description
---
For now, patch with workaround for issue on QA cluster.
Ambari Schema Upgrade Failed, Upgrade from : 2510 to 2600 (Also seen from 2430
to 2600)
ambari-server.log
{code}
03 Oct 2017 06:30:26,349 INFO [main] DBAccessorImpl:874 - Executing query:
ALTER TABLE servicecomponentdesiredstate ADD desired_repo_version_id BIGINT NOT
NULL DEFAULT 1
03 Oct 2017 06:30:26,389 INFO [main] DBAccessorImpl:874 - Executing query:
ALTER TABLE servicecomponentdesiredstate MODIFY desired_repo_version_id BIGINT
03 Oct 2017 06:30:26,435 INFO [main] DBAccessorImpl:874 - Executing query:
ALTER TABLE servicecomponentdesiredstate MODIFY desired_repo_version_id BIGINT
NOT NULL
03 Oct 2017 06:30:26,490 INFO [main] DBAccessorImpl:874 - Executing query:
ALTER TABLE servicecomponentdesiredstate ADD repo_state VARCHAR(255) NOT NULL
DEFAULT 'CURRENT'
03 Oct 2017 06:30:26,549 INFO [main] DBAccessorImpl:874 - Executing query:
ALTER TABLE servicecomponentdesiredstate MODIFY repo_state VARCHAR(255)
03 Oct 2017 06:30:26,598 INFO [main] DBAccessorImpl:874 - Executing query:
ALTER TABLE servicecomponentdesiredstate MODIFY repo_state VARCHAR(255) NOT
NULL
03 Oct 2017 06:30:26,742 INFO [main] DBAccessorImpl:874 - Executing query:
ALTER TABLE servicecomponentdesiredstate ADD CONSTRAINT FK_scds_desired_repo_id
FOREIGN KEY (desired_repo_version_id) REFERENCES repo_version (repo_version_id)
03 Oct 2017 06:30:26,782 INFO [main] DBAccessorImpl:874 - Executing query:
ALTER TABLE servicecomponentdesiredstate DROP FOREIGN KEY
FK_scds_desired_stack_id
03 Oct 2017 06:30:26,802 INFO [main] DBAccessorImpl:874 - Executing query:
ALTER TABLE servicecomponentdesiredstate DROP COLUMN desired_stack_id
03 Oct 2017 06:30:26,817 ERROR [main] DBAccessorImpl:880 - Error executing
query: ALTER TABLE servicecomponentdesiredstate DROP COLUMN desired_stack_id
java.sql.SQLException: Cannot drop index
'FK_servicecomponentdesiredstate_desired_stack_id': needed in a foreign key
constraint
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:996)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2526)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2484)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:848)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:742)
at
org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:877)
at
org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:869)
at
org.apache.ambari.server.orm.DBAccessorImpl.dropColumn(DBAccessorImpl.java:974)
at
org.apache.ambari.server.upgrade.UpgradeCatalog260.updateServiceComponentDesiredStateTable(UpgradeCatalog260.java:386)
at
org.apache.ambari.server.upgrade.UpgradeCatalog260.executeDDLUpdates(UpgradeCatalog260.java:185)
at
org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:923)
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:200)
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:418)
03 Oct 2017 06:30:26,822 ERROR [main] SchemaUpgradeHelper:202 - Upgrade failed.
{code}
Diffs
-
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
5a24bbb22b
Diff: https://reviews.apache.org/r/62931/diff/1/
Testing (updated)
---
mvn clean test
Also checked Ambari upgrade with this patch and affected DB from QA cluster
Thanks,
Dmitro Lisnichenko
Review Request 62931: Ambari Schema Upgrade Failed during Ambari Upgrade (patch with workaround)
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62931/
---
Review request for Ambari, Jonathan Hurley and Nate Cole.
Bugs: AMBARI-22216
https://issues.apache.org/jira/browse/AMBARI-22216
Repository: ambari
Description
---
For now, patch with workaround for issue on QA cluster.
Ambari Schema Upgrade Failed, Upgrade from : 2510 to 2600 (Also seen from 2430
to 2600)
ambari-server.log
{code}
03 Oct 2017 06:30:26,349 INFO [main] DBAccessorImpl:874 - Executing query:
ALTER TABLE servicecomponentdesiredstate ADD desired_repo_version_id BIGINT NOT
NULL DEFAULT 1
03 Oct 2017 06:30:26,389 INFO [main] DBAccessorImpl:874 - Executing query:
ALTER TABLE servicecomponentdesiredstate MODIFY desired_repo_version_id BIGINT
03 Oct 2017 06:30:26,435 INFO [main] DBAccessorImpl:874 - Executing query:
ALTER TABLE servicecomponentdesiredstate MODIFY desired_repo_version_id BIGINT
NOT NULL
03 Oct 2017 06:30:26,490 INFO [main] DBAccessorImpl:874 - Executing query:
ALTER TABLE servicecomponentdesiredstate ADD repo_state VARCHAR(255) NOT NULL
DEFAULT 'CURRENT'
03 Oct 2017 06:30:26,549 INFO [main] DBAccessorImpl:874 - Executing query:
ALTER TABLE servicecomponentdesiredstate MODIFY repo_state VARCHAR(255)
03 Oct 2017 06:30:26,598 INFO [main] DBAccessorImpl:874 - Executing query:
ALTER TABLE servicecomponentdesiredstate MODIFY repo_state VARCHAR(255) NOT
NULL
03 Oct 2017 06:30:26,742 INFO [main] DBAccessorImpl:874 - Executing query:
ALTER TABLE servicecomponentdesiredstate ADD CONSTRAINT FK_scds_desired_repo_id
FOREIGN KEY (desired_repo_version_id) REFERENCES repo_version (repo_version_id)
03 Oct 2017 06:30:26,782 INFO [main] DBAccessorImpl:874 - Executing query:
ALTER TABLE servicecomponentdesiredstate DROP FOREIGN KEY
FK_scds_desired_stack_id
03 Oct 2017 06:30:26,802 INFO [main] DBAccessorImpl:874 - Executing query:
ALTER TABLE servicecomponentdesiredstate DROP COLUMN desired_stack_id
03 Oct 2017 06:30:26,817 ERROR [main] DBAccessorImpl:880 - Error executing
query: ALTER TABLE servicecomponentdesiredstate DROP COLUMN desired_stack_id
java.sql.SQLException: Cannot drop index
'FK_servicecomponentdesiredstate_desired_stack_id': needed in a foreign key
constraint
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:996)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2526)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2484)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:848)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:742)
at
org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:877)
at
org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:869)
at
org.apache.ambari.server.orm.DBAccessorImpl.dropColumn(DBAccessorImpl.java:974)
at
org.apache.ambari.server.upgrade.UpgradeCatalog260.updateServiceComponentDesiredStateTable(UpgradeCatalog260.java:386)
at
org.apache.ambari.server.upgrade.UpgradeCatalog260.executeDDLUpdates(UpgradeCatalog260.java:185)
at
org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:923)
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:200)
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:418)
03 Oct 2017 06:30:26,822 ERROR [main] SchemaUpgradeHelper:202 - Upgrade failed.
{code}
Diffs
-
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
5a24bbb22b
Diff: https://reviews.apache.org/r/62931/diff/1/
Testing
---
mvn clean test
Thanks,
Dmitro Lisnichenko
Re: Review Request 62895: HdpCoreMpack: Quick Links should show NN web UI links
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62895/#review187753 --- Ship it! Ship It! - Dmitro Lisnichenko On Oct. 11, 2017, 7:17 p.m., Vitalyi Brodetskyi wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62895/ > --- > > (Updated Oct. 11, 2017, 7:17 p.m.) > > > Review request for Ambari, Dmitro Lisnichenko, Jayush Luniya, and Madhuvanthi > Radhakrishnan. > > > Bugs: AMBARI-22209 > https://issues.apache.org/jira/browse/AMBARI-22209 > > > Repository: ambari > > > Description > --- > > Fix quick links in mpacks. > > > Diffs > - > > ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java > 1cad4df > > > Diff: https://reviews.apache.org/r/62895/diff/1/ > > > Testing > --- > > will check tests after approve > > > Thanks, > > Vitalyi Brodetskyi > >
Re: Review Request 62922: "ambari-server upgrade" failed on db schema [Upgrade]
integer
Hint: You will need to rewrite or cast the expression.
at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:403)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:395)
at
org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:877)
at
org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:869)
at
org.apache.ambari.server.orm.DBAccessorImpl.addColumn(DBAccessorImpl.java:649)
at
org.apache.ambari.server.upgrade.UpgradeCatalog260.addLegacyColumn(UpgradeCatalog260.java:237)
at
org.apache.ambari.server.upgrade.UpgradeCatalog260.executeDDLUpdates(UpgradeCatalog260.java:199)
at
org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:923)
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:200)
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:418)
10 Oct 2017 23:22:03,351 ERROR [main] SchemaUpgradeHelper:437 - Exception
occurred during upgrade, failed
org.apache.ambari.server.AmbariException: ERROR: column "legacy" is of type
boolean but default expression is of type integer
Hint: You will need to rewrite or cast the expression.
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:203)
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:418)
Caused by: org.postgresql.util.PSQLException: ERROR: column "legacy" is of type
boolean but default expression is of type integer
Hint: You will need to rewrite or cast the expression.
at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:403)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:395)
at
org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:877)
at
org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:869)
at
org.apache.ambari.server.orm.DBAccessorImpl.addColumn(DBAccessorImpl.java:649)
at
org.apache.ambari.server.upgrade.UpgradeCatalog260.addLegacyColumn(UpgradeCatalog260.java:237)
at
org.apache.ambari.server.upgrade.UpgradeCatalog260.executeDDLUpdates(UpgradeCatalog260.java:199)
at
org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:923)
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:200)
... 1 more{code}
Diffs
-
ambari-server/src/main/java/org/apache/ambari/server/orm/DBAccessorImpl.java
20b991b279
Diff: https://reviews.apache.org/r/62922/diff/1/
Testing
---
mvn clean test
manual check on live cluster
Thanks,
Dmitro Lisnichenko
Review Request 62922: "ambari-server upgrade" failed on db schema [Upgrade]
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:403)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:395)
at
org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:877)
at
org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:869)
at
org.apache.ambari.server.orm.DBAccessorImpl.addColumn(DBAccessorImpl.java:649)
at
org.apache.ambari.server.upgrade.UpgradeCatalog260.addLegacyColumn(UpgradeCatalog260.java:237)
at
org.apache.ambari.server.upgrade.UpgradeCatalog260.executeDDLUpdates(UpgradeCatalog260.java:199)
at
org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:923)
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:200)
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:418)
10 Oct 2017 23:22:03,351 ERROR [main] SchemaUpgradeHelper:437 - Exception
occurred during upgrade, failed
org.apache.ambari.server.AmbariException: ERROR: column "legacy" is of type
boolean but default expression is of type integer
Hint: You will need to rewrite or cast the expression.
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:203)
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:418)
Caused by: org.postgresql.util.PSQLException: ERROR: column "legacy" is of type
boolean but default expression is of type integer
Hint: You will need to rewrite or cast the expression.
at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:403)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:395)
at
org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:877)
at
org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:869)
at
org.apache.ambari.server.orm.DBAccessorImpl.addColumn(DBAccessorImpl.java:649)
at
org.apache.ambari.server.upgrade.UpgradeCatalog260.addLegacyColumn(UpgradeCatalog260.java:237)
at
org.apache.ambari.server.upgrade.UpgradeCatalog260.executeDDLUpdates(UpgradeCatalog260.java:199)
at
org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:923)
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:200)
... 1 more{code}
Diffs
-
ambari-server/src/main/java/org/apache/ambari/server/orm/DBAccessorImpl.java
20b991b279
Diff: https://reviews.apache.org/r/62922/diff/1/
Testing
---
mvn clean test
manual check on live cluster
Thanks,
Dmitro Lisnichenko
Re: Review Request 62866: Make hive server create directories related to replication
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62866/#review187658 --- Ship it! Ship It! - Dmitro Lisnichenko On Oct. 11, 2017, 5:44 p.m., Eugene Chekanskiy wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62866/ > --- > > (Updated Oct. 11, 2017, 5:44 p.m.) > > > Review request for Ambari, Dmitro Lisnichenko and Robert Levas. > > > Bugs: AMBARI-22188 > https://issues.apache.org/jira/browse/AMBARI-22188 > > > Repository: ambari > > > Description > --- > > make sure that hive directories needed by replication is created on hive > server start if configured > > > Diffs > - > > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/hive-site.xml > d66cf4c1a8 > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py > 8e176b69ac > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py > 1d68ef4217 > > ambari-server/src/main/resources/common-services/HIVE/2.1.0.3.0/configuration/hive-site.xml > edbbc9d10e > > ambari-server/src/main/resources/common-services/HIVE/2.1.0.3.0/package/scripts/hive.py > e7e92ec985 > > ambari-server/src/main/resources/common-services/HIVE/2.1.0.3.0/package/scripts/params_linux.py > f6b676b65e > > > Diff: https://reviews.apache.org/r/62866/diff/3/ > > > Testing > --- > > mvn clean test, cluster deploy > > > Thanks, > > Eugene Chekanskiy > >
Re: Review Request 62866: Make hive server create directories related to replication
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62866/#review187651 --- ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/hive-site.xml Lines 700 (patched) <https://reviews.apache.org/r/62866/#comment264708> After Ambari 2.5, we usually tend to avoid adding/modifying configs during Ambari upgrade - Dmitro Lisnichenko On Oct. 11, 2017, 3:09 p.m., Eugene Chekanskiy wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62866/ > --- > > (Updated Oct. 11, 2017, 3:09 p.m.) > > > Review request for Ambari, Dmitro Lisnichenko and Robert Levas. > > > Bugs: AMBARI-22188 > https://issues.apache.org/jira/browse/AMBARI-22188 > > > Repository: ambari > > > Description > --- > > make sure that hive directories needed by replication is created on hive > server start if configured > > > Diffs > - > > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/hive-site.xml > d66cf4c1a8 > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py > 8e176b69ac > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py > 1d68ef4217 > > ambari-server/src/main/resources/common-services/HIVE/2.1.0.3.0/configuration/hive-site.xml > edbbc9d10e > > ambari-server/src/main/resources/common-services/HIVE/2.1.0.3.0/package/scripts/hive.py > e7e92ec985 > > ambari-server/src/main/resources/common-services/HIVE/2.1.0.3.0/package/scripts/params_linux.py > f6b676b65e > > > Diff: https://reviews.apache.org/r/62866/diff/2/ > > > Testing > --- > > mvn clean test, cluster deploy > > > Thanks, > > Eugene Chekanskiy > >
Re: Review Request 62871: Prevent Patch Upgrades With Known Service Incompatibilities
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62871/#review187641 --- Fix it, then Ship it! ambari-server/src/main/java/org/apache/ambari/server/checks/AbstractCheckDescriptor.java Lines 318 (patched) <https://reviews.apache.org/r/62871/#comment264700> comment seems to be irrelevant to the code - Dmitro Lisnichenko On Oct. 10, 2017, 11:42 p.m., Jonathan Hurley wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62871/ > --- > > (Updated Oct. 10, 2017, 11:42 p.m.) > > > Review request for Ambari, Dmitro Lisnichenko, Nate Cole, and Robert Levas. > > > Bugs: AMBARI-22194 > https://issues.apache.org/jira/browse/AMBARI-22194 > > > Repository: ambari > > > Description > --- > > Some stacks, such as HDP, do not support service-level upgrades for some > services since they have hard coded dependencies with other services which > might not be in the upgrade. > > For example, if YARN is configured to use Tez, then an upgrade of YARN or Tez > must also include Tez or YARN respectively. This is because the path is hard > coded inside of YARN to use its version and cannot be changed via any > external environment variable. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/checks/AbstractCheckDescriptor.java > d54e8a176a > > ambari-server/src/main/java/org/apache/ambari/server/checks/CheckDescription.java > 314ab83718 > > ambari-server/src/main/java/org/apache/ambari/server/checks/RequiredServicesInRepositoryCheck.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/checks/UpgradeCheck.java > 442bf0be12 > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProvider.java > a2fb44199f > > ambari-server/src/main/java/org/apache/ambari/server/state/RepositoryType.java > 954f705e9f > > ambari-server/src/main/java/org/apache/ambari/server/state/repository/VersionDefinitionXml.java > c17f5e90f9 > > ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json > 2db0a74195 > > ambari-server/src/test/java/org/apache/ambari/server/checks/AbstractCheckDescriptorTest.java > 092bd0bd88 > > ambari-server/src/test/java/org/apache/ambari/server/checks/AutoStartDisabledCheckTest.java > 671648ab71 > > ambari-server/src/test/java/org/apache/ambari/server/checks/ClientRetryPropertyCheckTest.java > 206b4519c1 > > ambari-server/src/test/java/org/apache/ambari/server/checks/ComponentExistsInRepoCheckTest.java > ede1daacff > > ambari-server/src/test/java/org/apache/ambari/server/checks/ComponentsInstallationCheckTest.java > 4ae33f8bbf > > ambari-server/src/test/java/org/apache/ambari/server/checks/ConfigurationMergeCheckTest.java > df7bd1fb70 > > ambari-server/src/test/java/org/apache/ambari/server/checks/DruidHighAvailabilityCheckTest.java > e90931bc11 > > ambari-server/src/test/java/org/apache/ambari/server/checks/HiveMultipleMetastoreCheckTest.java > 7a1fca3161 > > ambari-server/src/test/java/org/apache/ambari/server/checks/HiveNotRollingWarningTest.java > 3b0c900991 > > ambari-server/src/test/java/org/apache/ambari/server/checks/HostsHeartbeatCheckTest.java > daf2e4cb76 > > ambari-server/src/test/java/org/apache/ambari/server/checks/HostsMasterMaintenanceCheckTest.java > 8afa1eaf89 > > ambari-server/src/test/java/org/apache/ambari/server/checks/HostsRepositoryVersionCheckTest.java > 5539618c28 > > ambari-server/src/test/java/org/apache/ambari/server/checks/InstallPackagesCheckTest.java > cdfbb0a790 > > ambari-server/src/test/java/org/apache/ambari/server/checks/KafkaKerberosCheckTest.java > 9a95c42327 > > ambari-server/src/test/java/org/apache/ambari/server/checks/MapReduce2JobHistoryStatePreservingCheckTest.java > fc322b3d91 > > ambari-server/src/test/java/org/apache/ambari/server/checks/RangerAuditDbCheckTest.java > ec91e483b3 > > ambari-server/src/test/java/org/apache/ambari/server/checks/RangerPasswordCheckTest.java > f9569c3820 > > ambari-server/src/test/java/org/apache/ambari/server/checks/RangerSSLConfigCheckTest.java > af016b01e0 > > ambari-server/src/test/java/org/apache/ambari/server/checks/RequiredServicesInRepositoryCheckTest.java > PRE-CREATION > > am
Re: Review Request 62851: Install and upgrade options are shown for current and already upgraded stack versions
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62851/#review187510 --- Ship it! Ship It! - Dmitro Lisnichenko On Oct. 10, 2017, 12:19 a.m., Nate Cole wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62851/ > --- > > (Updated Oct. 10, 2017, 12:19 a.m.) > > > Review request for Ambari, Dmitro Lisnichenko and Jonathan Hurley. > > > Bugs: AMBARI-22178 > https://issues.apache.org/jira/browse/AMBARI-22178 > > > Repository: ambari > > > Description > --- > > The HostVersionOutOfSyncListener is incorrectly setting ALL other > host_version records to OUT_OF_SYNC other than what the newly-added component > is for. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/events/listeners/upgrade/HostVersionOutOfSyncListener.java > 78744613ed > > ambari-server/src/test/java/org/apache/ambari/server/events/listeners/upgrade/HostVersionOutOfSyncListenerTest.java > 076190a47d > > > Diff: https://reviews.apache.org/r/62851/diff/1/ > > > Testing > --- > > Manual added host and added new service to verify records. Unit tests > pending. > > > Thanks, > > Nate Cole > >
Re: Review Request 62848: Not able to perform revert after deleting the upgraded service
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62848/#review187509 --- Ship it! Ship It! - Dmitro Lisnichenko On Oct. 9, 2017, 10:06 p.m., Jonathan Hurley wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62848/ > --- > > (Updated Oct. 9, 2017, 10:06 p.m.) > > > Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole. > > > Bugs: AMBARI-22166 > https://issues.apache.org/jira/browse/AMBARI-22166 > > > Repository: ambari > > > Description > --- > > After deleting a service, you can't revert a PATCH or MAINT upgrade. This is > because we iterate over all prior services when creating the upgrade context > for the reversion. > > STR: > - Install ZK, HDFS, and Storm > - Upgrade ZK and Storm using a PATCH or MAINT VDF > - Remove Storm > - Revert the prior patch > > ``` > 09 Oct 2017 10:02:59,142 ERROR [ambari-client-thread-22258] > AbstractResourceProvider:285 - Caught AmbariException when creating a resource > org.apache.ambari.server.ServiceNotFoundException: Service not found, > clusterName=cl1, serviceName=ATLAS > at > org.apache.ambari.server.state.cluster.ClusterImpl.getService(ClusterImpl.java:863) > at > org.apache.ambari.server.state.UpgradeContext.(UpgradeContext.java:403) > at > org.apache.ambari.server.state.UpgradeContext$$FastClassByGuice$$6e2e5fe5.newInstance() > at > com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) > at > com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60) > at > com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) > at > com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) > at > com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978) > at > com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024) > at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974) > at > com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:632) > at com.sun.proxy.$Proxy31.create(Unknown Source) > ``` > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeContext.java > 9f31ab8a62 > > ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeContextTest.java > 5cf9a4c859 > > > Diff: https://reviews.apache.org/r/62848/diff/2/ > > > Testing > --- > > Manual reversion of a PATCH after removing a service. > > [INFO] Results: > [INFO] > [WARNING] Tests run: 4915, Failures: 0, Errors: 0, Skipped: 34 > > > Thanks, > > Jonathan Hurley > >
Re: Review Request 62807: Ambari schema upgrade fails when upgrading ambari from 2.5.1.0 to 2.6.0.0 and using oracle as database
is referenced by some foreign keys
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:440)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:837)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:445)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523)
at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:193)
at oracle.jdbc.driver.T4CStatement.executeForRows(T4CStatement.java:999)
at
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1315)
at oracle.jdbc.driver.OracleStatement.executeInternal(OracleStatement.java:1890)
at oracle.jdbc.driver.OracleStatement.execute(OracleStatement.java:1855)
at
oracle.jdbc.driver.OracleStatementWrapper.execute(OracleStatementWrapper.java:304)
at
org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:877)
at
org.apache.ambari.server.orm.DBAccessorImpl.dropPKConstraint(DBAccessorImpl.java:1045)
at
org.apache.ambari.server.orm.DBAccessorImpl.dropPKConstraint(DBAccessorImpl.java:1053)
at
org.apache.ambari.server.orm.DBAccessorImpl.dropPKConstraint(DBAccessorImpl.java:1344)
at
org.apache.ambari.server.upgrade.UpgradeCatalog260.addViewUrlPKConstraint(UpgradeCatalog260.java:206)
at
org.apache.ambari.server.upgrade.UpgradeCatalog260.executeDDLUpdates(UpgradeCatalog260.java:196)
at
org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:923)
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:200)
... 1 more
{code}
Cluster info :
Base Ambari : 2.5.1.0
Upgraded Ambari : 2.6.0.0-183
Diffs
-
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
4bc7489ad8
Diff: https://reviews.apache.org/r/62807/diff/1/
Testing
---
mvn clean test
and live cluster check
Thanks,
Dmitro Lisnichenko
Re: Review Request 62807: Ambari schema upgrade fails when upgrading ambari from 2.5.1.0 to 2.6.0.0 and using oracle as database
is referenced by some foreign keys
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:440)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:837)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:445)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523)
at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:193)
at oracle.jdbc.driver.T4CStatement.executeForRows(T4CStatement.java:999)
at
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1315)
at oracle.jdbc.driver.OracleStatement.executeInternal(OracleStatement.java:1890)
at oracle.jdbc.driver.OracleStatement.execute(OracleStatement.java:1855)
at
oracle.jdbc.driver.OracleStatementWrapper.execute(OracleStatementWrapper.java:304)
at
org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:877)
at
org.apache.ambari.server.orm.DBAccessorImpl.dropPKConstraint(DBAccessorImpl.java:1045)
at
org.apache.ambari.server.orm.DBAccessorImpl.dropPKConstraint(DBAccessorImpl.java:1053)
at
org.apache.ambari.server.orm.DBAccessorImpl.dropPKConstraint(DBAccessorImpl.java:1344)
at
org.apache.ambari.server.upgrade.UpgradeCatalog260.addViewUrlPKConstraint(UpgradeCatalog260.java:206)
at
org.apache.ambari.server.upgrade.UpgradeCatalog260.executeDDLUpdates(UpgradeCatalog260.java:196)
at
org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:923)
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:200)
... 1 more
{code}
Cluster info :
Base Ambari : 2.5.1.0
Upgraded Ambari : 2.6.0.0-183
Diffs
-
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
4bc7489ad8
Diff: https://reviews.apache.org/r/62807/diff/1/
Testing (updated)
---
mvn clean test
and live cluster check
Thanks,
Dmitro Lisnichenko
Review Request 62807: Ambari schema upgrade fails when upgrading ambari from 2.5.1.0 to 2.6.0.0 and using oracle as database
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:440)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:837)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:445)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523)
at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:193)
at oracle.jdbc.driver.T4CStatement.executeForRows(T4CStatement.java:999)
at
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1315)
at oracle.jdbc.driver.OracleStatement.executeInternal(OracleStatement.java:1890)
at oracle.jdbc.driver.OracleStatement.execute(OracleStatement.java:1855)
at
oracle.jdbc.driver.OracleStatementWrapper.execute(OracleStatementWrapper.java:304)
at
org.apache.ambari.server.orm.DBAccessorImpl.executeQuery(DBAccessorImpl.java:877)
at
org.apache.ambari.server.orm.DBAccessorImpl.dropPKConstraint(DBAccessorImpl.java:1045)
at
org.apache.ambari.server.orm.DBAccessorImpl.dropPKConstraint(DBAccessorImpl.java:1053)
at
org.apache.ambari.server.orm.DBAccessorImpl.dropPKConstraint(DBAccessorImpl.java:1344)
at
org.apache.ambari.server.upgrade.UpgradeCatalog260.addViewUrlPKConstraint(UpgradeCatalog260.java:206)
at
org.apache.ambari.server.upgrade.UpgradeCatalog260.executeDDLUpdates(UpgradeCatalog260.java:196)
at
org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:923)
at
org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:200)
... 1 more
{code}
Cluster info :
Base Ambari : 2.5.1.0
Upgraded Ambari : 2.6.0.0-183
Diffs
-
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
4bc7489ad8
Diff: https://reviews.apache.org/r/62807/diff/1/
Testing
---
mvn clean test
Thanks,
Dmitro Lisnichenko
Re: Review Request 62794: CURRENT cluster Shows Upgrade If Component Didn't Report Version
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62794/#review187241 --- Ship it! ambari-common/src/main/python/resource_management/libraries/script/script.py Lines 216 (patched) <https://reviews.apache.org/r/62794/#comment264177> instlled - Dmitro Lisnichenko On Oct. 6, 2017, 6:16 a.m., Jonathan Hurley wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62794/ > --- > > (Updated Oct. 6, 2017, 6:16 a.m.) > > > Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole. > > > Bugs: AMBARI-22139 > https://issues.apache.org/jira/browse/AMBARI-22139 > > > Repository: ambari > > > Description > --- > > CURRENT cluster Shows Upgrade If Component Didn't Report Version > > STR: > - Install a cluster with 2.6.0.0 and Atlas > - Atlas is busted in this version and doesn't report it's version > - This causes the main install stack to be OUT_OF_SYNC which it should never > be since it's the only one. > > > Diffs > - > > ambari-agent/src/test/python/resource_management/TestScript.py 75726d686d > > ambari-common/src/main/python/resource_management/libraries/script/script.py > 829868e03a > ambari-server/src/test/python/stacks/utils/RMFTestCase.py 665fc202c5 > > > Diff: https://reviews.apache.org/r/62794/diff/2/ > > > Testing > --- > > Install of a cluster with a broken component and verified that the stack > version is CURRENT after all services reported in. > > -- > Total run:1193 > Total errors:0 > Total failures:0 > OK > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 55.313 s > [INFO] Finished at: 2017-10-05T23:15:30-04:00 > [INFO] Final Memory: 21M/619M > [INFO] > > > > Thanks, > > Jonathan Hurley > >
Re: Review Request 62785: Hitting pause during upgrade jumps progress to 100%
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62785/#review187196 --- Ship it! Ship It! - Dmitro Lisnichenko On Oct. 5, 2017, 4:01 p.m., Nate Cole wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62785/ > --- > > (Updated Oct. 5, 2017, 4:01 p.m.) > > > Review request for Ambari, Dmitro Lisnichenko and Jonathan Hurley. > > > Bugs: AMBARI-22144 > https://issues.apache.org/jira/browse/AMBARI-22144 > > > Repository: ambari > > > Description > --- > > When performing an upgrade, hitting pause jumps your progress right to 100%. > So if you're at 63%, as soon as you hit Pause it goes to 100%. As soon as > you hit "Resume", it will go back to 63%. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java > a28fb02f52 > > ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java > 7b9307854a > > > Diff: https://reviews.apache.org/r/62785/diff/1/ > > > Testing > --- > > Manual. Unit test: > > [INFO] Results: > [INFO] > [WARNING] Tests run: 4915, Failures: 0, Errors: 0, Skipped: 34 > [INFO] > > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 24:53.175s > [INFO] Finished at: Thu Oct 05 10:49:21 EDT 2017 > [INFO] Final Memory: 79M/1940M > [INFO] > > > > Thanks, > > Nate Cole > >
Re: Review Request 62785: Hitting pause during upgrade jumps progress to 100%
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62785/#review187184 --- ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java Lines 478 (patched) <https://reviews.apache.org/r/62785/#comment264091> what does 0.35/0.09 mean here? Would it play well for big/small numbers of tasks? - Dmitro Lisnichenko On Oct. 5, 2017, 4:01 p.m., Nate Cole wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62785/ > --- > > (Updated Oct. 5, 2017, 4:01 p.m.) > > > Review request for Ambari, Dmitro Lisnichenko and Jonathan Hurley. > > > Bugs: AMBARI-22144 > https://issues.apache.org/jira/browse/AMBARI-22144 > > > Repository: ambari > > > Description > --- > > When performing an upgrade, hitting pause jumps your progress right to 100%. > So if you're at 63%, as soon as you hit Pause it goes to 100%. As soon as > you hit "Resume", it will go back to 63%. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java > a28fb02f52 > > ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java > 7b9307854a > > > Diff: https://reviews.apache.org/r/62785/diff/1/ > > > Testing > --- > > Manual. Unit tests pending. > > > Thanks, > > Nate Cole > >
Re: Review Request 62754: Adding Components On Patched Clusters Can Result In Symlink Issues With conf Directories
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62754/#review187080 --- Ship it! Ship It! - Dmitro Lisnichenko On Oct. 4, 2017, 12:11 a.m., Jonathan Hurley wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62754/ > --- > > (Updated Oct. 4, 2017, 12:11 a.m.) > > > Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole. > > > Bugs: AMBARI-22123 > https://issues.apache.org/jira/browse/AMBARI-22123 > > > Repository: ambari > > > Description > --- > > Essentially, the symlinking was doing a quick-exit if the > /etc//conf directory already existed. However, it might have been > created by another RPM and not the component we actually care about. > > The refactor here did two things: > - Invokes conf-select on the necessary packages > - Combined most of the logic between conf_select.select(...) and > conf_select.convert_conf_directories_to_symlinks(...) > - Removed legacy code from back in the pre-HDP 2.3 days > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py > 86821bf47a > ambari-server/src/main/resources/custom_actions/scripts/install_packages.py > 3ace3b1570 > > ambari-server/src/main/resources/stacks/BigInsights/4.0/services/ZOOKEEPER/package/scripts/zookeeper_service.py > 0727970adb > > ambari-server/src/main/resources/stacks/BigInsights/4.2/services/ZOOKEEPER/package/scripts/zookeeper_service.py > 0727970adb > > ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py > 20fd48de15 > ambari-server/src/test/python/TestAmbariServer.py 5db79137ee > ambari-server/src/test/python/TestMpacks.py fd30bf6f88 > > ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py > 92f5011978 > ambari-server/src/test/python/stacks/2.2/common/test_conf_select.py > 2eeec46b4a > > contrib/management-packs/odpi-ambari-mpack/src/main/resources/stacks/ODPi/2.0/services/HIVE/package/scripts/hive_client.py > 3d9bfd7c3e > > contrib/management-packs/odpi-ambari-mpack/src/main/resources/stacks/ODPi/2.0/services/HIVE/package/scripts/hive_metastore.py > b88f3858a7 > > contrib/management-packs/odpi-ambari-mpack/src/main/resources/stacks/ODPi/2.0/services/HIVE/package/scripts/hive_server.py > 31b083bd0d > > contrib/management-packs/odpi-ambari-mpack/src/main/resources/stacks/ODPi/2.0/services/HIVE/package/scripts/hive_server_interactive.py > 2df001cf21 > > contrib/management-packs/odpi-ambari-mpack/src/main/resources/stacks/ODPi/2.0/services/HIVE/package/scripts/webhcat_server.py > 34687c453b > > contrib/management-packs/odpi-ambari-mpack/src/main/resources/stacks/ODPi/2.0/services/YARN/package/scripts/application_timeline_server.py > 4ec6aa788f > > contrib/management-packs/odpi-ambari-mpack/src/main/resources/stacks/ODPi/2.0/services/YARN/package/scripts/historyserver.py > 34c683ab2f > > contrib/management-packs/odpi-ambari-mpack/src/main/resources/stacks/ODPi/2.0/services/YARN/package/scripts/mapreduce2_client.py > 424157b128 > > contrib/management-packs/odpi-ambari-mpack/src/main/resources/stacks/ODPi/2.0/services/YARN/package/scripts/nodemanager.py > b235cad164 > > contrib/management-packs/odpi-ambari-mpack/src/main/resources/stacks/ODPi/2.0/services/YARN/package/scripts/resourcemanager.py > 71c7bc17a2 > > contrib/management-packs/odpi-ambari-mpack/src/main/resources/stacks/ODPi/2.0/services/YARN/package/scripts/yarn_client.py > 4d65a404a2 > > > Diff: https://reviews.apache.org/r/62754/diff/3/ > > > Testing > --- > > Total run:1192 > Total errors:0 > Total failures:0 > OK > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 57.162 s > [INFO] Finished at: 2017-10-03T23:04:47-04:00 > [INFO] Final Memory: 19M/491M > [INFO] > > > > Thanks, > > Jonathan Hurley > >
Re: Review Request 62764: AMBARI-22127. Installation of stack selector fails on Debian
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62764/#review187078 --- Ship it! Ship It! - Dmitro Lisnichenko On Oct. 4, 2017, 1:02 p.m., Attila Doroszlai wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62764/ > --- > > (Updated Oct. 4, 2017, 1:02 p.m.) > > > Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, Jonathan > Hurley, Nate Cole, and Sebastian Toader. > > > Bugs: AMBARI-22127 > https://issues.apache.org/jira/browse/AMBARI-22127 > > > Repository: ambari > > > Description > --- > > Need to pass both repo IDs and repo file names via `use_repos`, because > Debian/Ubuntu selects specific repos by file name, while RedHat/SuSE by repo > ID. > > > Diffs > - > > ambari-agent/src/test/python/resource_management/TestPackageResource.py > 49e3adf6782021977ff3edce850da35e3adc7116 > > ambari-common/src/main/python/resource_management/core/providers/package/__init__.py > 94047571b5113716ee041e890fea559b4ce263ee > > ambari-common/src/main/python/resource_management/core/providers/package/apt.py > f6a5538a30cf86eb4125ff088a1a8ca24c5adece > > ambari-common/src/main/python/resource_management/core/providers/package/choco.py > db552964279f509de90d2fde884bfc0fea12f7b0 > > ambari-common/src/main/python/resource_management/core/providers/package/yumrpm.py > a4a55da7c9c5e9ac554d91edd19e9cb668615671 > > ambari-common/src/main/python/resource_management/core/providers/package/zypper.py > 5b8e5ab94091bf181e97defd6a13dea4e3a46863 > > ambari-common/src/main/python/resource_management/core/resources/packaging.py > e3adc30f46abd3f33670f817421d88270b3e7ade > > ambari-common/src/main/python/resource_management/libraries/functions/repository_util.py > f1074ee7b3748cfedb3e1a813be52df894cff97f > ambari-server/src/main/resources/custom_actions/scripts/install_packages.py > dcf354447756d2e1c826feb8515cfe3b9711f19d > ambari-server/src/test/python/custom_actions/TestInstallPackages.py > 0f303bee541ea3ac06fadb67eb44f5aab97fa1d5 > > > Diff: https://reviews.apache.org/r/62764/diff/1/ > > > Testing > --- > > Tested package installation (both pre-upgrade "Install Version" and regular > "Add Service") on Debian 7.6 and CentOS 6.4. > > Unit tests (on `branch-2.6`): > > ``` > $ mvn -am -pl ambari-agent,ambari-server -DskipSurefireTests test > ... > Total run:1192 > Total errors:0 > Total failures:0 > OK > ... > Ran 453 tests in 137.865s > > OK > ... > [INFO] Ambari Server .. SUCCESS [01:28 > min] > [INFO] Ambari Agent ... SUCCESS [02:21 > min] > ``` > > Unit tests of Ambari Agent on `trunk` are wrong due to an unrelated change. > > > Thanks, > > Attila Doroszlai > >
Re: Review Request 62718: Refactor existing server side actions to use the common AbstractUpgradeServerAction
/RangerKmsProxyConfigTest.java 2be8db4226 ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/RangerUsersyncConfigCalculationTest.java 427fb33862 ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/RangerWebAlertConfigActionTest.java 362f372b10 ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/SparkShufflePropertyConfigTest.java 518ab42df0 ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/UpgradeUserKerberosDescriptorTest.java 59a8a4c676 Diff: https://reviews.apache.org/r/62718/diff/2/ Changes: https://reviews.apache.org/r/62718/diff/1-2/ Testing --- mvn clean test Thanks, Dmitro Lisnichenko
Re: Review Request 62726: Some ResourceProviders Are Not Transactional
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62726/#review186937 --- Ship it! Ship It! - Dmitro Lisnichenko On Oct. 2, 2017, 5:54 p.m., Nate Cole wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62726/ > --- > > (Updated Oct. 2, 2017, 5:54 p.m.) > > > Review request for Ambari, Dmitro Lisnichenko and Jonathan Hurley. > > > Bugs: AMBARI-22110 > https://issues.apache.org/jira/browse/AMBARI-22110 > > > Repository: ambari > > > Description > --- > > While investigating an issue with VDFs which don't contain any OS's which > match the cluster, it was observed that, even after throwing an Exception > inside of a Transaction, the entities created inside that transaction were > being committed. > Eventual investigation led to a problem between Guice/AOP and our > ResourceProviders. The Guice documentations says that @Transactional methods > can only be intercepted on objects created by Guice: > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java > e3f1458202 > > ambari-server/src/main/java/org/apache/ambari/server/controller/ResourceProviderFactory.java > 9cd1d745c6 > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AbstractControllerResourceProvider.java > 0782685d07 > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AlertTargetResourceProvider.java > 0b7f1db9c9 > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProvider.java > 9d2b39effc > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/DefaultProviderModule.java > 781b01525f > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ViewInstanceResourceProvider.java > 63e6c24fc8 > > > Diff: https://reviews.apache.org/r/62726/diff/1/ > > > Testing > --- > > Manual. Automated: > > [INFO] Results: > [INFO] > [WARNING] Tests run: 4914, Failures: 0, Errors: 0, Skipped: 34 > [INFO] > > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 24:53.576s > [INFO] Finished at: Mon Oct 02 11:20:47 EDT 2017 > [INFO] Final Memory: 78M/1941M > [INFO] > > > > Thanks, > > Nate Cole > >
Re: Review Request 62719: Upgrades Do Not Finalize If Other Host Versions Are Out of Sync
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62719/#review186822
---
Ship it!
Ship It!
- Dmitro Lisnichenko
On Oct. 2, 2017, 2:43 a.m., Jonathan Hurley wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62719/
> ---
>
> (Updated Oct. 2, 2017, 2:43 a.m.)
>
>
> Review request for Ambari, Dmitro Lisnichenko and Nate Cole.
>
>
> Bugs: AMBARI-22105
> https://issues.apache.org/jira/browse/AMBARI-22105
>
>
> Repository: ambari
>
>
> Description
> ---
>
> STR:
> - Install base version 2.6.0.0 with ZK and Storm
> - Cause that base version to be out of sync for 1 component, say ZK.
> - Install a patch for Storm
> - Upgrade and attempt to finalize Storm
>
> {noformat}
> Finalizing the upgrade to 2.6.0.3-8 for the following services: STORM
> The following 3 host(s) have not been upgraded to version 2.6.0.3-8. Please
> install and upgrade the Stack Version on those hosts and try again.
> Hosts: ctr-e134-1499953498516-185715-01-02.hwx.site,
> ctr-e134-1499953498516-185715-01-03.hwx.site,
> ctr-e134-1499953498516-185715-01-04.hwx.site
> {noformat}
>
> This is because the code is detecting the OUT_OF_SYNC of other
> components/repos on that host.
>
>
> Diffs
> -
>
>
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/RepositoryVersionEntity.java
> 072863daed
>
> ambari-server/src/main/java/org/apache/ambari/server/state/svccomphost/ServiceComponentHostImpl.java
> f9d3cfc52a
>
> ambari-server/src/main/java/org/apache/ambari/server/state/svccomphost/ServiceComponentHostSummary.java
> e9359ef1eb
>
>
> Diff: https://reviews.apache.org/r/62719/diff/1/
>
>
> Testing
> ---
>
> PENDING...
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Review Request 62718: Refactor existing server side actions to use the common AbstractUpgradeServerAction
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62718/ --- Review request for Ambari, Jonathan Hurley and Nate Cole. Bugs: AMBARI-22104 https://issues.apache.org/jira/browse/AMBARI-22104 Repository: ambari Description --- Other server-side classes need to use the abstract class in the summary. Identify fields that are largely common with the server-side actions and put them in the abstract class. Diffs - ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/AutoSkipFailedSummaryAction.java 664cb2c9a9 ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/ConfigureAction.java ec4d383dee ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/FixCapacitySchedulerOrderingPolicy.java f14d702a45 ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/FixLzoCodecPath.java 483372943b ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/FixOozieAdminUsers.java 75588d5921 ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/FixYarnWebServiceUrl.java 5823c8bdf6 ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/HBaseConfigCalculation.java 739dd7e985 ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/HBaseEnvMaxDirectMemorySizeAction.java fb1347 ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/HiveEnvClasspathAction.java c5000bf9b5 ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/HiveZKQuorumConfigAction.java 23eacec7d2 ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/KerberosKeytabsAction.java 4c6371b054 ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/ManualStageAction.java 39b23bc930 ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/OozieConfigCalculation.java 9b8a7dc583 ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/RangerConfigCalculation.java e53c95f703 ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/RangerKerberosConfigCalculation.java 348b69eac7 ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/RangerKmsProxyConfig.java 71c3a078f5 ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/RangerUsersyncConfigCalculation.java 3573748251 ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/RangerWebAlertConfigAction.java a6b94f4876 ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/SparkShufflePropertyConfig.java b1aa6e187a ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/UpdateDesiredRepositoryAction.java 10d6630698 ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/UpgradeUserKerberosDescriptor.java 864060095b ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/YarnConfigCalculation.java d6388584f0 Diff: https://reviews.apache.org/r/62718/diff/1/ Testing --- mvn clean test Thanks, Dmitro Lisnichenko
Re: Review Request 62675: Wrong Hadoop Home Directory Is Being Picked Up on MAINT/PATCH Upgraded Clusters
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62675/#review186647
---
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
Line 220 (original), 222 (patched)
<https://reviews.apache.org/r/62675/#comment263404>
looks like doc is unfinished
ambari-server/src/main/java/org/apache/ambari/server/agent/StatusCommand.java
Lines 43 (patched)
<https://reviews.apache.org/r/62675/#comment263405>
How it differs from componentName?
- Dmitro Lisnichenko
On Sept. 28, 2017, 9:27 p.m., Jonathan Hurley wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62675/
> ---
>
> (Updated Sept. 28, 2017, 9:27 p.m.)
>
>
> Review request for Ambari, Dmitro Lisnichenko, Nate Cole, and Robert Levas.
>
>
> Bugs: AMBARI-22083
> https://issues.apache.org/jira/browse/AMBARI-22083
>
>
> Repository: ambari
>
>
> Description
> ---
>
> STR:
>
> - Install HDP 2.6.0.0 with HDFS and YARN/MapR
> - Perform a MAINT upgrade of HDFS to 2.6.3.0
> - Stop all services
> - Restart all services
>
> The following is observed:
> - {{/usr/hdp/current/hadoop-client -> /usr/hdp/2.6.0.0/hadoop}}
> - HDFS picks up 2.6.0.0
> - YARN/MapR pick up 2.6.0.0
>
> The upgraded components should pickup the version of the hadoop home
> directory that is theirs.
>
>
> Diffs
> -
>
>
> ambari-common/src/main/python/resource_management/libraries/functions/component_version.py
> a1fd6b2e7c
>
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
> ffcaad5cbc
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
> f5068e4295
>
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatMonitor.java
> 0042f53c7e
>
> ambari-server/src/main/java/org/apache/ambari/server/agent/StatusCommand.java
> 5dec53c32f
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
> 77e1bed96f
>
> ambari-server/src/main/resources/stacks/BigInsights/4.0/hooks/after-INSTALL/scripts/params.py
> d3332db3af
>
> ambari-server/src/main/resources/stacks/BigInsights/4.0/hooks/before-ANY/scripts/params.py
> 5ffd28ca94
>
> ambari-server/src/main/resources/stacks/BigInsights/4.0/hooks/before-START/scripts/params.py
> 5c84a05b0d
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/hooks/after-INSTALL/scripts/params.py
> c4970543d6
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/hooks/before-ANY/scripts/params.py
> b0467a9a75
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/hooks/before-START/scripts/params.py
> be9db58fef
>
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/params.py
> 4d7eaee3b9
>
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py
> 1fd7f3ef76
>
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/shared_initialization.py
> 3997117ade
>
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py
> 5ca2d94403
>
> ambari-server/src/main/resources/stacks/HDP/2.3.ECS/services/ECS/package/scripts/params.py
> c304a93b5e
>
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-ANY/test_before_any.py
> 9dceb69fee
>
>
> Diff: https://reviews.apache.org/r/62675/diff/1/
>
>
> Testing
> ---
>
> End-to-End manual tests
> Lots of Python test fixes PENDING...
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 62578: Stack selection page does not load the HDP stacks (Redux)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62578/#review186287 --- Ship it! Ship It! - Dmitro Lisnichenko On Sept. 26, 2017, 7:29 p.m., Nate Cole wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62578/ > --- > > (Updated Sept. 26, 2017, 7:29 p.m.) > > > Review request for Ambari, Dmitro Lisnichenko and Jonathan Hurley. > > > Bugs: AMBARI-22062 > https://issues.apache.org/jira/browse/AMBARI-22062 > > > Repository: ambari > > > Description > --- > > Continuation of a previous bug that will block on VDF parsing, and also > ensure that stack definitions are not double-loaded as seen in the logs. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java > d03081f48d > > ambari-server/src/main/java/org/apache/ambari/server/stack/StackContext.java > e201e570ee > > > Diff: https://reviews.apache.org/r/62578/diff/1/ > > > Testing > --- > > Manual. Automated pending (no new tests) > > > Thanks, > > Nate Cole > >
Re: Review Request 62571: Fail to restart Ranger Admin during HDP downgrade.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62571/
---
(Updated Sept. 26, 2017, 6:58 p.m.)
Review request for Ambari, Jonathan Hurley and Nate Cole.
Bugs: AMBARI-22060
https://issues.apache.org/jira/browse/AMBARI-22060
Repository: ambari
Description
---
During the downgrade process, run into the following error whilst it's
restating Ranger Admin:
{code}
Traceback (most recent call last):
File
"/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py",
line 216, in
RangerAdmin().execute()
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 329, in execute
method(env)
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 850, in restart
self.start(env, upgrade_type=upgrade_type)
File
"/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py",
line 93, in start
setup_ranger_audit_solr()
File
"/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py",
line 705, in setup_ranger_audit_solr
new_service_principals = [params.ranger_admin_jaas_principal])
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/functions/solr_cloud_util.py",
line 329, in add_solr_roles
new_service_users.append(__remove_host_from_principal(new_service_user,
kerberos_realm))
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/functions/solr_cloud_util.py",
line 266, in __remove_host_from_principal
if not realm:
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py",
line 73, in __getattr__
raise Fail("Configuration parameter '" + self.name + "' was not found in
configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter
'kerberos-env' was not found in configurations dictionary!
{code}
The reason was that server did not have many configs selected, and did not send
them to agent during downgrade. There are few issues here:
- During upgrade from 2.4 to 2.5, finalize did not update current cluster
version. As a result config helpers have gone mad
- As a result of previous issue, some Configure tasks failed to execute
- During downgrade from 2.6 , looks like cluster entity DB state was not
consistent after config selection, so sometimes configs were not selected is
some cases. I managed to reproduce that only once,
it's a race condition that is very hard to catch/trace in debugger.
Diffs (updated)
-
ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/FinalizeUpgradeAction.java
bbff9fd813
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
9fd7663b96
Diff: https://reviews.apache.org/r/62571/diff/2/
Changes: https://reviews.apache.org/r/62571/diff/1-2/
Testing
---
mvn clean test
live test on cluster. Not 100% sure that the last issue is fixed, but probably
yes
Thanks,
Dmitro Lisnichenko
Re: Review Request 62413: Ambari upgrade don't delete stale constraints
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62413/
---
(Updated Sept. 26, 2017, 6:55 p.m.)
Review request for Ambari, Jonathan Hurley and Nate Cole.
Summary (updated)
-
Ambari upgrade don't delete stale constraints
Bugs: AMBARI-21998
https://issues.apache.org/jira/browse/AMBARI-21998
Repository: ambari
Description
---
During Ambari upgrade from 2.2.2 to 2.5 I discovered that some UNIQUE
constraints are definned without name in older Ambari versions, and then
altered by name in UpgadeCatalogs/newer DDLs.
For example:
Ambari 2.2.2
https://github.com/apache/ambari/blob/branch-2.2.2/ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql
{code} UNIQUE (ldap_user, user_name));{code}
Ambari-2.4.0
https://github.com/apache/ambari/blob/branch-2.4/ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql#L288
{code} CONSTRAINT UNQ_users_0 UNIQUE (user_name, user_type)){code}
As a result, during upgrade from Ambari 2.2.2 to Ambari 2.5.0, the stale unique
constraint with default name is not deleted.
That will reproduce on Postgres for any upgrade path that includes these
versions.
{code}ambari=# \d+ users
Table "ambari.users"
Column |Type | Modifiers
| Storage | Description
---+-++--+-
user_id | integer | not null
| plain|
principal_id | bigint | not null
| plain|
ldap_user | integer | not null default 0
| plain|
user_name | character varying(255) | not null
| extended |
create_time | timestamp without time zone | default now()
| plain|
user_password | character varying(255) |
| extended |
active| integer | not null default 1
| plain|
active_widget_layouts | character varying(1024) | default NULL::character
varying| extended |
user_type | character varying(255) | default
'LOCAL'::character varying | extended |
Indexes:
"users_pkey" PRIMARY KEY, btree (user_id)
"unq_users_0" UNIQUE, btree (user_name, user_type)
"users_ldap_user_key" UNIQUE, btree (ldap_user, user_name)
Foreign-key constraints:
"fk_users_principal_id" FOREIGN KEY (principal_id) REFERENCES
adminprincipal(principal_id)
Referenced by:
TABLE "members" CONSTRAINT "fk_members_user_id" FOREIGN KEY (user_id)
REFERENCES users(user_id)
Has OIDs: no
{code}
Now we have two constraints here
"unq_users_0" UNIQUE, btree (user_name, user_type)
"users_ldap_user_key" UNIQUE, btree (ldap_user, user_name) <- WRONG
Diffs
-
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
79c021c369
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog260Test.java
63e3840a5d
Diff: https://reviews.apache.org/r/62413/diff/2/
Testing (updated)
---
tests on live cluster
mvn clean test
Thanks,
Dmitro Lisnichenko
Re: Review Request 62413: (Preview) Ambari upgrade don't delete stale constraints
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62413/
---
(Updated Sept. 26, 2017, 4:24 p.m.)
Review request for Ambari, Jonathan Hurley and Nate Cole.
Changes
---
Rebased on latest 2.6, added test coverage. Unit tests are running
Bugs: AMBARI-21998
https://issues.apache.org/jira/browse/AMBARI-21998
Repository: ambari
Description
---
During Ambari upgrade from 2.2.2 to 2.5 I discovered that some UNIQUE
constraints are definned without name in older Ambari versions, and then
altered by name in UpgadeCatalogs/newer DDLs.
For example:
Ambari 2.2.2
https://github.com/apache/ambari/blob/branch-2.2.2/ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql
{code} UNIQUE (ldap_user, user_name));{code}
Ambari-2.4.0
https://github.com/apache/ambari/blob/branch-2.4/ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql#L288
{code} CONSTRAINT UNQ_users_0 UNIQUE (user_name, user_type)){code}
As a result, during upgrade from Ambari 2.2.2 to Ambari 2.5.0, the stale unique
constraint with default name is not deleted.
That will reproduce on Postgres for any upgrade path that includes these
versions.
{code}ambari=# \d+ users
Table "ambari.users"
Column |Type | Modifiers
| Storage | Description
---+-++--+-
user_id | integer | not null
| plain|
principal_id | bigint | not null
| plain|
ldap_user | integer | not null default 0
| plain|
user_name | character varying(255) | not null
| extended |
create_time | timestamp without time zone | default now()
| plain|
user_password | character varying(255) |
| extended |
active| integer | not null default 1
| plain|
active_widget_layouts | character varying(1024) | default NULL::character
varying| extended |
user_type | character varying(255) | default
'LOCAL'::character varying | extended |
Indexes:
"users_pkey" PRIMARY KEY, btree (user_id)
"unq_users_0" UNIQUE, btree (user_name, user_type)
"users_ldap_user_key" UNIQUE, btree (ldap_user, user_name)
Foreign-key constraints:
"fk_users_principal_id" FOREIGN KEY (principal_id) REFERENCES
adminprincipal(principal_id)
Referenced by:
TABLE "members" CONSTRAINT "fk_members_user_id" FOREIGN KEY (user_id)
REFERENCES users(user_id)
Has OIDs: no
{code}
Now we have two constraints here
"unq_users_0" UNIQUE, btree (user_name, user_type)
"users_ldap_user_key" UNIQUE, btree (ldap_user, user_name) <- WRONG
Diffs (updated)
-
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
79c021c369
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog260Test.java
63e3840a5d
Diff: https://reviews.apache.org/r/62413/diff/2/
Changes: https://reviews.apache.org/r/62413/diff/1-2/
Testing
---
tests on live cluster
Thanks,
Dmitro Lisnichenko
Review Request 62571: Fail to restart Ranger Admin during HDP downgrade.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62571/
---
Review request for Ambari, Jonathan Hurley and Nate Cole.
Bugs: AMBARI-22060
https://issues.apache.org/jira/browse/AMBARI-22060
Repository: ambari
Description
---
During the downgrade process, run into the following error whilst it's
restating Ranger Admin:
{code}
Traceback (most recent call last):
File
"/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py",
line 216, in
RangerAdmin().execute()
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 329, in execute
method(env)
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 850, in restart
self.start(env, upgrade_type=upgrade_type)
File
"/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py",
line 93, in start
setup_ranger_audit_solr()
File
"/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py",
line 705, in setup_ranger_audit_solr
new_service_principals = [params.ranger_admin_jaas_principal])
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/functions/solr_cloud_util.py",
line 329, in add_solr_roles
new_service_users.append(__remove_host_from_principal(new_service_user,
kerberos_realm))
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/functions/solr_cloud_util.py",
line 266, in __remove_host_from_principal
if not realm:
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py",
line 73, in __getattr__
raise Fail("Configuration parameter '" + self.name + "' was not found in
configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter
'kerberos-env' was not found in configurations dictionary!
{code}
The reason was that server did not have many configs selected, and did not send
them to agent during downgrade. There are few issues here:
- During upgrade from 2.4 to 2.5, finalize did not update current cluster
version. As a result config helpers have gone mad
- As a result of previous issue, some Configure tasks failed to execute
- During downgrade from 2.6 , looks like cluster entity DB state was not
consistent after config selection, so sometimes configs were not selected is
some cases. I managed to reproduce that only once,
it's a race condition that is very hard to catch/trace in debugger.
Diffs
-
ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/FinalizeUpgradeAction.java
bbff9fd813
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
9fd7663b96
Diff: https://reviews.apache.org/r/62571/diff/1/
Testing
---
mvn clean test
live test on cluster. Not 100% sure that the last issue is fixed, but probably
yes
Thanks,
Dmitro Lisnichenko
Re: Review Request 62467: Add UID/GID related issue with external users not listed in /etc/passwd
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62467/#review185902 --- Ship it! ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/shared_initialization.py Line 159 (original), 158 (patched) <https://reviews.apache.org/r/62467/#comment262241> how about adding comment with meaning of None? - Dmitro Lisnichenko On Sept. 21, 2017, 6:32 p.m., Eugene Chekanskiy wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62467/ > --- > > (Updated Sept. 21, 2017, 6:32 p.m.) > > > Review request for Ambari, Dmitro Lisnichenko and Robert Levas. > > > Bugs: AMBARI-22027 > https://issues.apache.org/jira/browse/AMBARI-22027 > > > Repository: ambari > > > Description > --- > > Why this works? > If there is no custom uids it will use system-defined uid(behaivor before > GID\UID feature) and will not try to use UID that occupied by external > systems, like ldap. > > > Diffs > - > > > ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/shared_initialization.py > ee950e8 > > ambari-server/src/test/python/stacks/2.0.6/hooks/before-ANY/test_before_any.py > a13ac24 > > > Diff: https://reviews.apache.org/r/62467/diff/1/ > > > Testing > --- > > mvn clean test, cluster deployment. > > > Thanks, > > Eugene Chekanskiy > >
Re: Review Request 62468: OOZIE/OOZIE_SERVER restart failed saying AttributeError: 'NoneType' object has no attribute 'upper' during EU
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62468/#review185901 --- Ship it! Ship It! - Dmitro Lisnichenko On Sept. 21, 2017, 6:56 p.m., Nate Cole wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62468/ > --- > > (Updated Sept. 21, 2017, 6:56 p.m.) > > > Review request for Ambari, Dmitro Lisnichenko and Sid Wagle. > > > Bugs: AMBARI-22030 > https://issues.apache.org/jira/browse/AMBARI-22030 > > > Repository: ambari > > > Description > --- > > A code path that was previously added is not working with new structures. > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/upgrade_summary.py > 507bc42985 > > ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py > 2f474a186a > ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py > dfa22fd07b > > > Diff: https://reviews.apache.org/r/62468/diff/1/ > > > Testing > --- > > Manual. Python change only unit tests: > > -- > Total run:1192 > Total errors:0 > Total failures:0 > > > Thanks, > > Nate Cole > >
Re: Review Request 62454: Could not copy tez tarball to HDFS during prepare upgrade (EU)
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62454/#review185866
---
Ship it!
Ship It!
- Dmitro Lisnichenko
On Sept. 20, 2017, 11:51 p.m., Nate Cole wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62454/
> ---
>
> (Updated Sept. 20, 2017, 11:51 p.m.)
>
>
> Review request for Ambari, Dmitro Lisnichenko and Jonathan Hurley.
>
>
> Bugs: AMBARI-22011
> https://issues.apache.org/jira/browse/AMBARI-22011
>
>
> Repository: ambari
>
>
> Description
> ---
>
> The lookup of versions was failing because a property is no longer being
> passed. Use the UpgradeSummary structure to check for versions. New
> behavior is consistent with older versions of Ambari.
>
> 2017-09-19 11:42:05,742 - Before starting Stack Upgrade, check if tez tarball
> has been copied to HDFS.
> 2017-09-19 11:42:05,742 - Stack version 2.6.0.0 is sufficient to check if
> need to copy tez.tar.gz to HDFS.
> 2017-09-19 11:42:05,742 - Called copy_to_hdfs tarball: tez
> 2017-09-19 11:42:05,743 - Stack Feature Version Info: Cluster Stack=2.6,
> Command Stack=None, Command Version=2.5.3.0-37, Upgrade Direction=upgrade ->
> 2.5.3.0-37
> 2017-09-19 11:42:05,743 - Skipping stack-select on RU_EXECUTE_TASKS because
> it does not exist in the stack-select package structure.
> 2017-09-19 11:42:05,743 - Unable to determine the current version of the
> component for this command; unable to copy the tarball
> 2017-09-19 11:42:05,743 - Cannot copy tez tarball to HDFS because stack
> version could be be determined.
> 2017-09-19 11:42:05,743 - Cannot copy tez tarball to HDFS because stack
> version could be be determined.
> 2017-09-19 11:42:05,744 - Could not copy tarball tez due to a missing or
> incorrect parameter.
> 2017-09-19 11:42:05,744 - Could not copy tarball tez due to a missing or
> incorrect parameter.
> Traceback (most recent call last):
> File
> "/var/lib/ambari-agent/cache/common-services/TEZ/0.4.0.2.1/package/scripts/pre_upgrade.py",
> line 58, in
> TezPreUpgrade().execute()
> File
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
> line 327, in execute
> method(env)
> File
> "/var/lib/ambari-agent/cache/common-services/TEZ/0.4.0.2.1/package/scripts/pre_upgrade.py",
> line 55, in prepare
> raise Fail("Could not copy tez tarball to HDFS.")
> resource_management.core.exceptions.Fail: Could not copy tez tarball to HDFS.
>
>
> Diffs
> -
>
>
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
> e0dd96de1e
>
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py
> 64e1cc1c94
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/OOZIE/package/scripts/oozie.py
> 8bdfe46230
>
>
> Diff: https://reviews.apache.org/r/62454/diff/2/
>
>
> Testing
> ---
>
> Manual EU including Tez succeeded. We have no direct tests of copy_tarballs
> so only manual verification.
>
>
> Thanks,
>
> Nate Cole
>
>
Re: Review Request 62448: Addition of service component after patching a service still keeps the component at base version
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62448/#review185865
---
Ship it!
Ship It!
- Dmitro Lisnichenko
On Sept. 20, 2017, 10:59 p.m., Jonathan Hurley wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62448/
> ---
>
> (Updated Sept. 20, 2017, 10:59 p.m.)
>
>
> Review request for Ambari, Dmitro Lisnichenko and Nate Cole.
>
>
> Bugs: AMBARI-22007
> https://issues.apache.org/jira/browse/AMBARI-22007
>
>
> Repository: ambari
>
>
> Description
> ---
>
> This bug will affect adding components to hosts where other versions of the
> component was already installed. It occurs because Storm was already
> installed on the machine you added the Supervisor to and the symlinks were
> already created with the older version.
>
> Example:
> {code:title=After Patch, Before Adding Supervisor}
> /usr/hdp/current/storm-client -> /usr/hdp/2.6.0.3-8/storm
> /usr/hdp/current/storm-nimbus -> /usr/hdp/2.6.0.3-8/storm
> /usr/hdp/current/storm-supervisor -> /usr/hdp/2.6.0.0-102/storm
> {code}
>
> The installation of any Storm component causes the symlink for
> {{storm-supervisor}} to also get created. However, when the upgrade was
> applied, it didn't upgrade supervisor on this machine since it technically
> wasn't installed there (at least as far as Ambari is concerned).
>
> Any other test which follows this pattern will fail until I can fix this.
> However, adding a branch new host should work correctly and install the new
> version.
>
>
> Diffs
> -
>
>
> ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py
> 9fbb42bec5
>
> ambari-common/src/main/python/resource_management/libraries/script/script.py
> 59401669f8
>
> ambari-server/src/main/java/org/apache/ambari/server/agent/CommandRepository.java
> 135171391f
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
> 3c15d92948
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RepositoryVersionResourceProvider.java
> e1da8131ea
>
> ambari-server/src/main/java/org/apache/ambari/server/events/listeners/upgrade/DistributeRepositoriesActionListener.java
> aa298944fe
>
> ambari-server/src/main/java/org/apache/ambari/server/events/listeners/upgrade/StackVersionListener.java
> b812476d31
>
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/RepositoryVersionEntity.java
> 72269f0f7c
>
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/RepositoryVersionHelper.java
> 3ab28e9a5c
>
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
> afd3d1d6a5
> ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql 796d9b2d7a
> ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql c0026013c0
> ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql d78c1c78c6
> ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 90f1397b94
> ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql
> c4bb76314b
> ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 9f7aa912ec
>
> ambari-server/src/test/java/org/apache/ambari/server/events/listeners/upgrade/StackVersionListenerTest.java
> cd8b499e1e
>
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog260Test.java
> ddff0e9560
> ambari-server/src/test/python/TestVersionSelectUtil.py 38798e2b70
>
>
> Diff: https://reviews.apache.org/r/62448/diff/3/
>
>
> Testing
> ---
>
> mvn clean test
>
> Patch Upgrade follow by adding components
> New cluster deployment via UI using ambiguous repo
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 62413: (Preview) Ambari upgrade don't delete stale constraints
> On Sept. 19, 2017, 11:44 p.m., Jonathan Hurley wrote:
> > Do you plan to include the ability to extract constraint names from tables
> > in this review? It's marked currently as preview...
Yes. It's not a very simple thing, just wanted to make sure that we want it
(otherwise, current patch would be enough)
- Dmitro
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62413/#review185731
---
On Sept. 19, 2017, 7:17 p.m., Dmitro Lisnichenko wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62413/
> ---
>
> (Updated Sept. 19, 2017, 7:17 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley and Nate Cole.
>
>
> Bugs: AMBARI-21998
> https://issues.apache.org/jira/browse/AMBARI-21998
>
>
> Repository: ambari
>
>
> Description
> ---
>
> During Ambari upgrade from 2.2.2 to 2.5 I discovered that some UNIQUE
> constraints are definned without name in older Ambari versions, and then
> altered by name in UpgadeCatalogs/newer DDLs.
>
> For example:
> Ambari 2.2.2
> https://github.com/apache/ambari/blob/branch-2.2.2/ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql
> {code} UNIQUE (ldap_user, user_name));{code}
> Ambari-2.4.0
> https://github.com/apache/ambari/blob/branch-2.4/ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql#L288
> {code} CONSTRAINT UNQ_users_0 UNIQUE (user_name, user_type)){code}
>
> As a result, during upgrade from Ambari 2.2.2 to Ambari 2.5.0, the stale
> unique constraint with default name is not deleted.
> That will reproduce on Postgres for any upgrade path that includes these
> versions.
>
> {code}ambari=# \d+ users
> Table "ambari.users"
> Column |Type | Modifiers
> | Storage | Description
> ---+-++--+-
> user_id | integer | not null
>| plain|
> principal_id | bigint | not null
>| plain|
> ldap_user | integer | not null default 0
>| plain|
> user_name | character varying(255) | not null
>| extended |
> create_time | timestamp without time zone | default now()
>| plain|
> user_password | character varying(255) |
>| extended |
> active| integer | not null default 1
>| plain|
> active_widget_layouts | character varying(1024) | default NULL::character
> varying| extended |
> user_type | character varying(255) | default
> 'LOCAL'::character varying | extended |
> Indexes:
> "users_pkey" PRIMARY KEY, btree (user_id)
> "unq_users_0" UNIQUE, btree (user_name, user_type)
> "users_ldap_user_key" UNIQUE, btree (ldap_user, user_name)
> Foreign-key constraints:
> "fk_users_principal_id" FOREIGN KEY (principal_id) REFERENCES
> adminprincipal(principal_id)
> Referenced by:
> TABLE "members" CONSTRAINT "fk_members_user_id" FOREIGN KEY (user_id)
> REFERENCES users(user_id)
> Has OIDs: no
>
> {code}
>
> Now we have two constraints here
> "unq_users_0" UNIQUE, btree (user_name, user_type)
> "users_ldap_user_key" UNIQUE, btree (ldap_user, user_name) <- WRONG
>
>
> Diffs
> -
>
>
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
> afd3d1d6a5
>
>
> Diff: https://reviews.apache.org/r/62413/diff/1/
>
>
> Testing
> ---
>
> tests on live cluster
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 62413: (Preview) Ambari upgrade don't delete stale constraints
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62413/#review185701
---
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
Lines 200 (patched)
<https://reviews.apache.org/r/62413/#comment262013>
This PK constraint has been given a name in DDL for 2.6
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
Lines 208 (patched)
<https://reviews.apache.org/r/62413/#comment262012>
That is a minimal hotfix for current issue. I have to hardcode generated
constraint name, because we have no utility to extract existing constraints for
table.
As a more general fix, I'm trying to implement utility methods that would
fetch
- name(s) of unique constraints that target given columns
- name of primary key constraint for table
- names of foreign key constraints for table
- Dmitro Lisnichenko
On Sept. 19, 2017, 7:17 p.m., Dmitro Lisnichenko wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62413/
> ---
>
> (Updated Sept. 19, 2017, 7:17 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley and Nate Cole.
>
>
> Bugs: AMBARI-21998
> https://issues.apache.org/jira/browse/AMBARI-21998
>
>
> Repository: ambari
>
>
> Description
> ---
>
> During Ambari upgrade from 2.2.2 to 2.5 I discovered that some UNIQUE
> constraints are definned without name in older Ambari versions, and then
> altered by name in UpgadeCatalogs/newer DDLs.
>
> For example:
> Ambari 2.2.2
> https://github.com/apache/ambari/blob/branch-2.2.2/ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql
> {code} UNIQUE (ldap_user, user_name));{code}
> Ambari-2.4.0
> https://github.com/apache/ambari/blob/branch-2.4/ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql#L288
> {code} CONSTRAINT UNQ_users_0 UNIQUE (user_name, user_type)){code}
>
> As a result, during upgrade from Ambari 2.2.2 to Ambari 2.5.0, the stale
> unique constraint with default name is not deleted.
> That will reproduce on Postgres for any upgrade path that includes these
> versions.
>
> {code}ambari=# \d+ users
> Table "ambari.users"
> Column |Type | Modifiers
> | Storage | Description
> ---+-++--+-
> user_id | integer | not null
>| plain|
> principal_id | bigint | not null
>| plain|
> ldap_user | integer | not null default 0
>| plain|
> user_name | character varying(255) | not null
>| extended |
> create_time | timestamp without time zone | default now()
>| plain|
> user_password | character varying(255) |
>| extended |
> active| integer | not null default 1
>| plain|
> active_widget_layouts | character varying(1024) | default NULL::character
> varying| extended |
> user_type | character varying(255) | default
> 'LOCAL'::character varying | extended |
> Indexes:
> "users_pkey" PRIMARY KEY, btree (user_id)
> "unq_users_0" UNIQUE, btree (user_name, user_type)
> "users_ldap_user_key" UNIQUE, btree (ldap_user, user_name)
> Foreign-key constraints:
> "fk_users_principal_id" FOREIGN KEY (principal_id) REFERENCES
> adminprincipal(principal_id)
> Referenced by:
> TABLE "members" CONSTRAINT "fk_members_user_id" FOREIGN KEY (user_id)
> REFERENCES users(user_id)
> Has OIDs: no
>
> {code}
>
> Now we have two constraints here
> "unq_users_0" UNIQUE, btree (user_name, user_type)
> "users_ldap_user_key" UNIQUE, btree (ldap_user, user_name) <- WRONG
>
>
> Diffs
> -
>
>
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
> afd3d1d6a5
>
>
> Diff: https://reviews.apache.org/r/62413/diff/1/
>
>
> Testing
> ---
>
> tests on live cluster
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Review Request 62413: (Preview) Ambari upgrade don't delete stale constraints
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62413/
---
Review request for Ambari, Jonathan Hurley and Nate Cole.
Bugs: AMBARI-21998
https://issues.apache.org/jira/browse/AMBARI-21998
Repository: ambari
Description
---
During Ambari upgrade from 2.2.2 to 2.5 I discovered that some UNIQUE
constraints are definned without name in older Ambari versions, and then
altered by name in UpgadeCatalogs/newer DDLs.
For example:
Ambari 2.2.2
https://github.com/apache/ambari/blob/branch-2.2.2/ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql
{code} UNIQUE (ldap_user, user_name));{code}
Ambari-2.4.0
https://github.com/apache/ambari/blob/branch-2.4/ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql#L288
{code} CONSTRAINT UNQ_users_0 UNIQUE (user_name, user_type)){code}
As a result, during upgrade from Ambari 2.2.2 to Ambari 2.5.0, the stale unique
constraint with default name is not deleted.
That will reproduce on Postgres for any upgrade path that includes these
versions.
{code}ambari=# \d+ users
Table "ambari.users"
Column |Type | Modifiers
| Storage | Description
---+-++--+-
user_id | integer | not null
| plain|
principal_id | bigint | not null
| plain|
ldap_user | integer | not null default 0
| plain|
user_name | character varying(255) | not null
| extended |
create_time | timestamp without time zone | default now()
| plain|
user_password | character varying(255) |
| extended |
active| integer | not null default 1
| plain|
active_widget_layouts | character varying(1024) | default NULL::character
varying| extended |
user_type | character varying(255) | default
'LOCAL'::character varying | extended |
Indexes:
"users_pkey" PRIMARY KEY, btree (user_id)
"unq_users_0" UNIQUE, btree (user_name, user_type)
"users_ldap_user_key" UNIQUE, btree (ldap_user, user_name)
Foreign-key constraints:
"fk_users_principal_id" FOREIGN KEY (principal_id) REFERENCES
adminprincipal(principal_id)
Referenced by:
TABLE "members" CONSTRAINT "fk_members_user_id" FOREIGN KEY (user_id)
REFERENCES users(user_id)
Has OIDs: no
{code}
Now we have two constraints here
"unq_users_0" UNIQUE, btree (user_name, user_type)
"users_ldap_user_key" UNIQUE, btree (ldap_user, user_name) <- WRONG
Diffs
-
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
afd3d1d6a5
Diff: https://reviews.apache.org/r/62413/diff/1/
Testing
---
tests on live cluster
Thanks,
Dmitro Lisnichenko
Re: Review Request 62388: APPLICATION_TIMELINE_SERVER filed to start, coz were already started
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62388/#review185594 --- ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/service.py Lines 88 (patched) <https://reviews.apache.org/r/62388/#comment261898> Sounds like a very dirty hack. Does that reproduce in latest Ambari versions? Maybe it would be better to check status() before attempt to run start() ? - Dmitro Lisnichenko On Sept. 18, 2017, 9 p.m., Eugene Chekanskiy wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62388/ > --- > > (Updated Sept. 18, 2017, 9 p.m.) > > > Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, and Robert > Levas. > > > Bugs: AMBARI-21984 > https://issues.apache.org/jira/browse/AMBARI-21984 > > > Repository: ambari > > > Description > --- > > Sometimes scripts starting yarn services but they are already running. It is > rarely reproduced and hard to debug. > This patch checks output of start command and ignore start failure in case > script reporting "service is running" and validates pid pile after that. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/service.py > 7c59b60 > > ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/package/scripts/service.py > 1c612ef > ambari-server/src/test/python/stacks/2.0.6/YARN/test_historyserver.py > 6a99dfa > ambari-server/src/test/python/stacks/2.0.6/YARN/test_nodemanager.py 642043d > ambari-server/src/test/python/stacks/2.0.6/YARN/test_resourcemanager.py > 8a703df > ambari-server/src/test/python/stacks/2.1/YARN/test_apptimelineserver.py > b352920 > > > Diff: https://reviews.apache.org/r/62388/diff/1/ > > > Testing > --- > > mvn clean test, cluster deploy > > > Thanks, > > Eugene Chekanskiy > >
