Re: Review Request 48044: Provide context for hdp-select failures during ambari component install
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48044/ --- (Updated June 7, 2016, 8:17 a.m.) Review request for Ambari, Andrew Onischuk, Oliver Szabo, Sumit Mohanty, and Sebastian Toader. Changes --- Reverted the last version of the patch. Bugs: AMBARI-16952 https://issues.apache.org/jira/browse/AMBARI-16952 Repository: ambari Description (updated) --- In case the hdp-select command fails during a service / component installation there's no contextual information about the cause of the failure. This issue is for logging information about the machine on which the hdp-select command fails. This solution wraps hdp-select command calls in a try/catch block and logs failure / hdp installationrelated information. The patch only applies for 2.2-next versions. Diffs (updated) - ambari-common/src/main/python/resource_management/libraries/functions/hdp_select.py 9a3201e Diff: https://reviews.apache.org/r/48044/diff/ Testing --- Unit tests passed. Manual testing underway. Thanks, Laszlo Puskas
Re: Review Request 48044: Provide context for hdp-select failures during ambari component install
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48044/ --- (Updated June 7, 2016, 9:18 a.m.) Review request for Ambari, Andrew Onischuk, Oliver Szabo, Sumit Mohanty, and Sebastian Toader. Changes --- Applied review notes. Bugs: AMBARI-16952 https://issues.apache.org/jira/browse/AMBARI-16952 Repository: ambari Description --- In case the hdp-select command fails during a service / component installation there's no contextual information about the cause of the failure. This issue is for logging information about the machine on which the hdp-select command fails. This solution wraps hdp-select command calls in a try/catch block and logs failure / hdp installationrelated information. The patch only applies for 2.2-next versions. Diffs (updated) - ambari-common/src/main/python/resource_management/libraries/functions/hdp_select.py 9a3201e Diff: https://reviews.apache.org/r/48044/diff/ Testing (updated) --- Unit tests passed. Manual testing underway. // need to verify if ls is configured with sudo on agents. Thanks, Laszlo Puskas
Re: Review Request 48044: Provide context for hdp-select failures during ambari component install
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48044/ --- (Updated June 7, 2016, 8:02 a.m.) Review request for Ambari, Andrew Onischuk, Oliver Szabo, Sumit Mohanty, and Sebastian Toader. Bugs: AMBARI-16952 https://issues.apache.org/jira/browse/AMBARI-16952 Repository: ambari Description --- In case the hdp-select command fails during a service / component installation there's no contextual information about the cause of the failure. This issue is for logging information about the machine on which the hdp-select command fails. This solution wraps hdp-select command calls in a try/catch block and logs failure / hdp installationrelated information. Diffs - ambari-common/src/main/python/resource_management/libraries/functions/hdp_select.py 9a3201e Diff: https://reviews.apache.org/r/48044/diff/ Testing (updated) --- Unit tests passed. Manual testing underway. Thanks, Laszlo Puskas
Re: Review Request 48044: Provide context for hdp-select failures during ambari component install
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48044/ --- (Updated June 7, 2016, 8:03 a.m.) Review request for Ambari, Andrew Onischuk, Oliver Szabo, Sumit Mohanty, and Sebastian Toader. Bugs: AMBARI-16952 https://issues.apache.org/jira/browse/AMBARI-16952 Repository: ambari Description --- In case the hdp-select command fails during a service / component installation there's no contextual information about the cause of the failure. This issue is for logging information about the machine on which the hdp-select command fails. This solution wraps hdp-select command calls in a try/catch block and logs failure / hdp installationrelated information. Diffs - ambari-common/src/main/python/resource_management/libraries/functions/hdp_select.py 9a3201e Diff: https://reviews.apache.org/r/48044/diff/ Testing --- Unit tests passed. Manual testing underway. Thanks, Laszlo Puskas
Re: Review Request 48044: Provide context for hdp-select failures during ambari component install
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48044/ --- (Updated June 7, 2016, 12:14 p.m.) Review request for Ambari, Andrew Onischuk, Oliver Szabo, Sumit Mohanty, and Sebastian Toader. Changes --- Removed sudo from ls command executions. Bugs: AMBARI-16952 https://issues.apache.org/jira/browse/AMBARI-16952 Repository: ambari Description --- In case the hdp-select command fails during a service / component installation there's no contextual information about the cause of the failure. This issue is for logging information about the machine on which the hdp-select command fails. This solution wraps hdp-select command calls in a try/catch block and logs failure / hdp installationrelated information. The patch only applies for 2.2-next versions. Diffs (updated) - ambari-common/src/main/python/resource_management/libraries/functions/hdp_select.py 9a3201e Diff: https://reviews.apache.org/r/48044/diff/ Testing --- Unit tests passed. Manual testing underway. // need to verify if ls is configured with sudo on agents. Thanks, Laszlo Puskas
Re: Review Request 48044: Provide context for hdp-select failures during ambari component install
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48044/ --- (Updated June 7, 2016, 12:15 p.m.) Review request for Ambari, Andrew Onischuk, Oliver Szabo, Sumit Mohanty, and Sebastian Toader. Bugs: AMBARI-16952 https://issues.apache.org/jira/browse/AMBARI-16952 Repository: ambari Description --- In case the hdp-select command fails during a service / component installation there's no contextual information about the cause of the failure. This issue is for logging information about the machine on which the hdp-select command fails. This solution wraps hdp-select command calls in a try/catch block and logs failure / hdp installationrelated information. The patch only applies for 2.2-next versions. Diffs - ambari-common/src/main/python/resource_management/libraries/functions/hdp_select.py 9a3201e Diff: https://reviews.apache.org/r/48044/diff/ Testing (updated) --- Unit tests passed. Thanks, Laszlo Puskas
Re: Review Request 48266: Add explicit ambari-server log line indicating cluster creation complete
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48266/#review136244 --- ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java (line 162) <https://reviews.apache.org/r/48266/#comment201269> I'd rather perform this check out of this method, in order to always have a boolean return value for this method. - Laszlo Puskas On June 6, 2016, 8:47 a.m., Daniel Gergely wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48266/ > --- > > (Updated June 6, 2016, 8:47 a.m.) > > > Review request for Ambari, Laszlo Puskas, Oliver Szabo, Robert Nettleton, > Sandor Magyari, Sumit Mohanty, and Sebastian Toader. > > > Bugs: AMBARI-17053 > https://issues.apache.org/jira/browse/AMBARI-17053 > > > Repository: ambari > > > Description > --- > > Add a log message to see when the cluster is ready to use (cluster creation > finishes). > > An event after each heartbeat. At that time cluster provision request is > checked if it is finished or not. In case of an ambari server restart, the > replayed requests are used to determine which one was the provision request. > I could not find a nice way to do it, but I could make a workaround. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatProcessor.java > c6036c2 > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ProvisionClusterRequest.java > 3feac55 > > ambari-server/src/main/java/org/apache/ambari/server/events/AmbariEvent.java > 1079806 > > ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java > e3f5b49 > > ambari-server/src/test/java/org/apache/ambari/server/topology/TopologyManagerTest.java > fd8653c > > Diff: https://reviews.apache.org/r/48266/diff/ > > > Testing > --- > > Still running locally... > > > Thanks, > > Daniel Gergely > >
Re: Review Request 48044: Provide context for hdp-select failures during ambari component install
> On June 2, 2016, 4:23 p.m., Alejandro Fernandez wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/hdp_select.py,
> > line 129
> > <https://reviews.apache.org/r/48044/diff/1/?file=1400993#file1400993line129>
> >
> > This cannot assume the directory will be /usr/hdp, instead use
> > stack_root variable.
>
> Sumit Mohanty wrote:
> Looks like different patches are needed for branch-2.4 and branch-2.2.
> Starting 2.4 {stack_root}/{version_to_select} can be used and for the older
> version it can be /usr/hdp.
Using the stack_root to the log commands.
- Laszlo
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48044/#review135943
-------
On June 6, 2016, 2:48 p.m., Laszlo Puskas wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48044/
> ---
>
> (Updated June 6, 2016, 2:48 p.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Oliver Szabo, Sumit Mohanty, and
> Sebastian Toader.
>
>
> Bugs: AMBARI-16952
> https://issues.apache.org/jira/browse/AMBARI-16952
>
>
> Repository: ambari
>
>
> Description
> ---
>
> In case the hdp-select command fails during a service / component
> installation there's no contextual information about the cause of the failure.
> This issue is for logging information about the machine on which the
> hdp-select command fails.
> This solution wraps hdp-select command calls in a try/catch block and logs
> failure / hdp installationrelated information.
>
>
> Diffs
> -
>
>
> ambari-common/src/main/python/resource_management/libraries/functions/hdp_select.py
> 9a3201e
>
> Diff: https://reviews.apache.org/r/48044/diff/
>
>
> Testing
> ---
>
> Manual testing underway.
>
>
> Thanks,
>
> Laszlo Puskas
>
>
Re: Review Request 48044: Provide context for hdp-select failures during ambari component install
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48044/ --- (Updated June 6, 2016, 2:48 p.m.) Review request for Ambari, Andrew Onischuk, Oliver Szabo, Sumit Mohanty, and Sebastian Toader. Changes --- Added stack_root variable Bugs: AMBARI-16952 https://issues.apache.org/jira/browse/AMBARI-16952 Repository: ambari Description --- In case the hdp-select command fails during a service / component installation there's no contextual information about the cause of the failure. This issue is for logging information about the machine on which the hdp-select command fails. This solution wraps hdp-select command calls in a try/catch block and logs failure / hdp installationrelated information. Diffs (updated) - ambari-common/src/main/python/resource_management/libraries/functions/hdp_select.py 9a3201e Diff: https://reviews.apache.org/r/48044/diff/ Testing --- Manual testing underway. Thanks, Laszlo Puskas
Re: Review Request 48044: Provide context for hdp-select failures during ambari component install
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48044/ --- (Updated June 6, 2016, 3:01 p.m.) Review request for Ambari, Andrew Onischuk, Oliver Szabo, Sumit Mohanty, and Sebastian Toader. Bugs: AMBARI-16952 https://issues.apache.org/jira/browse/AMBARI-16952 Repository: ambari Description --- In case the hdp-select command fails during a service / component installation there's no contextual information about the cause of the failure. This issue is for logging information about the machine on which the hdp-select command fails. This solution wraps hdp-select command calls in a try/catch block and logs failure / hdp installationrelated information. Diffs - ambari-common/src/main/python/resource_management/libraries/functions/hdp_select.py 9a3201e Diff: https://reviews.apache.org/r/48044/diff/ Testing (updated) --- Unit tests passed. Manual testing underway please don't commit yet. Thanks, Laszlo Puskas
Re: Review Request 48044: Provide context for hdp-select failures during ambari component install
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48044/ --- (Updated June 6, 2016, 3 p.m.) Review request for Ambari, Andrew Onischuk, Oliver Szabo, Sumit Mohanty, and Sebastian Toader. Bugs: AMBARI-16952 https://issues.apache.org/jira/browse/AMBARI-16952 Repository: ambari Description --- In case the hdp-select command fails during a service / component installation there's no contextual information about the cause of the failure. This issue is for logging information about the machine on which the hdp-select command fails. This solution wraps hdp-select command calls in a try/catch block and logs failure / hdp installationrelated information. Diffs - ambari-common/src/main/python/resource_management/libraries/functions/hdp_select.py 9a3201e Diff: https://reviews.apache.org/r/48044/diff/ Testing (updated) --- Unit tests passed. Manual testing underway. Thanks, Laszlo Puskas
Re: Review Request 48044: Provide context for hdp-select failures during ambari component install
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48044/ --- (Updated June 6, 2016, 3:15 p.m.) Review request for Ambari, Andrew Onischuk, Oliver Szabo, Sumit Mohanty, and Sebastian Toader. Bugs: AMBARI-16952 https://issues.apache.org/jira/browse/AMBARI-16952 Repository: ambari Description --- In case the hdp-select command fails during a service / component installation there's no contextual information about the cause of the failure. This issue is for logging information about the machine on which the hdp-select command fails. This solution wraps hdp-select command calls in a try/catch block and logs failure / hdp installationrelated information. Diffs - ambari-common/src/main/python/resource_management/libraries/functions/hdp_select.py 9a3201e Diff: https://reviews.apache.org/r/48044/diff/ Testing (updated) --- Unit tests passed. Manual testing underway please don't commit yet. // found some major differences between the branches / investigating it Thanks, Laszlo Puskas
Re: Review Request 48044: Provide context for hdp-select failures during ambari component install
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48044/ --- (Updated June 13, 2016, 9:22 a.m.) Review request for Ambari, Andrew Onischuk, Jayush Luniya, Sandor Magyari, Sumit Mohanty, and Sebastian Toader. Changes --- Added Jayush. Bugs: AMBARI-16952 https://issues.apache.org/jira/browse/AMBARI-16952 Repository: ambari Description (updated) --- In case the hdp-select command fails during a service / component installation there's no contextual information about the cause of the failure. This issue is for logging information about the machine on which the hdp-select command fails. This solution wraps hdp-select command calls in a try/catch block and logs failure / hdp installation related information. The patch only applies for 2.2-next versions. Based on the git logs the affected codebase has been refactored as part of AMBARI-15329. (This addition should be ported to the newer versions isf applicable) Diffs - ambari-common/src/main/python/resource_management/libraries/functions/hdp_select.py 9a3201e Diff: https://reviews.apache.org/r/48044/diff/ Testing --- Unit tests passed. Thanks, Laszlo Puskas
Re: Review Request 48044: Provide context for hdp-select failures during ambari component install
> On June 7, 2016, 6:49 p.m., Alejandro Fernandez wrote: > > ambari-common/src/main/python/resource_management/libraries/functions/hdp_select.py, > > line 130 > > <https://reviews.apache.org/r/48044/diff/6/?file=1408744#file1408744line130> > > > > Please consult with Jayush Luniya. > > The stack already has a config for the stack root, so hdp_select (or > > distro select as it should be called) shouldn't be hardcoding paths. This patch is for 2.2-next branch that doesn't have those improvements. (Tried to do this following one of your previous notes - seems that those weren't backported to this branch) - Laszlo --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48044/#review136517 ------- On June 13, 2016, 8:28 a.m., Laszlo Puskas wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48044/ > --- > > (Updated June 13, 2016, 8:28 a.m.) > > > Review request for Ambari, Andrew Onischuk, Daniel Gergely, Sandor Magyari, > Sumit Mohanty, and Sebastian Toader. > > > Bugs: AMBARI-16952 > https://issues.apache.org/jira/browse/AMBARI-16952 > > > Repository: ambari > > > Description > --- > > In case the hdp-select command fails during a service / component > installation there's no contextual information about the cause of the failure. > This issue is for logging information about the machine on which the > hdp-select command fails. > This solution wraps hdp-select command calls in a try/catch block and logs > failure / hdp installation related information. > > The patch only applies for 2.2-next versions. > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/hdp_select.py > 9a3201e > > Diff: https://reviews.apache.org/r/48044/diff/ > > > Testing > --- > > Unit tests passed. > > > Thanks, > > Laszlo Puskas > >
Re: Review Request 48044: Provide context for hdp-select failures during ambari component install
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48044/ --- (Updated June 13, 2016, 8:28 a.m.) Review request for Ambari, Andrew Onischuk, Daniel Gergely, Oliver Szabo, Sandor Magyari, Sumit Mohanty, and Sebastian Toader. Changes --- Applied review notes. Bugs: AMBARI-16952 https://issues.apache.org/jira/browse/AMBARI-16952 Repository: ambari Description --- In case the hdp-select command fails during a service / component installation there's no contextual information about the cause of the failure. This issue is for logging information about the machine on which the hdp-select command fails. This solution wraps hdp-select command calls in a try/catch block and logs failure / hdp installation related information. The patch only applies for 2.2-next versions. Diffs (updated) - ambari-common/src/main/python/resource_management/libraries/functions/hdp_select.py 9a3201e Diff: https://reviews.apache.org/r/48044/diff/ Testing --- Unit tests passed. Thanks, Laszlo Puskas
Re: Review Request 48691: Removing and re-adding hosts makes database inconsitent
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48691/#review137527 --- Ship it! Ship It! - Laszlo Puskas On June 14, 2016, 2:51 p.m., Daniel Gergely wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48691/ > --- > > (Updated June 14, 2016, 2:51 p.m.) > > > Review request for Ambari, Laszlo Puskas, Robert Nettleton, Sandor Magyari, > Sumit Mohanty, and Sebastian Toader. > > > Bugs: AMBARI-17219 > https://issues.apache.org/jira/browse/AMBARI-17219 > > > Repository: ambari > > > Description > --- > > When a host is removed, it is necessary to have all the components stopped > and removed, before the host itself can be deleted. > If a host component is not stopped, it is not allowed to be removed, so the > host itself cannot be removed. However if re-adding the host is tried, even > if it is still in the cluster, a row is inserted to topology_host_info table. > Multiple lines of the same host in that table is incorrect, so topology > validation fails. > Expected behaviour: if host is still in the cluster, a duplicate should not > be able to be persisted in any of the related tables. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java > e3f5b49 > > ambari-server/src/test/java/org/apache/ambari/server/topology/TopologyManagerTest.java > fd8653c > > Diff: https://reviews.apache.org/r/48691/diff/ > > > Testing > --- > > > Thanks, > > Daniel Gergely > >
Re: Review Request 48266: Add explicit ambari-server log line indicating cluster creation complete
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48266/#review137749 --- ambari-server/src/main/java/org/apache/ambari/server/topology/PersistedStateImpl.java (line 149) <https://reviews.apache.org/r/48266/#comment202952> Checking the clusterId here is superfluous as the finder should return only those entities that have this clusterId. ambari-server/src/test/java/org/apache/ambari/server/topology/TopologyManagerTest.java (line 87) <https://reviews.apache.org/r/48266/#comment202953> The member annotated with @TestSubject is instantiated by the framework. Why do we need two instances here? - Laszlo Puskas On June 15, 2016, 11:33 a.m., Daniel Gergely wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48266/ > --- > > (Updated June 15, 2016, 11:33 a.m.) > > > Review request for Ambari, Laszlo Puskas, Oliver Szabo, Robert Nettleton, > Sandor Magyari, Sumit Mohanty, and Sebastian Toader. > > > Bugs: AMBARI-17053 > https://issues.apache.org/jira/browse/AMBARI-17053 > > > Repository: ambari > > > Description > --- > > Add a log message to see when the cluster is ready to use (cluster creation > finishes). > > An event after each heartbeat. At that time cluster provision request is > checked if it is finished or not. In case of an ambari server restart, the > replayed requests are used to determine which one was the provision request. > I could not find a nice way to do it, but I could make a workaround. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionDBAccessorImpl.java > 8e6fb3f > > ambari-server/src/main/java/org/apache/ambari/server/events/AmbariEvent.java > 1079806 > > ambari-server/src/main/java/org/apache/ambari/server/events/RequestFinishedEvent.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/topology/PersistedState.java > 77419d8 > > ambari-server/src/main/java/org/apache/ambari/server/topology/PersistedStateImpl.java > 324a397 > > ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java > e3f5b49 > > ambari-server/src/test/java/org/apache/ambari/server/topology/TopologyManagerTest.java > fd8653c > > Diff: https://reviews.apache.org/r/48266/diff/ > > > Testing > --- > > Succeeded locally > > > Thanks, > > Daniel Gergely > >
Review Request 48732: (Client) components that are dependencies of services in the stack definitions are always added to blueprint deployments
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48732/ --- Review request for Ambari, Robert Nettleton, Sandor Magyari, and Sebastian Toader. Bugs: AMBARI-17251 https://issues.apache.org/jira/browse/AMBARI-17251 Repository: ambari Description --- Problem: Component dependencies from stack definitions are automatically added to blueprint deployments even the services components belong to are not listed in the blueprint. eg.: SPARK_CLIENT/TEZ_CLIENT components that are listed as dependencies of the ATS and RESOURCEMANAGER in the stack definitions are always added to blueprints. Solution: Component dependencies defined in stack definitions are only added to blueprints if services belonging to the listed components are part of the blueprint Diffs - ambari-server/src/main/java/org/apache/ambari/server/topology/BlueprintValidatorImpl.java 432c6f8 ambari-server/src/test/java/org/apache/ambari/server/topology/BlueprintValidatorImplTest.java f78d86d Diff: https://reviews.apache.org/r/48732/diff/ Testing --- Unit tests running; Manually tested on local env. Thanks, Laszlo Puskas
Re: Review Request 48044: Provide context for hdp-select failures during ambari component install
> On May 31, 2016, 12:37 a.m., Sumit Mohanty wrote: > > ambari-common/src/main/python/resource_management/libraries/functions/hdp_select.py, > > line 129 > > <https://reviews.apache.org/r/48044/diff/1/?file=1400993#file1400993line129> > > > > We should also add the following commands: > > > > ls -la /usr/hdp/current > > hdp-select Added requested commands. - Laszlo --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48044/#review135565 ------- On June 2, 2016, 12:55 p.m., Laszlo Puskas wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48044/ > --- > > (Updated June 2, 2016, 12:55 p.m.) > > > Review request for Ambari, Oliver Szabo, Sumit Mohanty, and Sebastian Toader. > > > Bugs: AMBARI-16952 > https://issues.apache.org/jira/browse/AMBARI-16952 > > > Repository: ambari > > > Description > --- > > In case the hdp-select command fails during a service / component > installation there's no contextual information about the cause of the failure. > This issue is for logging information about the machine on which the > hdp-select command fails. > This solution wraps hdp-select command calls in a try/catch block and logs > failure / hdp installationrelated information. > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/hdp_select.py > 9a3201e > > Diff: https://reviews.apache.org/r/48044/diff/ > > > Testing > --- > > Manual testing underway. > > > Thanks, > > Laszlo Puskas > >
Re: Review Request 48169: Ensure smokeuser HDFS folder exists before running MR, YARN, PIG, OOZIE service checks
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48169/#review135950
---
Ship it!
Ship It!
- Laszlo Puskas
On June 2, 2016, 2:38 p.m., Sandor Magyari wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48169/
> ---
>
> (Updated June 2, 2016, 2:38 p.m.)
>
>
> Review request for Ambari, Laszlo Puskas, Oliver Szabo, and Sebastian Toader.
>
>
> Bugs: AMBARI-16956
> https://issues.apache.org/jira/browse/AMBARI-16956
>
>
> Repository: ambari
>
>
> Description
> ---
>
> Modify service check scripts for YARN, MR, PIG, OOZIE, MAHOUT to ensure that
> before running actual service checks using '/user/{smokeuser}' HDFS folder
> (YARN, MR, PIG, OOZIE, MAHOUT) create this resource if doesn't already exists.
>
>
> Diffs
> -
>
>
> ambari-server/src/main/resources/common-services/MAHOUT/1.0.0.2.3/package/scripts/params.py
> 60ebc42
>
> ambari-server/src/main/resources/common-services/MAHOUT/1.0.0.2.3/package/scripts/service_check.py
> dbb8079
>
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py
> b76bc89
>
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/service_check.py
> 8d14836
>
> ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/params_linux.py
> b7f798c
>
> ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/service_check.py
> 395295f
>
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/mapred_service_check.py
> d450701
>
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
> 150fedf
>
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/service_check.py
> c802c41
> ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_service_check.py
> 84ecf76
> ambari-server/src/test/python/stacks/2.0.6/PIG/test_pig_service_check.py
> 100d78a
>
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_mapreduce2_service_check.py
> 5f97efe
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_yarn_service_check.py
> 320092d
> ambari-server/src/test/python/stacks/2.2/PIG/test_pig_service_check.py
> 906c627
>
> ambari-server/src/test/python/stacks/2.3/MAHOUT/test_mahout_service_check.py
> 7a3f010
>
> Diff: https://reviews.apache.org/r/48169/diff/
>
>
> Testing
> ---
>
> Tested running service checks after removing Hdfs folder /uysers/ambari-qa.
>
> --
> Ran 261 tests in 6.702s
>
> OK
> --
> Total run:1052
> Total errors:0
> Total failures:0
>
>
> Thanks,
>
> Sandor Magyari
>
>
Re: Review Request 48732: (Client) components that are dependencies of services in the stack definitions are always added to blueprint deployments
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48732/ --- (Updated June 15, 2016, 7:15 p.m.) Review request for Ambari, Robert Nettleton, Sandor Magyari, and Sebastian Toader. Bugs: AMBARI-17251 https://issues.apache.org/jira/browse/AMBARI-17251 Repository: ambari Description --- Problem: Component dependencies from stack definitions are automatically added to blueprint deployments even the services components belong to are not listed in the blueprint. eg.: SPARK_CLIENT/TEZ_CLIENT components that are listed as dependencies of the ATS and RESOURCEMANAGER in the stack definitions are always added to blueprints. Solution: Component dependencies defined in stack definitions are only added to blueprints if services belonging to the listed components are part of the blueprint Diffs - ambari-server/src/main/java/org/apache/ambari/server/topology/BlueprintValidatorImpl.java 432c6f8 ambari-server/src/test/java/org/apache/ambari/server/topology/BlueprintValidatorImplTest.java f78d86d Diff: https://reviews.apache.org/r/48732/diff/ Testing (updated) --- Unit tests succeeded Manually tested on local env. Thanks, Laszlo Puskas
Re: Review Request 48722: Reduce the idle time before first command from next stage is executed on a host
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48722/#review138587 --- Ship it! Ship It! - Laszlo Puskas On June 16, 2016, 4:43 p.m., Sebastian Toader wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48722/ > --- > > (Updated June 16, 2016, 4:43 p.m.) > > > Review request for Ambari, Andrew Onischuk, Laszlo Puskas, Robert Levas, > Sandor Magyari, and Sumit Mohanty. > > > Bugs: AMBARI-17248 > https://issues.apache.org/jira/browse/AMBARI-17248 > > > Repository: ambari > > > Description > --- > > Commands to be executed by ambari-agents are being sent down by the server in > the response message to agent heartbeat messages. > The server processes the received heartbeat, it checks if there are next > commands scheduled to be executed by ambari-agent and adds those to the > heartbeat response for the ambari-agent. > The server organises the commands that can be executed in parallel into > stages. Ambari server ensures that only the commands of a single stage is > scheduled to be executed by the agent and starts scheduling the commands of > the next stage only after all commands of current stage has finished > successfully. > The processing of command status received with the heartbeat message happens > asynchronously to heartbeat response in HeartBeatProcessor and > ActionScheduler creation thus when the heartbeat response is created the > commands for the next stage are not scheduled yet. This means that the next > commands will be sent to agent only with the next heartbeat. > Agents currently sends a heartbeat to the server on command a completion or > at a timeout = self.netutil.HEARTBEAT_IDDLE_INTERVAL_SEC – > self.netutil.MINIMUM_INTERVAL_BETWEEN_HEARTBEATS interval which is ~10 > seconds if there are no commands to be executed. > This means that when the server receives a heartbeat triggered by the > completion of the last command from the current stage the server will send > the commands for the next stage only 10 seconds later when the next heartbeat > is received. This leads to agents spending considerable amount of time idle > when there are multiple stages to be executed. > Agents should heartbeat at a higher rate while there are still pending stages > to be executed. > > > Diffs > - > > ambari-agent/conf/unix/ambari-agent.ini 8f2ab1b > ambari-agent/conf/windows/ambari-agent.ini df88be6 > ambari-agent/src/main/python/ambari_agent/AmbariConfig.py 89a881a > ambari-agent/src/main/python/ambari_agent/Controller.py e981a76 > ambari-agent/src/main/python/ambari_agent/NetUtil.py 80bf3ae > ambari-agent/src/test/python/ambari_agent/TestNetUtil.py d72e319 > ambari-agent/src/test/python/ambari_agent/examples/ControllerTester.py > 8103872 > > ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java > 35a37e3 > > ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatResponse.java > 1ab7ae9 > ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java > ac0ddd2 > ambari-server/src/main/java/org/apache/ambari/server/state/Clusters.java > bd9de13 > > ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java > 3d2388e > > ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClustersImpl.java > c26e1e9 > > ambari-server/src/test/java/org/apache/ambari/server/state/cluster/ClusterImplTest.java > 627ade9 > > Diff: https://reviews.apache.org/r/48722/diff/ > > > Testing > --- > > Manual testing. > > Unit tests in succeeded. > > > Thanks, > > Sebastian Toader > >
Re: Review Request 48722: Reduce the idle time before first command from next stage is executed on a host
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48722/#review138586 --- ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java (line 532) <https://reviews.apache.org/r/48722/#comment203792> This call here is superfluous. - Laszlo Puskas On June 16, 2016, 4:43 p.m., Sebastian Toader wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/48722/ > --- > > (Updated June 16, 2016, 4:43 p.m.) > > > Review request for Ambari, Andrew Onischuk, Laszlo Puskas, Robert Levas, > Sandor Magyari, and Sumit Mohanty. > > > Bugs: AMBARI-17248 > https://issues.apache.org/jira/browse/AMBARI-17248 > > > Repository: ambari > > > Description > --- > > Commands to be executed by ambari-agents are being sent down by the server in > the response message to agent heartbeat messages. > The server processes the received heartbeat, it checks if there are next > commands scheduled to be executed by ambari-agent and adds those to the > heartbeat response for the ambari-agent. > The server organises the commands that can be executed in parallel into > stages. Ambari server ensures that only the commands of a single stage is > scheduled to be executed by the agent and starts scheduling the commands of > the next stage only after all commands of current stage has finished > successfully. > The processing of command status received with the heartbeat message happens > asynchronously to heartbeat response in HeartBeatProcessor and > ActionScheduler creation thus when the heartbeat response is created the > commands for the next stage are not scheduled yet. This means that the next > commands will be sent to agent only with the next heartbeat. > Agents currently sends a heartbeat to the server on command a completion or > at a timeout = self.netutil.HEARTBEAT_IDDLE_INTERVAL_SEC – > self.netutil.MINIMUM_INTERVAL_BETWEEN_HEARTBEATS interval which is ~10 > seconds if there are no commands to be executed. > This means that when the server receives a heartbeat triggered by the > completion of the last command from the current stage the server will send > the commands for the next stage only 10 seconds later when the next heartbeat > is received. This leads to agents spending considerable amount of time idle > when there are multiple stages to be executed. > Agents should heartbeat at a higher rate while there are still pending stages > to be executed. > > > Diffs > - > > ambari-agent/conf/unix/ambari-agent.ini 8f2ab1b > ambari-agent/conf/windows/ambari-agent.ini df88be6 > ambari-agent/src/main/python/ambari_agent/AmbariConfig.py 89a881a > ambari-agent/src/main/python/ambari_agent/Controller.py e981a76 > ambari-agent/src/main/python/ambari_agent/NetUtil.py 80bf3ae > ambari-agent/src/test/python/ambari_agent/TestNetUtil.py d72e319 > ambari-agent/src/test/python/ambari_agent/examples/ControllerTester.py > 8103872 > > ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java > 35a37e3 > > ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatResponse.java > 1ab7ae9 > ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java > ac0ddd2 > ambari-server/src/main/java/org/apache/ambari/server/state/Clusters.java > bd9de13 > > ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java > 3d2388e > > ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClustersImpl.java > c26e1e9 > > ambari-server/src/test/java/org/apache/ambari/server/state/cluster/ClusterImplTest.java > 627ade9 > > Diff: https://reviews.apache.org/r/48722/diff/ > > > Testing > --- > > Manual testing. > > Unit tests in succeeded. > > > Thanks, > > Sebastian Toader > >
Re: Review Request 49260: Update desired states in case of service restarts
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49260/ --- (Updated June 29, 2016, 9:10 a.m.) Review request for Ambari, Daniel Gergely, Oliver Szabo, Robert Nettleton, and Sandor Magyari. Bugs: AMBARI-17313 https://issues.apache.org/jira/browse/AMBARI-17313 Repository: ambari Description --- Problem: In case services / components are restarted, their desired states are not changed. As during restart -per definition- services are expected to be in STARTED state this should be OK, however if the restart is triggered for services in STOPPED state this causes problems (The UI allows triggering restart for stopped services). (Eg.: recovery manager doesn't bring them back to running state) Solution: When assembling the RESTART custom command desired states of affected services are updated to STARTED. Diffs - ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java b60592d ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelperTest.java 68b31c0 ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java fd70df5 Diff: https://reviews.apache.org/r/49260/diff/ Testing (updated) --- Manually tested on dev-env. Unit tests passed. Thanks, Laszlo Puskas
Re: Review Request 49260: Update desired states in case of service restarts
> On June 29, 2016, 1:20 a.m., Sumit Mohanty wrote: > > ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelperTest.java, > > line 20 > > <https://reviews.apache.org/r/49260/diff/2/?file=1431867#file1431867line20> > > > > Are changes to this file purely refactoring or addition of new tests? > > If its just refactoring then lets do it in trunk or branch-2.4 only. As the initial code change was in the AmbariCustomCommandExecutionHelper i thougth this test class needs to be modified / extended with new tests. As it was easier / quicker to add assertions to the existing test i gave up adding a new test here, but i have already removed mockito dependencies and used easymock instead (afaik easymock is what we use for testing) > On June 29, 2016, 1:20 a.m., Sumit Mohanty wrote: > > ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelperTest.java, > > line 324 > > <https://reviews.apache.org/r/49260/diff/2/?file=1431867#file1431867line324> > > > > Any reason the verify construct was removed? I have changed the test framework from Mockito to EasyMock. The test "logic" didn't change at all. - Laszlo --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49260/#review139909 ------- On June 28, 2016, 2:09 p.m., Laszlo Puskas wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/49260/ > --- > > (Updated June 28, 2016, 2:09 p.m.) > > > Review request for Ambari, Daniel Gergely, Oliver Szabo, Robert Nettleton, > and Sandor Magyari. > > > Bugs: AMBARI-17313 > https://issues.apache.org/jira/browse/AMBARI-17313 > > > Repository: ambari > > > Description > --- > > Problem: > In case services / components are restarted, their desired states are not > changed. As during restart -per definition- services are expected to be in > STARTED state this should be OK, however if the restart is triggered for > services in STOPPED state this causes problems (The UI allows triggering > restart for stopped services). (Eg.: recovery manager doesn't bring them back > to running state) > > Solution: > When assembling the RESTART custom command desired states of affected > services are updated to STARTED. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java > b60592d > > ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelperTest.java > 68b31c0 > > ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java > fd70df5 > > Diff: https://reviews.apache.org/r/49260/diff/ > > > Testing > --- > > Manually tested on dev-env. > > Unit tests running > > > Thanks, > > Laszlo Puskas > >
Re: Review Request 46765: Killing hive metastore and webhcat might fail with "no process" error
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46765/#review130897 --- Ship it! Ship It! - Laszlo Puskas On April 28, 2016, 9:06 a.m., Daniel Gergely wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/46765/ > --- > > (Updated April 28, 2016, 9:06 a.m.) > > > Review request for Ambari, Laszlo Puskas, Oliver Szabo, Sandor Magyari, and > Sebastian Toader. > > > Bugs: AMBARI-16150 > https://issues.apache.org/jira/browse/AMBARI-16150 > > > Repository: ambari > > > Description > --- > > When hive metastore or webhcat is killed, the process is the following: > 1. If process is running, then graceful kill > 2. If process is still running wait 5 seconds > 3. If process is still running, hard kill with kill -9 > > It is possible that process is running when check is done, but finishes > before issuing kill -9. As a result kill -9 fails with "no process" error. > Adding the flag "ignore_failures" swallows this exception. This is not a > problem at all, since if there is no process, then nothing to be done, if > there is a different error, then it means some serious issues with the linux > kernel itself. (signal SIGKILL is handled by the kernel) > > Checking other parts of the code, this ignore_failures flag was everywhere > except here, so I guess is is missing by accident. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py > 8399f9c > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/webhcat_service.py > 7d0a862 > ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_metastore.py > 6e27ded > ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py ea361fb > ambari-server/src/test/python/stacks/2.0.6/HIVE/test_webhcat_server.py > c401375 > ambari-server/src/test/python/stacks/2.1/HIVE/test_hive_metastore.py > f238ecc > > Diff: https://reviews.apache.org/r/46765/diff/ > > > Testing > --- > > Total run:998 > Total errors:0 > Total failures:0 > OK > > > Thanks, > > Daniel Gergely > >
Re: Review Request 46654: Auth-to-local rule generation duplicates default rules when adding case-insensitive default rules
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46654/#review130606 --- Ship it! Ship It! - Laszlo Puskas On April 25, 2016, 7:03 p.m., Robert Levas wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/46654/ > --- > > (Updated April 25, 2016, 7:03 p.m.) > > > Review request for Ambari, Dmitro Lisnichenko, Jonathan Hurley, Laszlo > Puskas, and Nate Cole. > > > Bugs: AMBARI-16023 > https://issues.apache.org/jira/browse/AMBARI-16023 > > > Repository: ambari > > > Description > --- > > When re-generating auth-to-local rules where existing rules are already set, > the default (or fallback) rule for the default and additional realms is > duplicated but the extra instance(s) have the case-insensitive flag: > > Example: > # Was > ``` > ... > RULE:[1:$1@$0](.*@EXAMPLE.COM)s/@.*// > ... > ``` > # Becomes} > ``` > ... > RULE:[1:$1@$0](.*@EXAMPLE.COM)s/@.*// > RULE:[1:$1@$0](.*@EXAMPLE.COM)s/@.*///L > ... > ``` > > *Steps to Reproduce* > 1. Create cluster with (at least) HDFS > 2. Enable Kerberos (do not check the box next to "Enable case insensitive > username rules"; kerberos-env/case_insensitive_username_rules should be false > 3. Edit Kerberos configuration and check "Enable case insensitive username > rules" to set kerberos-env/case_insensitive_username_rules to true > 4. Regenerate Keytabs > 5. See duplicate entry in HDFS configs > (core-site/hadoop.security.auth_to_local) > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/controller/AuthToLocalBuilder.java > 9d6db0a > > ambari-server/src/test/java/org/apache/ambari/server/controller/AuthToLocalBuilderTest.java > c88acc1 > > Diff: https://reviews.apache.org/r/46654/diff/ > > > Testing > --- > > Manually tested > > Running org.apache.ambari.server.controller.AuthToLocalBuilderTest > Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.06 sec - > in org.apache.ambari.server.controller.AuthToLocalBuilderTest > > Results : > > Tests run: 20, Failures: 0, Errors: 0, Skipped: 0 > > # Local test result: PENDING > > # Jenkins test results: PENDING > > > Thanks, > > Robert Levas > >
Re: Review Request 47596: HiveServer interactive - incorrect default memory value
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47596/#review134149 --- Ship it! Ship It! - Laszlo Puskas On May 20, 2016, 8:01 a.m., Daniel Gergely wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47596/ > --- > > (Updated May 20, 2016, 8:01 a.m.) > > > Review request for Ambari, Laszlo Puskas, Oliver Szabo, Sandor Magyari, and > Sebastian Toader. > > > Bugs: AMBARI-16776 > https://issues.apache.org/jira/browse/AMBARI-16776 > > > Repository: ambari > > > Description > --- > > Default value of hive.llap.daemon.yarn.container.mb is not aligned to > yarn.scheduler.minimum-allocation-mb > If the former one is less than the latter one, then hive server interactive > wont start. So the default values must be in a proper relation. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py > 9f5d799 > > ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml > db5f616 > ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py > 11aac72 > > Diff: https://reviews.apache.org/r/47596/diff/ > > > Testing > --- > > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 2:15.746s > [INFO] Finished at: Thu May 19 18:52:43 CEST 2016 > [INFO] Final Memory: 58M/1120M > [INFO] > > > > Thanks, > > Daniel Gergely > >
Re: Review Request 46899: Disable alternate user search functionality by default
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46899/#review131493 --- Ship it! Ship It! - Laszlo Puskas On May 2, 2016, 8:01 p.m., Sebastian Toader wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/46899/ > --- > > (Updated May 2, 2016, 8:01 p.m.) > > > Review request for Ambari, Balázs Bence Sári, Daniel Gergely, Laszlo Puskas, > Robert Levas, and Sandor Magyari. > > > Bugs: AMBARI-16205 > https://issues.apache.org/jira/browse/AMBARI-16205 > > > Repository: ambari > > > Description > --- > > Added `authentication.ldap.alternateUserSearchEnabled` configuration property > through which it can be controlled if the ldap alternate user search is > enabled or not. The default value for this property is `false.` > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java > 87f40d5 > > ambari-server/src/main/java/org/apache/ambari/server/security/authorization/AmbariLdapAuthenticationProvider.java > 7b2a95c > > ambari-server/src/test/java/org/apache/ambari/server/configuration/ConfigurationTest.java > 99ec786 > > ambari-server/src/test/java/org/apache/ambari/server/security/authorization/AmbariLdapAuthenticationProviderForDuplicateUserTest.java > f5d1412 > > ambari-server/src/test/java/org/apache/ambari/server/security/authorization/AmbariLdapAuthenticationProviderTest.java > b076e85 > > Diff: https://reviews.apache.org/r/46899/diff/ > > > Testing > --- > > Performed manual testing. > > Unit test results: > Results : > > Tests run: 4305, Failures: 0, Errors: 0, Skipped: 32 > > > Thanks, > > Sebastian Toader > >
Re: Review Request 46973: Amanded handling of excluded config-types handling in case of blueprint deployments
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46973/ --- (Updated May 4, 2016, 3:11 p.m.) Review request for Ambari, Robert Levas, Robert Nettleton, Sandor Magyari, and Sebastian Toader. Changes --- Added reviewers Bugs: AMBARI-16203 https://issues.apache.org/jira/browse/AMBARI-16203 Repository: ambari Description --- In case of blueprint deployments excluded properties defined in stack-definitions are only added if excluded config-type related service is also part of the blueprint. Added unit test to cover the above mentioned case. Diffs - ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java 29f937a ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java 2759869 Diff: https://reviews.apache.org/r/46973/diff/ Testing --- In progress... Thanks, Laszlo Puskas
Re: Review Request 46973: Amanded handling of excluded config-types handling in case of blueprint deployments
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46973/ --- (Updated May 5, 2016, 4:20 p.m.) Review request for Ambari, Robert Levas, Robert Nettleton, Sandor Magyari, and Sebastian Toader. Bugs: AMBARI-16203 https://issues.apache.org/jira/browse/AMBARI-16203 Repository: ambari Description --- In case of blueprint deployments excluded properties defined in stack-definitions are only added if excluded config-type related service is also part of the blueprint. Added unit test to cover the above mentioned case. Diffs - ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java 29f937a ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java 2759869 Diff: https://reviews.apache.org/r/46973/diff/ Testing (updated) --- Tested manually. Unit tests passed. Thanks, Laszlo Puskas
Review Request 46198: Oozie - falcon integration properties not xconsidered on blueprint deployments
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46198/ --- Review request for Ambari, Robert Levas, Sandor Magyari, and Sebastian Toader. Bugs: AMBARI-15882 https://issues.apache.org/jira/browse/AMBARI-15882 Repository: ambari Description --- Properties defined in external stack definition were excluded in case of bp deployments. (eg. : oozie-site.xml from the falcon stack definition) Diffs - ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java 1433a1b ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java 834953b Diff: https://reviews.apache.org/r/46198/diff/ Testing --- Unit test done. (mvn test underway) Manually tested on local / openstack cluster. Thanks, Laszlo Puskas
Re: Review Request 46496: Host_status stuck in UNKNOWN status after blueprint deploy with host in heartbeat-lost
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46496/#review129906 --- ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java (line 395) <https://reviews.apache.org/r/46496/#comment193458> Add the host to the log message - Laszlo Puskas On April 21, 2016, 3:19 p.m., Sebastian Toader wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/46496/ > --- > > (Updated April 21, 2016, 3:19 p.m.) > > > Review request for Ambari, Daniel Gergely, Laszlo Puskas, Sandor Magyari, > Sumit Mohanty, and Sid Wagle. > > > Bugs: AMBARI-16013 > https://issues.apache.org/jira/browse/AMBARI-16013 > > > Repository: ambari > > > Description > --- > > When hosts register to Ambari server the `TopologyManager` adds these to its > `availableHosts` collection. When a cluster is provisioned using Blueprints > `TopologyManager` tries to allocate required hosts to hostgroups from the > available hosts collection. In case hosts turned into HEARTBEAT_LOST state > these were not removed from `availableHosts` this resulting scheduling > logical tasks to unreachable hosts. When these unreachable hosts become > available re-register with Ambari server. The server since already scheduled > logical tasks for these it won't try again thus will never create role > commands to be executed by the hosts. > > `TopologyManager` has been hooked now to the HEARTBEAT_LOST state transition > to remove the host in question from its internal `availableHosts` collection. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/state/host/HostImpl.java > d221112 > > ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java > 5a0aca0 > > Diff: https://reviews.apache.org/r/46496/diff/ > > > Testing > --- > > Manual testign with a 5 node cluster using Blueprints. > > Unit tests: > Results : > > Tests run: 3561, Failures: 0, Errors: 0, Skipped: 36 > > > Thanks, > > Sebastian Toader > >
Re: Review Request 46198: Oozie - falcon integration properties not xconsidered on blueprint deployments
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46198/ --- (Updated April 14, 2016, 3:16 p.m.) Review request for Ambari, Robert Levas, Sandor Magyari, and Sebastian Toader. Changes --- Testing OK Bugs: AMBARI-15882 https://issues.apache.org/jira/browse/AMBARI-15882 Repository: ambari Description --- Properties defined in external stack definition were excluded in case of bp deployments. (eg. : oozie-site.xml from the falcon stack definition) Diffs - ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java 1433a1b ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java 834953b Diff: https://reviews.apache.org/r/46198/diff/ Testing (updated) --- Unit test done. Tests ran successfully. Manually tested on local / openstack cluster. Thanks, Laszlo Puskas
Re: Review Request 46198: Oozie - falcon integration properties not xconsidered on blueprint deployments
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46198/ --- (Updated April 14, 2016, 3:19 p.m.) Review request for Ambari, Robert Levas, Sandor Magyari, and Sebastian Toader. Bugs: AMBARI-15882 https://issues.apache.org/jira/browse/AMBARI-15882 Repository: ambari Description --- Properties defined in external stack definition were excluded in case of bp deployments. (eg. : oozie-site.xml from the falcon stack definition) Diffs - ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java 1433a1b ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java 834953b Diff: https://reviews.apache.org/r/46198/diff/ Testing (updated) --- Unit test done. Tests ran successfully. Manually tested on local / openstack cluster. Java Results : Tests run: 4226, Failures: 0, Errors: 0, Skipped: 32 Python: Total run:959 Total errors:0 Total failures:0 OK Thanks, Laszlo Puskas
Re: Review Request 47117: HiveServer interactive fails to start
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47117/#review132266 --- Ship it! Ship It! - Laszlo Puskas On May 9, 2016, 1:01 p.m., Daniel Gergely wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47117/ > --- > > (Updated May 9, 2016, 1:01 p.m.) > > > Review request for Ambari, Balázs Bence Sári, Laszlo Puskas, Oliver Szabo, > Sandor Magyari, and Sebastian Toader. > > > Bugs: AMBARI-16303 > https://issues.apache.org/jira/browse/AMBARI-16303 > > > Repository: ambari > > > Description > --- > > If Total Daemon Size of hive server interactive is less than yarn container > minimum size, then hive server interactive fails to start. Stack advisor was > corrected to take yarn container size into account. > The property *hive_server_interactive_host* was not resolved, not it is. It > was simply missing from the BlueprintConfigurationProcessor. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java > 29f937a > ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py > 37e6ef6 > > Diff: https://reviews.apache.org/r/47117/diff/ > > > Testing > --- > > The stack advisor issue is checked manually. > The blueprint issue is also checked manually, a separated unit test checks if > the resolver itself works correctly. > > Unit tests are still running locally... > > > Thanks, > > Daniel Gergely > >
Re: Review Request 50336: Added namenode HA configuration adjustments to the upgrade configurations
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50336/ --- (Updated July 22, 2016, 4:16 p.m.) Review request for Ambari, Alejandro Fernandez, Oliver Szabo, and Robert Levas. Changes --- fixed patch Bugs: AMBARI-17853 https://issues.apache.org/jira/browse/AMBARI-17853 Repository: ambari Description --- Problem: While upgrading HDP 2.4 to HDP 2.5 a cluster having namenode HA unwanted configurations get added to the cluster configurations. Resolution: Affected upgrade scripts have benn updated to perform hdfs namenode HA related configuration adjustments. (Similar to Earlier upgrade configurations) Diffs (updated) - ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml bd1bef2 ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml 065a933 ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml d152f34 Diff: https://reviews.apache.org/r/50336/diff/ Testing --- Manual testing underway. Thanks, Laszlo Puskas
Re: Review Request 50336: Added namenode HA configuration adjustments to the upgrade configurations
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50336/ --- (Updated July 22, 2016, 4:09 p.m.) Review request for Ambari, Alejandro Fernandez, Oliver Szabo, and Robert Levas. Changes --- Applied review note. Bugs: AMBARI-17853 https://issues.apache.org/jira/browse/AMBARI-17853 Repository: ambari Description --- Problem: While upgrading HDP 2.4 to HDP 2.5 a cluster having namenode HA unwanted configurations get added to the cluster configurations. Resolution: Affected upgrade scripts have benn updated to perform hdfs namenode HA related configuration adjustments. (Similar to Earlier upgrade configurations) Diffs (updated) - ambari-agent/src/main/python/ambari_agent/ActionQueue.py 1e7b1b6 ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionScheduler.java 205ef9f ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ExecutionCommandWrapper.java ef12c3a ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java 28de8ed ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java 6f612af ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java 399f26c ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java 9c2db1c ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 2174a64 ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 066acab ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProvider.java af24a69 ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java c390f44 ambari-server/src/main/java/org/apache/ambari/server/state/ConfigHelper.java 216d850 ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java 8dcd163 ambari-server/src/main/java/org/apache/ambari/server/topology/ConfigurationFactory.java f4dc879 ambari-server/src/main/resources/common-services/RANGER/0.6.0/configuration/ranger-admin-site.xml 477df7a ambari-server/src/main/resources/common-services/RANGER/0.6.0/kerberos.json 94c681d ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_server.py 1e859a8 ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py 61f73fb ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py b77980a ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml bd1bef2 ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml 065a933 ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml d152f34 ambari-server/src/main/resources/stacks/stack_advisor.py 437fe4f ambari-server/src/test/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommandTest.java 25d634b ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelperTest.java 7a5f377 ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerImplTest.java 37a2e63 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProviderTest.java 9603226 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ProvisionClusterRequestTest.java 0610d10 ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderHDP22Test.java 115b518 ambari-server/src/test/java/org/apache/ambari/server/topology/ConfigurationFactoryTest.java 1c2b177 ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py 5fe534a ambari-server/src/test/python/stacks/2.5/SPARK/test_spark_livy.py PRE-CREATION ambari-server/src/test/python/stacks/2.5/configs/default.json 148616b ambari-web/app/controllers/wizard/step7_controller.js 62a48d7 ambari-web/app/mixins/common/configs/enhanced_configs.js 4436cbb ambari-web/app/routes/main.js ebf9cb3 ambari-web/test/controllers/wizard/step7_test.js f54802e Diff: https://reviews.apache.org/r/50336/diff/ Testing --- Manual testing underway. Thanks, Laszlo Puskas
Review Request 50753: Cleared cached resources from ambari-server on host removal
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50753/ --- Review request for Ambari, Robert Nettleton, Sandor Magyari, and Sebastian Toader. Bugs: AMBARI-18005 https://issues.apache.org/jira/browse/AMBARI-18005 Repository: ambari Description --- When a host is removed from the cluster and later from ambari there's a chance the agent registers back to the ambari server before the agent is stopped. Stopping the machine running the agent without the host being deleted again leads to an inconsistent state in the ambari-server due to cached state. Resolution: The cached resources get cleared on host delete event. Diffs - ambari-server/src/main/java/org/apache/ambari/server/state/host/HostImpl.java a757010 ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java 0190478 Diff: https://reviews.apache.org/r/50753/diff/ Testing --- Manually, in progress. Thanks, Laszlo Puskas
Re: Review Request 50753: Cleared cached resources from ambari-server on host removal
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50753/ --- (Updated Aug. 3, 2016, 3:17 p.m.) Review request for Ambari, Robert Nettleton, Sandor Magyari, and Sebastian Toader. Changes --- Applied review notes. Bugs: AMBARI-18005 https://issues.apache.org/jira/browse/AMBARI-18005 Repository: ambari Description --- When a host is removed from the cluster and later from ambari there's a chance the agent registers back to the ambari server before the agent is stopped. Stopping the machine running the agent without the host being deleted again leads to an inconsistent state in the ambari-server due to cached state. Resolution: The cached resources get cleared on host delete event. Diffs (updated) - ambari-server/src/main/java/org/apache/ambari/server/state/host/HostImpl.java a757010 ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java 0190478 Diff: https://reviews.apache.org/r/50753/diff/ Testing --- Manually, in progress. Thanks, Laszlo Puskas
Re: Review Request 50753: Cleared cached resources from ambari-server on host removal
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50753/ --- (Updated Aug. 4, 2016, 3:11 p.m.) Review request for Ambari, Robert Nettleton, Sandor Magyari, and Sebastian Toader. Changes --- Manual test succeeded. Bugs: AMBARI-18005 https://issues.apache.org/jira/browse/AMBARI-18005 Repository: ambari Description --- When a host is removed from the cluster and later from ambari there's a chance the agent registers back to the ambari server before the agent is stopped. Stopping the machine running the agent without the host being deleted again leads to an inconsistent state in the ambari-server due to cached state. Resolution: The cached resources get cleared on host delete event. Diffs - ambari-server/src/main/java/org/apache/ambari/server/state/host/HostImpl.java a757010 ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java 0190478 Diff: https://reviews.apache.org/r/50753/diff/ Testing (updated) --- Suceesfully tested. (Manually) Unit tests in progress. Thanks, Laszlo Puskas
Re: Review Request 49260: Update desired states in case of service restarts
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49260/ --- (Updated June 28, 2016, 2:09 p.m.) Review request for Ambari, Daniel Gergely, Oliver Szabo, Robert Nettleton, and Sandor Magyari. Changes --- Added test assertions for the componenthost desiredstate changes on restart. Removed mockito dependency and optimized unit/functional test. Rebased to the head of the branch. Bugs: AMBARI-17313 https://issues.apache.org/jira/browse/AMBARI-17313 Repository: ambari Description --- Problem: In case services / components are restarted, their desired states are not changed. As during restart -per definition- services are expected to be in STARTED state this should be OK, however if the restart is triggered for services in STOPPED state this causes problems (The UI allows triggering restart for stopped services). (Eg.: recovery manager doesn't bring them back to running state) Solution: When assembling the RESTART custom command desired states of affected services are updated to STARTED. Diffs (updated) - ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java b60592d ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelperTest.java 68b31c0 ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java fd70df5 Diff: https://reviews.apache.org/r/49260/diff/ Testing (updated) --- Manually tested on dev-env. Unit tests running Thanks, Laszlo Puskas
Re: Review Request 49260: Update desired states in case of service restarts
> On June 27, 2016, 2:32 p.m., Sumit Mohanty wrote: > > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java, > > line 482 > > <https://reviews.apache.org/r/49260/diff/1/?file=1431001#file1431001line482> > > > > Lets modify one of the existing unit tests, > > (testResourceFiltersWithCustomCommands) to add a test for the state change. > > You can add a new test as well if thats better. Added assertions for the desired states after restart. > On June 27, 2016, 2:32 p.m., Sumit Mohanty wrote: > > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java, > > line 485 > > <https://reviews.apache.org/r/49260/diff/1/?file=1431001#file1431001line485> > > > > Let this be INFO. Done. > On June 27, 2016, 2:32 p.m., Sumit Mohanty wrote: > > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java, > > line 487 > > <https://reviews.apache.org/r/49260/diff/1/?file=1431001#file1431001line487> > > > > This may be fine but is it needed? Service component state generally > > refers to all host components associated with the component. Removed the servicecomponent desiredstate update. - Laszlo --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49260/#review139577 --- On June 28, 2016, 2:09 p.m., Laszlo Puskas wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/49260/ > --- > > (Updated June 28, 2016, 2:09 p.m.) > > > Review request for Ambari, Daniel Gergely, Oliver Szabo, Robert Nettleton, > and Sandor Magyari. > > > Bugs: AMBARI-17313 > https://issues.apache.org/jira/browse/AMBARI-17313 > > > Repository: ambari > > > Description > --- > > Problem: > In case services / components are restarted, their desired states are not > changed. As during restart -per definition- services are expected to be in > STARTED state this should be OK, however if the restart is triggered for > services in STOPPED state this causes problems (The UI allows triggering > restart for stopped services). (Eg.: recovery manager doesn't bring them back > to running state) > > Solution: > When assembling the RESTART custom command desired states of affected > services are updated to STARTED. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java > b60592d > > ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelperTest.java > 68b31c0 > > ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java > fd70df5 > > Diff: https://reviews.apache.org/r/49260/diff/ > > > Testing > --- > > Manually tested on dev-env. > > Unit tests running > > > Thanks, > > Laszlo Puskas > >
Review Request 49260: Update desired states in case of service restarts
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49260/ --- Review request for Ambari, Daniel Gergely, Oliver Szabo, Robert Nettleton, and Sandor Magyari. Repository: ambari Description --- Problem: In case services / components are restarted, their desired states are not changed. As during restart -per definition- services are expected to be in STARTED state this should be OK, however if the restart is triggered for services in STOPPED state this causes problems (The UI allows triggering restart for stopped services). (Eg.: recovery manager doesn't bring them back to running state) Solution: When assembling the RESTART custom command desired states of affected services are updated to STARTED. Diffs - ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java b60592d Diff: https://reviews.apache.org/r/49260/diff/ Testing --- Manually tested on dev-env. Unit tests running. Thanks, Laszlo Puskas
Re: Review Request 49260: Update desired states in case of service restarts
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49260/ --- (Updated June 27, 2016, 12:12 p.m.) Review request for Ambari, Daniel Gergely, Oliver Szabo, Robert Nettleton, and Sandor Magyari. Bugs: AMBARI-17313 https://issues.apache.org/jira/browse/AMBARI-17313 Repository: ambari Description --- Problem: In case services / components are restarted, their desired states are not changed. As during restart -per definition- services are expected to be in STARTED state this should be OK, however if the restart is triggered for services in STOPPED state this causes problems (The UI allows triggering restart for stopped services). (Eg.: recovery manager doesn't bring them back to running state) Solution: When assembling the RESTART custom command desired states of affected services are updated to STARTED. Diffs - ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java b60592d Diff: https://reviews.apache.org/r/49260/diff/ Testing --- Manually tested on dev-env. Unit tests running. Thanks, Laszlo Puskas
Re: Review Request 49260: Update desired states in case of service restarts
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49260/ --- (Updated June 27, 2016, 12:42 p.m.) Review request for Ambari, Daniel Gergely, Oliver Szabo, Robert Nettleton, and Sandor Magyari. Bugs: AMBARI-17313 https://issues.apache.org/jira/browse/AMBARI-17313 Repository: ambari Description --- Problem: In case services / components are restarted, their desired states are not changed. As during restart -per definition- services are expected to be in STARTED state this should be OK, however if the restart is triggered for services in STOPPED state this causes problems (The UI allows triggering restart for stopped services). (Eg.: recovery manager doesn't bring them back to running state) Solution: When assembling the RESTART custom command desired states of affected services are updated to STARTED. Diffs - ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java b60592d Diff: https://reviews.apache.org/r/49260/diff/ Testing (updated) --- Manually tested on dev-env. OK -- Total run:1073 Total errors:0 Total failures:0 Thanks, Laszlo Puskas
Re: Review Request 50753: Cleared cached resources from ambari-server on host removal
> On Aug. 5, 2016, 12:36 p.m., Jonathan Hurley wrote: > > This is an AmbariEvent which has it's own EventBus; however the bus is an > > asynchronous, single-threaded bus. This means that it's possible for a > > heartbeat to be received in between when this event is broadcast and when > > it's received by your subscriber method. > > > > If the problem only happens on registration, I think this solution is OK. > > But if a heartbeat from an agent can cause problems as well, then there is > > still a slight chance that the timing here could still reproduce the bug. Thanks for the observation. The problem only occurred on registration; Even if the host is registered before the host remove event is processed, the registering host will have a new id, thus the removal won't interfere with the new method. - Laszlo --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50753/#review144920 ----------- On Aug. 4, 2016, 3:22 p.m., Laszlo Puskas wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/50753/ > --- > > (Updated Aug. 4, 2016, 3:22 p.m.) > > > Review request for Ambari, Jonathan Hurley, Robert Nettleton, Sandor Magyari, > and Sebastian Toader. > > > Bugs: AMBARI-18005 > https://issues.apache.org/jira/browse/AMBARI-18005 > > > Repository: ambari > > > Description > --- > > When a host is removed from the cluster and later from ambari there's a > chance the agent registers back to the ambari server before the agent is > stopped. > Stopping the machine running the agent without the host being deleted again > leads to an inconsistent state in the ambari-server due to cached state. > Resolution: > The cached resources get cleared on host delete event. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/state/host/HostImpl.java > a757010 > > ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java > 0190478 > > Diff: https://reviews.apache.org/r/50753/diff/ > > > Testing > --- > > Suceesfully tested. (Manually) > Unit tests in progress. > > > Thanks, > > Laszlo Puskas > >
Re: Review Request 50753: Cleared cached resources from ambari-server on host removal
> On Aug. 5, 2016, 12:36 p.m., Jonathan Hurley wrote: > > This is an AmbariEvent which has it's own EventBus; however the bus is an > > asynchronous, single-threaded bus. This means that it's possible for a > > heartbeat to be received in between when this event is broadcast and when > > it's received by your subscriber method. > > > > If the problem only happens on registration, I think this solution is OK. > > But if a heartbeat from an agent can cause problems as well, then there is > > still a slight chance that the timing here could still reproduce the bug. > > Laszlo Puskas wrote: > Thanks for the observation. The problem only occurred on registration; > Even if the host is registered before the host remove event is processed, the > registering host will have a new id, thus the removal won't interfere with > the new method. In my previous comment i erroneously stated, that the new method won't cause issues due to the new id the host will be assigned when it registers back. Actually the code will behave correctly because available hosts are stored in a list; thus the code will find the proper record to remove. (the older entry will be found first in the list) - Laszlo --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50753/#review144920 ------- On Aug. 4, 2016, 3:22 p.m., Laszlo Puskas wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/50753/ > --- > > (Updated Aug. 4, 2016, 3:22 p.m.) > > > Review request for Ambari, Jonathan Hurley, Robert Nettleton, Sandor Magyari, > and Sebastian Toader. > > > Bugs: AMBARI-18005 > https://issues.apache.org/jira/browse/AMBARI-18005 > > > Repository: ambari > > > Description > --- > > When a host is removed from the cluster and later from ambari there's a > chance the agent registers back to the ambari server before the agent is > stopped. > Stopping the machine running the agent without the host being deleted again > leads to an inconsistent state in the ambari-server due to cached state. > Resolution: > The cached resources get cleared on host delete event. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/state/host/HostImpl.java > a757010 > > ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java > 0190478 > > Diff: https://reviews.apache.org/r/50753/diff/ > > > Testing > --- > > Suceesfully tested. (Manually) > Unit tests in progress. > > > Thanks, > > Laszlo Puskas > >
Re: Review Request 50753: Cleared cached resources from ambari-server on host removal
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50753/ --- (Updated Aug. 8, 2016, 11:43 a.m.) Review request for Ambari, Jonathan Hurley, Robert Nettleton, Sandor Magyari, and Sebastian Toader. Bugs: AMBARI-18005 https://issues.apache.org/jira/browse/AMBARI-18005 Repository: ambari Description --- When a host is removed from the cluster and later from ambari there's a chance the agent registers back to the ambari server before the agent is stopped. Stopping the machine running the agent without the host being deleted again leads to an inconsistent state in the ambari-server due to cached state. Resolution: The cached resources get cleared on host delete event. Diffs (updated) - ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java 9a6ee94 Diff: https://reviews.apache.org/r/50753/diff/ Testing --- Suceesfully tested. (Manually) Unit tests OK Thanks, Laszlo Puskas
Re: Review Request 50753: Cleared cached resources from ambari-server on host removal
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50753/ --- (Updated Aug. 8, 2016, 11:31 a.m.) Review request for Ambari, Jonathan Hurley, Robert Nettleton, Sandor Magyari, and Sebastian Toader. Changes --- Fixed NPE prone code Bugs: AMBARI-18005 https://issues.apache.org/jira/browse/AMBARI-18005 Repository: ambari Description --- When a host is removed from the cluster and later from ambari there's a chance the agent registers back to the ambari server before the agent is stopped. Stopping the machine running the agent without the host being deleted again leads to an inconsistent state in the ambari-server due to cached state. Resolution: The cached resources get cleared on host delete event. Diffs (updated) - ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java 9a6ee94 Diff: https://reviews.apache.org/r/50753/diff/ Testing --- Suceesfully tested. (Manually) Unit tests OK Thanks, Laszlo Puskas
Review Request 50336: Added namenode HA configuration adjustments to the upgrade configurations
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50336/ --- Review request for Ambari, Alejandro Fernandez, Oliver Szabo, and Robert Levas. Bugs: AMBARI-17853 https://issues.apache.org/jira/browse/AMBARI-17853 Repository: ambari Description --- Problem: While upgrading HDP 2.4 to HDP 2.5 a cluster having namenode HA unwanted configurations get added to the cluster configurations. Resolution: Affected upgrade scripts have benn updated to perform hdfs namenode HA related configuration adjustments. (Similar to Earlier upgrade configurations) Diffs - ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml bd1bef2 ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml 065a933 ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml d152f34 Diff: https://reviews.apache.org/r/50336/diff/ Testing --- Manual testing underway. Thanks, Laszlo Puskas
Re: Review Request 49383: Add a validation of required services during a blueprint deployment
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49383/#review140150 --- Ship it! Ship It! - Laszlo Puskas On June 29, 2016, 4:14 p.m., Andrew Onischuk wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/49383/ > --- > > (Updated June 29, 2016, 4:14 p.m.) > > > Review request for Ambari, Laszlo Puskas, Sebastian Toader, and Vitalyi > Brodetskyi. > > > Repository: ambari > > > Description > --- > > Add a validation of required services during a blueprint deployment. > Currently a blueprint deployment allows to install YARN without HDFS. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/Stack.java > 16f75ee > > ambari-server/src/main/java/org/apache/ambari/server/topology/BlueprintValidatorImpl.java > de5e2b3 > > ambari-server/src/test/java/org/apache/ambari/server/topology/BlueprintValidatorImplTest.java > b85d454 > > Diff: https://reviews.apache.org/r/49383/diff/ > > > Testing > --- > > mvn clean test > > > Thanks, > > Andrew Onischuk > >
Re: Review Request 49455: Optimized classpath scannig for upgrade checks impelemtation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49455/ --- (Updated June 30, 2016, 4:12 p.m.) Review request for Ambari, Daniel Gergely, Sumit Mohanty, and Sebastian Toader. Bugs: AMBARI-17505 https://issues.apache.org/jira/browse/AMBARI-17505 Repository: ambari Description (updated) --- Problem: During startup the ambari server scasns the classpath for finding components to be bound in the IoC context. When binding upgrade check implementations the full ambari package is scanned that leads to prolonged startup time. Solution: As upgrade check implementations reside in a dedicated package, the scanner is modified to lookup them in this very package. (on the local env this shortens the startup time by ~25 seconds) Diffs - ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java e0bda13 Diff: https://reviews.apache.org/r/49455/diff/ Testing --- Unit tests running. Thanks, Laszlo Puskas
Review Request 49455: Optimized classpath scannig for upgrade checks impelemtation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49455/ --- Review request for Ambari, Sumit Mohanty and Sebastian Toader. Bugs: AMBARI-17505 https://issues.apache.org/jira/browse/AMBARI-17505 Repository: ambari Description --- Problem: During startup the ambari server scasns the classpath for finding components to be bound in the IoC context. When binding upgrade check implementations the full ambari package is scanned that leads to prolonged startup time. Solution: As upgrade check implementations reside in a dedicated package, the scanner is modified to lookup them in this very package. Diffs - ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java e0bda13 Diff: https://reviews.apache.org/r/49455/diff/ Testing --- Unit tests running. Thanks, Laszlo Puskas
Re: Review Request 49455: Optimized classpath scannig for upgrade check impelemtations
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49455/ --- (Updated June 30, 2016, 4:15 p.m.) Review request for Ambari, Daniel Gergely, Sumit Mohanty, and Sebastian Toader. Summary (updated) - Optimized classpath scannig for upgrade check impelemtations Bugs: AMBARI-17505 https://issues.apache.org/jira/browse/AMBARI-17505 Repository: ambari Description --- Problem: During startup the ambari server scasns the classpath for finding components to be bound in the IoC context. When binding upgrade check implementations the full ambari package is scanned that leads to prolonged startup time. Solution: As upgrade check implementations reside in a dedicated package, the scanner is modified to lookup them in this very package. (on the local env this shortens the startup time by ~25 seconds) Diffs - ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java e0bda13 Diff: https://reviews.apache.org/r/49455/diff/ Testing --- Unit tests running. Thanks, Laszlo Puskas
Re: Review Request 49455: Optimized classpath scannig for upgrade check impelementations
> On June 30, 2016, 11:47 p.m., Alejandro Fernandez wrote: > > ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java, > > line 648 > > <https://reviews.apache.org/r/49455/diff/1/?file=1434261#file1434261line648> > > > > Can we use reflection to look up the package name instead of hardcoding > > it? Fixed this, however ther are other packagenames hardcoded here; long term it would be good to have the whole classpath scanning logic refactored so that all beans are looked up with a single scan ... - Laszlo --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49455/#review140277 ------- On July 1, 2016, 12:12 p.m., Laszlo Puskas wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/49455/ > --- > > (Updated July 1, 2016, 12:12 p.m.) > > > Review request for Ambari, Daniel Gergely, Sumit Mohanty, and Sebastian > Toader. > > > Bugs: AMBARI-17505 > https://issues.apache.org/jira/browse/AMBARI-17505 > > > Repository: ambari > > > Description > --- > > Problem: > During startup the ambari server scasns the classpath for finding components > to be bound in the IoC context. > When binding upgrade check implementations the full ambari package is scanned > that leads to prolonged startup time. > > Solution: > As upgrade check implementations reside in a dedicated package, the scanner > is modified to lookup them in this very package. > (on the local env this shortens the startup time by ~25 seconds) > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java > e0bda13 > > Diff: https://reviews.apache.org/r/49455/diff/ > > > Testing > --- > > Unit tests running. > > > Thanks, > > Laszlo Puskas > >
Re: Review Request 49455: Optimized classpath scannig for upgrade check impelementations
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49455/ --- (Updated July 1, 2016, 12:12 p.m.) Review request for Ambari, Daniel Gergely, Sumit Mohanty, and Sebastian Toader. Summary (updated) - Optimized classpath scannig for upgrade check impelementations Bugs: AMBARI-17505 https://issues.apache.org/jira/browse/AMBARI-17505 Repository: ambari Description --- Problem: During startup the ambari server scasns the classpath for finding components to be bound in the IoC context. When binding upgrade check implementations the full ambari package is scanned that leads to prolonged startup time. Solution: As upgrade check implementations reside in a dedicated package, the scanner is modified to lookup them in this very package. (on the local env this shortens the startup time by ~25 seconds) Diffs - ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java e0bda13 Diff: https://reviews.apache.org/r/49455/diff/ Testing --- Unit tests running. Thanks, Laszlo Puskas
Re: Review Request 49455: Optimized classpath scannig for upgrade check impelemtations
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49455/ --- (Updated July 1, 2016, 9:31 a.m.) Review request for Ambari, Daniel Gergely, Sumit Mohanty, and Sebastian Toader. Changes --- Applied review notes. Bugs: AMBARI-17505 https://issues.apache.org/jira/browse/AMBARI-17505 Repository: ambari Description --- Problem: During startup the ambari server scasns the classpath for finding components to be bound in the IoC context. When binding upgrade check implementations the full ambari package is scanned that leads to prolonged startup time. Solution: As upgrade check implementations reside in a dedicated package, the scanner is modified to lookup them in this very package. (on the local env this shortens the startup time by ~25 seconds) Diffs (updated) - ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java e0bda13 Diff: https://reviews.apache.org/r/49455/diff/ Testing --- Unit tests running. Thanks, Laszlo Puskas
Re: Review Request 56175: Setup correct authentication and authorization mechanism between Yarn Registry and Zookeeper
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56175/#review163842 --- Ship it! Ship It! - Laszlo Puskas On Feb. 1, 2017, 4:43 p.m., Attila Magyar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/56175/ > --- > > (Updated Feb. 1, 2017, 4:43 p.m.) > > > Review request for Ambari, Laszlo Puskas, Robert Levas, and Sebastian Toader. > > > Bugs: AMBARI-19331 > https://issues.apache.org/jira/browse/AMBARI-19331 > > > Repository: ambari > > > Description > --- > > YARN registry and yarn leader election znodes must be protected by sasl ACLs > when kerberos enabled. The sasl ACLs should be removed after disabling > kerberos. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py > aed8abc > > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py > a659dd1 > > ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/kerberos.json > 29cc00a > > ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/package/scripts/params_linux.py > 4d47925 > > ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/package/scripts/resourcemanager.py > 4d8d95e > ambari-server/src/main/resources/stacks/HDP/2.6/services/YARN/kerberos.json > eaffec6 > > Diff: https://reviews.apache.org/r/56175/diff/ > > > Testing > --- > > Manual testing: > - created a cluster with yarn and hive > - enabled hive interactive mode > - enabled kerberos > - checked that hive interactive started successfully > - checked acls on /registry > - disabled kerberos > - checked acls on /registry > > Unittest run successfully (1 unrelated failure) > > Failed tests: > > AsyncCallableServiceTest.testCallableServiceShouldCancelTaskWhenTimeoutExceeded:94 > > Unexpected method call ScheduledExecutorService.schedule(EasyMock for > interface java.util.concurrent.Callable, 500, MILLISECONDS): > > Tests run: 4884, Failures: 1, Errors: 0, Skipped: 39 > Ran 270 tests in 6.748s > > > ambari-agent: >Ran 451 tests in 95.436s > > > Thanks, > > Attila Magyar > >
Re: Review Request 55762: Ambari db-cleanup tool fixed
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55762/ --- (Updated Jan. 24, 2017, 9:27 a.m.) Review request for Ambari, Attila Doroszlai, Sandor Magyari, and Sebastian Toader. Bugs: AMBARI-19649 https://issues.apache.org/jira/browse/AMBARI-19649 Repository: ambari Description (updated) --- The ambari-server db-cleanup tool failed due to unsatisfied guice dependencies. The cause was that when the audit logging has been added, the related guice module hasn;t been set into the tool's injector. The fix is to add the missing module to the db-cleanup tool's injector. branch-2.5: commit 8111ba662e849c27e4a9a04305b3ef2260c030e9 Author: Laszlo Puskas <lpus...@hortonworks.com> Date:?? Fri Jan 20 19:18:10 2017 +0100 AMBARI-19649. Ambari purge tool broken. (Laszlo Puskas via stoader) commit ec6ba0abbf2b1ce343f370f63512d45e7add9a40 Author: Laszlo Puskas <lpus...@hortonworks.com> Date:?? Fri Jan 20 23:18:13 2017 +0100 AMBARI-19649. Ambari purge tool broken. (Laszlo Puskas via stoader) Diffs - ambari-server/src/main/java/org/apache/ambari/server/cleanup/CleanupDriver.java f10250e ambari-server/src/main/java/org/apache/ambari/server/stack/StackManager.java 0e2e0d8 Diff: https://reviews.apache.org/r/55762/diff/ Testing --- Manually. Thanks, Laszlo Puskas
Re: Review Request 55680: On secure NN HA clusters ZKFC connects to zookeeper securely
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55680/ --- (Updated Jan. 24, 2017, 9:41 a.m.) Review request for Ambari, Attila Magyar, Robert Levas, and Sebastian Toader. Bugs: AMBARI-19613 https://issues.apache.org/jira/browse/AMBARI-19613 Repository: ambari Description --- On secure namenode HA clusters the ZKFC component needs to access the zookeeper securely. On enabling security appropriate settings are configured to secure this connection. Diffs - ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/hadoop-env.xml c2f37c1 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/kerberos.json 1cf1603 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py 3270430 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/zkfc_slave.py f1891a5 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/templates/hdfs_jaas.conf.j2 PRE-CREATION ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/configuration/hadoop-env.xml 24032fa ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/kerberos.json 4fdffcf ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/utils.py f76935a ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/zkfc_slave.py f1891a5 ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/templates/hdfs_jaas.conf.j2 PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py 783f811 ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/hadoop-env.xml 5be2b74 ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/configuration/hadoop-env.xml 24e0193 ambari-server/src/main/resources/stacks/HDP/2.4/services/HDFS/configuration/hadoop-env.xml 24e0193 ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/kerberos.json 766a014 ambari-server/src/main/resources/stacks/HDP/3.0/hooks/before-ANY/scripts/params.py f70c8e9 ambari-server/src/main/resources/stacks/HDP/3.0/services/HDFS/configuration/hadoop-env.xml e680c1b ambari-server/src/test/python/stacks/2.0.6/HDFS/test_zkfc.py e952108 Diff: https://reviews.apache.org/r/55680/diff/ Testing (updated) --- Testing done manually: Created an unsecure NN HA cluster * checked the configuration entry: ha.zookeeper.acl - doesn't exist * checked the hadoop-env.sh - doesn't contain the variable export HADOOP_ZKFC_OPTS * checked the hdfs_jaas.conf - doesn't exist * connected to zookeeper, listed znode acls - no limitations set Kerberized the NN HA cluster * checked the configuration entry: ha.zookeeper.acl - set to sasl:nn:cdrwa * checked the hadoop-env.sh - contains the variable export HADOOP_ZKFC_OPTS with proper value, points to the correct jaas file * checked the hdfs_jaas.conf - OK * connected to zookeeper, listed znode acls - set as required (/hadoop-ha/mycluster/ActiveStandbyElectorLock) Disabled Kerberos on the NN HA cluster * checked the configuration entry: ha.zookeeper.acl - removed * checked the hadoop-env.sh - doesn't contain the variable export HADOOP_ZKFC_OPTS * connected to zookeeper, listed znode acls - set as required (/hadoop-ha/mycluster/ActiveStandbyElectorLock) Unit tests: Success. Committed to trunk: commit a382bed7f55be632fd03e1b02bb8a01151234b24 Author: Laszlo Puskas <lpus...@hortonworks.com> Date: Fri Jan 20 12:41:02 2017 +0100 AMBARI-19613. ZKFC Zookeper connection is not secure. (Laszlo Puskas via stoader) branch-2.5 commit 00b2c42ccf6fe68267483a645f6e57e9c921f01b Author: Laszlo Puskas <lpus...@hortonworks.com> Date: Fri Jan 20 14:04:06 2017 +0100 AMBARI-19613. ZKFC Zookeper connection is not secure (Laszlo Puskas via magyari_sandor) Thanks, Laszlo Puskas
Re: Review Request 55883: Post user creation hook - input csv generated with READ permissions
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55883/ --- (Updated Jan. 25, 2017, 9:15 a.m.) Review request for Ambari, Attila Magyar, Robert Levas, Sandor Magyari, and Sebastian Toader. Changes --- Added test details Bugs: AMBARI-19694 https://issues.apache.org/jira/browse/AMBARI-19694 Repository: ambari Description --- Currently the post user creation hook mechanism generates the user input csv with read permissions only for the user that runs ambari. This may cause issues on secure clusters, where ambari is not ran by the root user, az auth to local rules may map to users that don't have READ right for the generated csv. Solution: input csv file is generated with REAd permissions for all users. Diffs - ambari-server/src/main/java/org/apache/ambari/server/serveraction/users/CsvFilePersisterService.java d8ffe98 ambari-server/src/test/java/org/apache/ambari/server/serveraction/users/CsvFilePersisterServiceFunctionalTest.java PRE-CREATION Diff: https://reviews.apache.org/r/55883/diff/ Testing (updated) --- Manually on local environment: - created a cluster (post user creation hook enabled) - from the ambari UI created a new local user - checked the permissions on the generated csv file - owner / group / all read permissions set on the file - created a new system user (not root ...) - opened the generated csv as the newlycreated user Unit test created. Thanks, Laszlo Puskas
Re: Review Request 55983: AMBARI-19712. Cluster creation fails due to database exception
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55983/#review163124 --- Ship it! Ship It! - Laszlo Puskas On Jan. 26, 2017, 1:31 p.m., Attila Doroszlai wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/55983/ > --- > > (Updated Jan. 26, 2017, 1:31 p.m.) > > > Review request for Ambari, Jonathan Hurley, Laszlo Puskas, Sandor Magyari, > and Sebastian Toader. > > > Bugs: AMBARI-19712 > https://issues.apache.org/jira/browse/AMBARI-19712 > > > Repository: ambari > > > Description > --- > > `RetryHelper` already invalidates the in-memory cluster data and re-loads it > from the DB after the exception, we just need to ensure that the database has > consistent state (all or nothing), safe for retry. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java > 0b3869bfc0a84246c78f47016edefde74136604d > > ambari-server/src/main/java/org/apache/ambari/server/state/ServiceComponentImpl.java > 236091b9180a9c4279ee061d713885ab7610 > > Diff: https://reviews.apache.org/r/55983/diff/ > > > Testing > --- > > Manual testing: simulated database failure by throwing `DatabaseException` > with some probability, verified that cluster creation was successful with > retries. > > Existing unit tests: > > ``` > $ mvn -am -pl ambari-server -Dcheckstyle.skip clean test > ... > Tests run: 4882, Failures: 0, Errors: 0, Skipped: 38 > ... > Total run:1179 > Total errors:0 > Total failures:0 > OK > ... > [INFO] BUILD SUCCESS > ``` > > > Thanks, > > Attila Doroszlai > >
Review Request 55987: Post user creation hook script creates user home dirs as the configured hdfs_user
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55987/ --- Review request for Ambari, Attila Doroszlai, Sandor Magyari, and Sebastian Toader. Bugs: AMBARI-19726 https://issues.apache.org/jira/browse/AMBARI-19726 Repository: ambari Description --- The post user creation hook had wired the **hdfs** user, user homes were created in the name of this user. In case of customized clusters this didn;t work due to permssion issues. The solution is to get the proper hdfs_user from the configureation and access the dfs as this user. Diffs - ambari-server/src/main/java/org/apache/ambari/server/hooks/users/UserHookParams.java 6970dcc ambari-server/src/main/java/org/apache/ambari/server/hooks/users/UserHookService.java c4ff1e4 ambari-server/src/main/java/org/apache/ambari/server/serveraction/users/CsvFilePersisterService.java fe6bf35 ambari-server/src/main/java/org/apache/ambari/server/serveraction/users/PostUserCreationHookServerAction.java e713128 ambari-server/src/main/resources/scripts/post-user-creation-hook.sh ee8d2d1 ambari-server/src/test/java/org/apache/ambari/server/hooks/users/UserHookServiceTest.java 293b22a Diff: https://reviews.apache.org/r/55987/diff/ Testing --- Unit tests running. Manual testing underway Thanks, Laszlo Puskas
Re: Review Request 55987: Post user creation hook script creates user home dirs as the configured hdfs_user
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55987/ --- (Updated Jan. 26, 2017, 5:03 p.m.) Review request for Ambari, Attila Doroszlai, Sandor Magyari, and Sebastian Toader. Changes --- Fixed failing unit test. Bugs: AMBARI-19726 https://issues.apache.org/jira/browse/AMBARI-19726 Repository: ambari Description --- The post user creation hook had wired the **hdfs** user, user homes were created in the name of this user. In case of customized clusters this didn;t work due to permssion issues. The solution is to get the proper hdfs_user from the configureation and access the dfs as this user. Diffs (updated) - ambari-server/src/main/java/org/apache/ambari/server/hooks/users/UserHookParams.java 6970dcc ambari-server/src/main/java/org/apache/ambari/server/hooks/users/UserHookService.java c4ff1e4 ambari-server/src/main/java/org/apache/ambari/server/serveraction/users/CsvFilePersisterService.java fe6bf35 ambari-server/src/main/java/org/apache/ambari/server/serveraction/users/PostUserCreationHookServerAction.java e713128 ambari-server/src/main/resources/scripts/post-user-creation-hook.sh ee8d2d1 ambari-server/src/test/java/org/apache/ambari/server/hooks/users/UserHookServiceTest.java 293b22a ambari-server/src/test/java/org/apache/ambari/server/serveraction/users/PostUserCreationHookServerActionTest.java f5cdf48 Diff: https://reviews.apache.org/r/55987/diff/ Testing --- Unit tests running. Manual testing underway Thanks, Laszlo Puskas
Re: Review Request 56064: Format ZKFC commands failing while enabling NameNode HA
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56064/#review163483 --- Ship it! Ship It! - Laszlo Puskas On Jan. 29, 2017, 8:43 p.m., Sebastian Toader wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/56064/ > --- > > (Updated Jan. 29, 2017, 8:43 p.m.) > > > Review request for Ambari, Attila Magyar, Laszlo Puskas, and Robert Levas. > > > Bugs: AMBARI-19736 > https://issues.apache.org/jira/browse/AMBARI-19736 > > > Repository: ambari > > > Description > --- > > Format ZKFC commands failing while enabling NameNode HA at Initializa > Metadata step. > > ``` > Caused by: org.apache.zookeeper.KeeperException$NoAuthException: > KeeperErrorCode = NoAuth for /hadoop-ha/nameservice > at org.apache.zookeeper.KeeperException.create(KeeperException.java:113) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) > at > org.apache.hadoop.ha.ActiveStandbyElector$3.run(ActiveStandbyElector.java:1000) > at > org.apache.hadoop.ha.ActiveStandbyElector$3.run(ActiveStandbyElector.java:997) > at > org.apache.hadoop.ha.ActiveStandbyElector.zkDoWithRetries(ActiveStandbyElector.java:1041) > at > org.apache.hadoop.ha.ActiveStandbyElector.createWithRetries(ActiveStandbyElector.java:997) > at > org.apache.hadoop.ha.ActiveStandbyElector.ensureParentZNode(ActiveStandbyElector.java:344) > ... 11 more > ``` > > The reason for the failure is that the `hdfs_jaas.conf` file is generated > during ZKFC component configuration. When NN HA is enabled via UI the ZKFC is > not added yet to the cluster thus the `hdfs_jaas.conf` file is not generated > yet, this leading the format ZKFC commands to fail as this require the jaas > file. > > The solution is to move the creation of `hdfs_jaas.conf` file into the NN > configuration. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py > a2edf38 > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py > 03aba7b > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/zkfc_slave.py > bfc9429 > > ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/hdfs_namenode.py > 7fae57f > > ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/utils.py > 9eebe63 > > ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/zkfc_slave.py > f2ea6ad > > ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py > 8e0e783 > > ambari-server/src/main/resources/stacks/HDP/3.0/hooks/before-ANY/scripts/params.py > 8e5d210 > > Diff: https://reviews.apache.org/r/56064/diff/ > > > Testing > --- > > Manual testing using both wizzard and blueprints. > > Unit tests: > ``` > mvn test -DskipSurefireTests -am -pl ambari-server > > > [INFO] Reactor Summary: > [INFO] > [INFO] Ambari Main ... SUCCESS [11.634s] > [INFO] Apache Ambari Project POM . SUCCESS [0.045s] > [INFO] Ambari Views .. SUCCESS [2.086s] > [INFO] utility ... SUCCESS [1.175s] > [INFO] ambari-metrics SUCCESS [0.608s] > [INFO] Ambari Metrics Common . SUCCESS [0.352s] > [INFO] Ambari Server . SUCCESS [2:25.628s] > [INFO] > > [INFO] BUILD SUCCESS > ``` > > > Thanks, > > Sebastian Toader > >
Review Request 55762: Ambari db-cleanup tool fixed
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55762/ --- Review request for Ambari, Attila Doroszlai, Sandor Magyari, and Sebastian Toader. Bugs: AMBARI-19649 https://issues.apache.org/jira/browse/AMBARI-19649 Repository: ambari Description --- The ambari-server db-cleanup tool failed due to unsatisfied guice dependencies. The cause was that when the audit logging has been added, the related guice module hasn;t been set into the tool's injector. The fix is to add the missing module to the db-cleanup tool's injector. Diffs - ambari-server/src/main/java/org/apache/ambari/server/cleanup/CleanupDriver.java f10250e ambari-server/src/main/java/org/apache/ambari/server/stack/StackManager.java 0e2e0d8 Diff: https://reviews.apache.org/r/55762/diff/ Testing --- Manually. Thanks, Laszlo Puskas
Re: Review Request 55762: Ambari db-cleanup tool fixed
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55762/ --- (Updated Jan. 20, 2017, 9:19 p.m.) Review request for Ambari, Attila Doroszlai, Sandor Magyari, and Sebastian Toader. Bugs: AMBARI-19649 https://issues.apache.org/jira/browse/AMBARI-19649 Repository: ambari Description (updated) --- The ambari-server db-cleanup tool failed due to unsatisfied guice dependencies. The cause was that when the audit logging has been added, the related guice module hasn;t been set into the tool's injector. The fix is to add the missing module to the db-cleanup tool's injector. branch-2.5: commit 8111ba662e849c27e4a9a04305b3ef2260c030e9 Author: Laszlo Puskas <lpus...@hortonworks.com> Date:?? Fri Jan 20 19:18:10 2017 +0100 AMBARI-19649. Ambari purge tool broken. (Laszlo Puskas via stoader) Diffs - ambari-server/src/main/java/org/apache/ambari/server/cleanup/CleanupDriver.java f10250e ambari-server/src/main/java/org/apache/ambari/server/stack/StackManager.java 0e2e0d8 Diff: https://reviews.apache.org/r/55762/diff/ Testing --- Manually. Thanks, Laszlo Puskas
Re: Review Request 55987: Post user creation hook script creates user home dirs as the configured hdfs_user
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55987/ --- (Updated Jan. 27, 2017, 2:59 p.m.) Review request for Ambari, Attila Doroszlai, Sandor Magyari, and Sebastian Toader. Changes --- Fixed hook script, removed uniused imports. Bugs: AMBARI-19726 https://issues.apache.org/jira/browse/AMBARI-19726 Repository: ambari Description --- The post user creation hook had wired the **hdfs** user, user homes were created in the name of this user. In case of customized clusters this didn;t work due to permssion issues. The solution is to get the proper hdfs_user from the configureation and access the dfs as this user. Diffs (updated) - ambari-server/src/main/java/org/apache/ambari/server/hooks/users/UserHookParams.java 6970dcc ambari-server/src/main/java/org/apache/ambari/server/hooks/users/UserHookService.java c4ff1e4 ambari-server/src/main/java/org/apache/ambari/server/serveraction/users/CsvFilePersisterService.java fe6bf35 ambari-server/src/main/java/org/apache/ambari/server/serveraction/users/PostUserCreationHookServerAction.java e713128 ambari-server/src/main/resources/scripts/post-user-creation-hook.sh ee8d2d1 ambari-server/src/test/java/org/apache/ambari/server/hooks/users/UserHookServiceTest.java 293b22a ambari-server/src/test/java/org/apache/ambari/server/serveraction/users/PostUserCreationHookServerActionTest.java f5cdf48 Diff: https://reviews.apache.org/r/55987/diff/ Testing --- Unit tests running. Manual testing underway Thanks, Laszlo Puskas
Re: Review Request 55987: Post user creation hook script creates user home dirs as the configured hdfs_user
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55987/ --- (Updated Jan. 27, 2017, 4:02 p.m.) Review request for Ambari, Attila Doroszlai, Sandor Magyari, and Sebastian Toader. Changes --- Fixed file permissions (permissions set after creation, thus umask settings don't change them) Bugs: AMBARI-19726 https://issues.apache.org/jira/browse/AMBARI-19726 Repository: ambari Description --- The post user creation hook had wired the **hdfs** user, user homes were created in the name of this user. In case of customized clusters this didn;t work due to permssion issues. The solution is to get the proper hdfs_user from the configureation and access the dfs as this user. Diffs (updated) - ambari-server/src/main/java/org/apache/ambari/server/hooks/users/UserHookParams.java 6970dcc ambari-server/src/main/java/org/apache/ambari/server/hooks/users/UserHookService.java c4ff1e4 ambari-server/src/main/java/org/apache/ambari/server/serveraction/users/CsvFilePersisterService.java fe6bf35 ambari-server/src/main/java/org/apache/ambari/server/serveraction/users/PostUserCreationHookServerAction.java e713128 ambari-server/src/main/resources/scripts/post-user-creation-hook.sh ee8d2d1 ambari-server/src/test/java/org/apache/ambari/server/hooks/users/UserHookServiceTest.java 293b22a ambari-server/src/test/java/org/apache/ambari/server/serveraction/users/PostUserCreationHookServerActionTest.java f5cdf48 Diff: https://reviews.apache.org/r/55987/diff/ Testing --- Unit tests running. Manual testing underway Thanks, Laszlo Puskas
Re: Review Request 55987: Post user creation hook script creates user home dirs as the configured hdfs_user
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55987/ --- (Updated Jan. 27, 2017, 4:12 p.m.) Review request for Ambari, Attila Doroszlai, Sandor Magyari, and Sebastian Toader. Changes --- Unused import ... Bugs: AMBARI-19726 https://issues.apache.org/jira/browse/AMBARI-19726 Repository: ambari Description --- The post user creation hook had wired the **hdfs** user, user homes were created in the name of this user. In case of customized clusters this didn;t work due to permssion issues. The solution is to get the proper hdfs_user from the configureation and access the dfs as this user. Diffs (updated) - ambari-server/src/main/java/org/apache/ambari/server/hooks/users/UserHookParams.java 6970dcc ambari-server/src/main/java/org/apache/ambari/server/hooks/users/UserHookService.java c4ff1e4 ambari-server/src/main/java/org/apache/ambari/server/serveraction/users/CsvFilePersisterService.java fe6bf35 ambari-server/src/main/java/org/apache/ambari/server/serveraction/users/PostUserCreationHookServerAction.java e713128 ambari-server/src/main/resources/scripts/post-user-creation-hook.sh ee8d2d1 ambari-server/src/test/java/org/apache/ambari/server/hooks/users/UserHookServiceTest.java 293b22a ambari-server/src/test/java/org/apache/ambari/server/serveraction/users/PostUserCreationHookServerActionTest.java f5cdf48 Diff: https://reviews.apache.org/r/55987/diff/ Testing --- Unit tests running. Manual testing underway Thanks, Laszlo Puskas
Re: Review Request 55987: Post user creation hook script creates user home dirs as the configured hdfs_user
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55987/ --- (Updated Jan. 27, 2017, 4:12 p.m.) Review request for Ambari, Attila Doroszlai, Sandor Magyari, and Sebastian Toader. Bugs: AMBARI-19726 https://issues.apache.org/jira/browse/AMBARI-19726 Repository: ambari Description --- The post user creation hook had wired the **hdfs** user, user homes were created in the name of this user. In case of customized clusters this didn;t work due to permssion issues. The solution is to get the proper hdfs_user from the configureation and access the dfs as this user. Diffs - ambari-server/src/main/java/org/apache/ambari/server/hooks/users/UserHookParams.java 6970dcc ambari-server/src/main/java/org/apache/ambari/server/hooks/users/UserHookService.java c4ff1e4 ambari-server/src/main/java/org/apache/ambari/server/serveraction/users/CsvFilePersisterService.java fe6bf35 ambari-server/src/main/java/org/apache/ambari/server/serveraction/users/PostUserCreationHookServerAction.java e713128 ambari-server/src/main/resources/scripts/post-user-creation-hook.sh ee8d2d1 ambari-server/src/test/java/org/apache/ambari/server/hooks/users/UserHookServiceTest.java 293b22a ambari-server/src/test/java/org/apache/ambari/server/serveraction/users/PostUserCreationHookServerActionTest.java f5cdf48 Diff: https://reviews.apache.org/r/55987/diff/ Testing (updated) --- Unit tests succeeded. Manual testing underway Thanks, Laszlo Puskas
Re: Review Request 55987: Post user creation hook script creates user home dirs as the configured hdfs_user
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55987/ --- (Updated Jan. 29, 2017, 9:13 a.m.) Review request for Ambari, Attila Doroszlai, Sandor Magyari, and Sebastian Toader. Bugs: AMBARI-19726 https://issues.apache.org/jira/browse/AMBARI-19726 Repository: ambari Description --- The post user creation hook had wired the **hdfs** user, user homes were created in the name of this user. In case of customized clusters this didn;t work due to permssion issues. The solution is to get the proper hdfs_user from the configureation and access the dfs as this user. Diffs - ambari-server/src/main/java/org/apache/ambari/server/hooks/users/UserHookParams.java 6970dcc ambari-server/src/main/java/org/apache/ambari/server/hooks/users/UserHookService.java c4ff1e4 ambari-server/src/main/java/org/apache/ambari/server/serveraction/users/CsvFilePersisterService.java fe6bf35 ambari-server/src/main/java/org/apache/ambari/server/serveraction/users/PostUserCreationHookServerAction.java e713128 ambari-server/src/main/resources/scripts/post-user-creation-hook.sh ee8d2d1 ambari-server/src/test/java/org/apache/ambari/server/hooks/users/UserHookServiceTest.java 293b22a ambari-server/src/test/java/org/apache/ambari/server/serveraction/users/PostUserCreationHookServerActionTest.java f5cdf48 Diff: https://reviews.apache.org/r/55987/diff/ Testing (updated) --- Unit tests succeeded. Manually tested changes. Thanks, Laszlo Puskas
Re: Review Request 56054: Inconsistent auth-to-local rules processing during Kerberos authentication
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56054/#review163430 --- Ship it! Ship It! - Laszlo Puskas On Jan. 29, 2017, 12:50 a.m., Robert Levas wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/56054/ > --- > > (Updated Jan. 29, 2017, 12:50 a.m.) > > > Review request for Ambari, Attila Magyar, Balázs Bence Sári, Eugene > Chekanskiy, Laszlo Puskas, and Sebastian Toader. > > > Bugs: AMBARI-19767 > https://issues.apache.org/jira/browse/AMBARI-19767 > > > Repository: ambari > > > Description > --- > > Facing issue with local to auth rules. > ambari-qa-...@example.com is getting converted to ambari-qa-cl1 as well as > ambari-qa with same ambari configuration ie > authentication.kerberos.auth_to_local.rules=DEFAULT. > > 1st translation : > ``` > 28 Jan 2017 11:44:45,529 INFO [ambari-client-thread-3298] > AmbariAuthToLocalUserDetailsService:102 - Translated > ambari-qa-...@example.com to ambari-qa-cl1 using auth-to-local rules during > Kerberos authentication. > ``` > > 2nd translation : > ``` > 28 Jan 2017 11:47:36,425 INFO [ambari-client-thread-3172] > AmbariAuthToLocalUserDetailsService:102 - Translated > ambari-qa-...@example.com to ambari-qa using auth-to-local rules during > Kerberos authentication. > 28 Jan 2017 11:47:36,428 WARN [ambari-client-thread-3172] > AmbariAuthToLocalUserDetailsService:136 - Failed find user account for user > with username of ambari-qa during Kerberos authentication. > 28 > ``` > > Since authentication.kerberos.auth_to_local.rules=DEFAULT , > 'ambari-qa-...@example.com' should have been translated to 'ambari-qa-cl1'. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/security/authentication/kerberos/AmbariAuthToLocalUserDetailsService.java > c85503c > > Diff: https://reviews.apache.org/r/56054/diff/ > > > Testing > --- > > # Local test results: > > ``` > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > > [INFO] Total time: 35:42.972s > [INFO] Finished at: Sat Jan 28 17:37:32 EST 2017 > [INFO] Final Memory: 66M/1180M > [INFO] > > ``` > > # Jenkins test results: PENDING > > > Thanks, > > Robert Levas > >
Re: Review Request 56783: Removing secure ACLs from Kafka znodes during dekerberization
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56783/#review165942 --- Ship it! Ship It! - Laszlo Puskas On Feb. 17, 2017, 3:16 p.m., Attila Magyar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/56783/ > --- > > (Updated Feb. 17, 2017, 3:16 p.m.) > > > Review request for Ambari, Laszlo Puskas, Robert Levas, and Sebastian Toader. > > > Bugs: AMBARI-20063 > https://issues.apache.org/jira/browse/AMBARI-20063 > > > Repository: ambari > > > Description > --- > > After enabling kerberos kafka sets secure ACLs on certain znodes. These > secure ACLs should be removed by ambari during dekerberization. > > Note that the znode names are hardcoded in the params.py because AFAIK they > are also hardcoded in kafka. I'm still waiting for a kafka developer to > confirm this. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/kafka_broker.py > 0901730 > > ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/params.py > 1d3a195 > > Diff: https://reviews.apache.org/r/56783/diff/ > > > Testing > --- > > Manually: > - created cluster with kafka > - enabled kerberos > - checked that acls on kafka znodes are secure > - disabled kerberos > - checked that acls on kafka znodes are 'world:anyone:crdwa' > > Ran 271 tests in 7.521s > OK > -- > Total run:1173 > Total errors:0 > Total failures:0 > -- > Ran 451 tests in 99.286s > OK > Results : > Tests run: 4925, Failures: 0, Errors: 0, Skipped: 39 > > > Thanks, > > Attila Magyar > >
Review Request 57040: Rebalance HDFS operation returns after the command is issued
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/57040/ --- Review request for Ambari, Robert Levas, Sandor Magyari, and Sebastian Toader. Bugs: AMBARI-20175 https://issues.apache.org/jira/browse/AMBARI-20175 Repository: ambari Description --- The rebalancing operation may take a long time (hours, days) thus when issued from the ambari UI the background operation may time out. As it's not possible to dynamically predict how long the rebalancing will last , the approach taken by this solution is to only trigger the operation and not wait to the operation to finish. (NOTE: after this change the progress of the rebalance operation won't be tracked anymore in the background operation; also the user won't be notified about the success/failure of the operation) Diffs - ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py 123486e Diff: https://reviews.apache.org/r/57040/diff/ Testing --- Manually. Unit tests running. Thanks, Laszlo Puskas
Re: Review Request 57040: Rebalance HDFS operation returns after the command is issued
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/57040/ --- (Updated Feb. 24, 2017, 4:18 p.m.) Review request for Ambari, Robert Levas, Sandor Magyari, and Sebastian Toader. Changes --- Fixed unit tests. Bugs: AMBARI-20175 https://issues.apache.org/jira/browse/AMBARI-20175 Repository: ambari Description --- The rebalancing operation may take a long time (hours, days) thus when issued from the ambari UI the background operation may time out. As it's not possible to dynamically predict how long the rebalancing will last , the approach taken by this solution is to only trigger the operation and not wait to the operation to finish. (NOTE: after this change the progress of the rebalance operation won't be tracked anymore in the background operation; also the user won't be notified about the success/failure of the operation) Diffs (updated) - ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py 123486e ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py fae500f Diff: https://reviews.apache.org/r/57040/diff/ Testing (updated) --- Manually. Unit tests running... Thanks, Laszlo Puskas
Re: Review Request 57040: Rebalance HDFS operation returns after the command is issued
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/57040/ --- (Updated Feb. 24, 2017, 4:34 p.m.) Review request for Ambari, Robert Levas, Sandor Magyari, and Sebastian Toader. Bugs: AMBARI-20175 https://issues.apache.org/jira/browse/AMBARI-20175 Repository: ambari Description --- The rebalancing operation may take a long time (hours, days) thus when issued from the ambari UI the background operation may time out. As it's not possible to dynamically predict how long the rebalancing will last , the approach taken by this solution is to only trigger the operation and not wait to the operation to finish. (NOTE: after this change the progress of the rebalance operation won't be tracked anymore in the background operation; also the user won't be notified about the success/failure of the operation) Diffs - ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py 123486e ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py fae500f Diff: https://reviews.apache.org/r/57040/diff/ Testing (updated) --- Manually. Unit tests success. Thanks, Laszlo Puskas
Re: Review Request 56599: ambari-server upgrade is not idempotent
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56599/#review165330 --- ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog250Test.java (line 1666) <https://reviews.apache.org/r/56599/#comment237174> Instead mocking tehe injector itself, add a module where instances are bound to mocks. - Laszlo Puskas On Feb. 13, 2017, 1:48 p.m., Attila Magyar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/56599/ > --- > > (Updated Feb. 13, 2017, 1:48 p.m.) > > > Review request for Ambari, Jonathan Hurley, Laszlo Puskas, Nate Cole, Robert > Levas, and Sebastian Toader. > > > Bugs: AMBARI-19984 > https://issues.apache.org/jira/browse/AMBARI-19984 > > > Repository: ambari > > > Description > --- > > Running ambari-upgrade multiple times cause an exception due to violation of > unique database constraint (the same RoleAuthorization can be added multiple > times to admin permissions). > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/orm/entities/PermissionEntity.java > fb01cca > > ambari-server/src/main/java/org/apache/ambari/server/upgrade/AbstractUpgradeCatalog.java > 4b33bcd > > ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog242Test.java > 0cd4f12f > > ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog250Test.java > 1c742ef > > Diff: https://reviews.apache.org/r/56599/diff/ > > > Testing > --- > > Added new unittest that checks if adding RoleAuthorization is idempotent. > > Tested manually: > - installed ambari 2.4 > - upgraded to ambari 2.5 > - set version back to 2.4 by running: update metainfo set metainfo_value = > '2.4.0' where metainfo_key = 'version'; > - upgraded to ambari 2.5 again > - started ambari-server checked the UI > > > Existing tests ran clealy > > -- > Ran 465 tests in 15.977s > > Tests run: 4915, Failures: 0, Errors: 0, Skipped: 39 > > > Thanks, > > Attila Magyar > >
Re: Review Request 56599: ambari-server upgrade is not idempotent
> On Feb. 13, 2017, 3:08 p.m., Laszlo Puskas wrote: > > ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog250Test.java, > > line 1672 > > <https://reviews.apache.org/r/56599/diff/1/?file=1631931#file1631931line1672> > > > > Instead mocking tehe injector itself, add a module where instances are > > bound to mocks. > > Attila Magyar wrote: > Tried it but didn't work, there are other dependencies that should be > initialized. Dropped as binding classes to instances cause the framework to interpret - Laszlo --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56599/#review165330 --- On Feb. 13, 2017, 1:48 p.m., Attila Magyar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/56599/ > --- > > (Updated Feb. 13, 2017, 1:48 p.m.) > > > Review request for Ambari, Jonathan Hurley, Laszlo Puskas, Nate Cole, Robert > Levas, and Sebastian Toader. > > > Bugs: AMBARI-19984 > https://issues.apache.org/jira/browse/AMBARI-19984 > > > Repository: ambari > > > Description > --- > > Running ambari-upgrade multiple times cause an exception due to violation of > unique database constraint (the same RoleAuthorization can be added multiple > times to admin permissions). > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/orm/entities/PermissionEntity.java > fb01cca > > ambari-server/src/main/java/org/apache/ambari/server/upgrade/AbstractUpgradeCatalog.java > 4b33bcd > > ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog242Test.java > 0cd4f12f > > ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog250Test.java > 1c742ef > > Diff: https://reviews.apache.org/r/56599/diff/ > > > Testing > --- > > Added new unittest that checks if adding RoleAuthorization is idempotent. > > Tested manually: > - installed ambari 2.4 > - upgraded to ambari 2.5 > - set version back to 2.4 by running: update metainfo set metainfo_value = > '2.4.0' where metainfo_key = 'version'; > - upgraded to ambari 2.5 again > - started ambari-server checked the UI > > > Existing tests ran clealy > > -- > Ran 465 tests in 15.977s > > Tests run: 4915, Failures: 0, Errors: 0, Skipped: 39 > > > Thanks, > > Attila Magyar > >
Re: Review Request 56599: ambari-server upgrade is not idempotent
> On Feb. 13, 2017, 3:08 p.m., Laszlo Puskas wrote: > > ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog250Test.java, > > line 1672 > > <https://reviews.apache.org/r/56599/diff/1/?file=1631931#file1631931line1672> > > > > Instead mocking tehe injector itself, add a module where instances are > > bound to mocks. > > Attila Magyar wrote: > Tried it but didn't work, there are other dependencies that should be > initialized. > > Laszlo Puskas wrote: > Dropped as binding classes to instances cause the framework to interpret annotations and tries to inject all dependencies. - Laszlo --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56599/#review165330 --- On Feb. 13, 2017, 1:48 p.m., Attila Magyar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/56599/ > --- > > (Updated Feb. 13, 2017, 1:48 p.m.) > > > Review request for Ambari, Jonathan Hurley, Laszlo Puskas, Nate Cole, Robert > Levas, and Sebastian Toader. > > > Bugs: AMBARI-19984 > https://issues.apache.org/jira/browse/AMBARI-19984 > > > Repository: ambari > > > Description > --- > > Running ambari-upgrade multiple times cause an exception due to violation of > unique database constraint (the same RoleAuthorization can be added multiple > times to admin permissions). > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/orm/entities/PermissionEntity.java > fb01cca > > ambari-server/src/main/java/org/apache/ambari/server/upgrade/AbstractUpgradeCatalog.java > 4b33bcd > > ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog242Test.java > 0cd4f12f > > ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog250Test.java > 1c742ef > > Diff: https://reviews.apache.org/r/56599/diff/ > > > Testing > --- > > Added new unittest that checks if adding RoleAuthorization is idempotent. > > Tested manually: > - installed ambari 2.4 > - upgraded to ambari 2.5 > - set version back to 2.4 by running: update metainfo set metainfo_value = > '2.4.0' where metainfo_key = 'version'; > - upgraded to ambari 2.5 again > - started ambari-server checked the UI > > > Existing tests ran clealy > > -- > Ran 465 tests in 15.977s > > Tests run: 4915, Failures: 0, Errors: 0, Skipped: 39 > > > Thanks, > > Attila Magyar > >
Re: Review Request 56713: Ambari server start returns prematurely before extracting views.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56713/#review165719 --- Ship it! Ship It! - Laszlo Puskas On Feb. 15, 2017, 3:24 p.m., Dmytro Sen wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/56713/ > --- > > (Updated Feb. 15, 2017, 3:24 p.m.) > > > Review request for Ambari, Laszlo Puskas, Sebastian Toader, Sid Wagle, and > Vitalyi Brodetskyi. > > > Bugs: AMBARI-20026 > https://issues.apache.org/jira/browse/AMBARI-20026 > > > Repository: ambari > > > Description > --- > > When ambari-server start is issued, It returns prematurely before the view > extraction is complete. > because of this we have introduced sleep of 3mins before ambari-server stop > for the view extraction to complete. > Ambari server should report start completed only after view extraction is > completed > > > Diffs > - > > ambari-server/src/main/python/ambari_server/serverConfiguration.py d3bfa70 > ambari-server/src/main/python/ambari_server/utils.py f65ceed > ambari-server/src/main/python/ambari_server_main.py df1b0a1 > ambari-server/src/test/python/TestAmbariServer.py 8692030 > > Diff: https://reviews.apache.org/r/56713/diff/ > > > Testing > --- > > [INFO] Reactor Summary: > [INFO] > [INFO] Ambari Main ... SUCCESS [4.125s] > [INFO] Apache Ambari Project POM . SUCCESS [0.006s] > [INFO] utility ... SUCCESS [2.523s] > [INFO] Ambari Agent .. SUCCESS [34.664s] > [INFO] > > [INFO] BUILD SUCCESS > > > Thanks, > > Dmytro Sen > >
Review Request 55883: Post user creation hook - input csv generated with READ permissions
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55883/ --- Review request for Ambari, Attila Magyar, Robert Levas, Sandor Magyari, and Sebastian Toader. Bugs: AMBARI-19694 https://issues.apache.org/jira/browse/AMBARI-19694 Repository: ambari Description --- Currently the post user creation hook mechanism generates the user input csv with read permissions only for the user that runs ambari. This may cause issues on secure clusters, where ambari is not ran by the root user, az auth to local rules may map to users that don't have READ right for the generated csv. Solution: input csv file is generated with REAd permissions for all users. Diffs - ambari-server/src/main/java/org/apache/ambari/server/serveraction/users/CsvFilePersisterService.java d8ffe98 ambari-server/src/test/java/org/apache/ambari/server/serveraction/users/CsvFilePersisterServiceFunctionalTest.java PRE-CREATION Diff: https://reviews.apache.org/r/55883/diff/ Testing --- Unit test created. Manual testing under way. Thanks, Laszlo Puskas
Re: Review Request 55833: Supporting zookeeper security only from HDP 2.6
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55833/#review162805 --- Ship it! Ship It! - Laszlo Puskas On Jan. 24, 2017, 1 p.m., Attila Magyar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/55833/ > --- > > (Updated Jan. 24, 2017, 1 p.m.) > > > Review request for Ambari, Laszlo Puskas, Robert Levas, and Sebastian Toader. > > > Bugs: AMBARI-19668 > https://issues.apache.org/jira/browse/AMBARI-19668 > > > Repository: ambari > > > Description > --- > > Secure zookeeper ACLs should be supported only in HDP 2.6 and above. This > patch is about reworking the existing patches (oozie, hive, yarn-rm) and > removing secure zk acl support if stack version is < 2.6 > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/constants.py > 02ce194 > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/hadoop-env.xml > bc64d1f > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/kerberos.json > ac3b782 > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py > 31431b9 > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/zkfc_slave.py > 69cd2a5 > > ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/params_linux.py > 59ae815 > > ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/zkfc_slave.py > 92e4182 > > ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py > 1a34b87 > > ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py > 48c8ef0 > > ambari-server/src/main/resources/common-services/OOZIE/4.2.0.2.3/kerberos.json > f1092f5 > > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/kerberos.json > c8b5989 > > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py > 7df82bf > > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py > 77bd363 > > ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/kerberos.json > eaffec6 > > ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py > d4e505a > > ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json > e4a499b > > ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/hadoop-env.xml > ef111e0 > ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/kerberos.json > a8ef83c > > ambari-server/src/main/resources/stacks/HDP/2.3.ECS/services/YARN/kerberos.json > 3059f14 > > ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/configuration/hadoop-env.xml > 0212ba0 > ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/kerberos.json > 5fff05c > > ambari-server/src/main/resources/stacks/HDP/2.4/services/HDFS/configuration/hadoop-env.xml > 0212ba0 > ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/kerberos.json > 58942aa > ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/kerberos.json > eaffec6 > > ambari-server/src/main/resources/stacks/HDP/2.6/services/HDFS/configuration/hadoop-env.xml > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.6/services/HDFS/kerberos.json > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.6/services/OOZIE/kerberos.json > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.6/services/YARN/kerberos.json > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/3.0/hooks/before-ANY/scripts/params.py > 74f56a8 > > ambari-server/src/main/resources/stacks/HDP/3.0/properties/stack_features.json > ddf8348 > > ambari-server/src/main/resources/stacks/HDP/3.0/services/HDFS/configuration/hadoop-env.xml > 13ef4ba > > ambari-server/src/main/resources/stacks/PERF/1.0/properties/stack_features.json > 81640b6 > ambari-server/src/test/python/stacks/2.0.6/HDFS/test_zkfc.py aa9e9bc > > contrib/management-packs/hdf-ambari-mpack/src/main/resources/stacks/HDF/2.0/properties/stack_features.json > 0b6b3ab > > Diff: https://re
Review Request 56600: Default group permissions for the user home directories created by the post user creation script should be 'hdfs' instead of 'hadoop'
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56600/ --- Review request for Ambari, Attila Doroszlai, Attila Magyar, and Sebastian Toader. Bugs: AMBARI-19986 https://issues.apache.org/jira/browse/AMBARI-19986 Repository: ambari Description --- Default group permissions for the user home directories created by the post user creation script should be 'hdfs' instead of 'hadoop' Diffs - ambari-server/src/main/resources/scripts/post-user-creation-hook.sh 91511a0 Diff: https://reviews.apache.org/r/56600/diff/ Testing --- Manually, underway. Thanks, Laszlo Puskas
Re: Review Request 55680: On secure NN HA clusters ZKFC connects to zookeeper securely
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55680/ --- (Updated Jan. 19, 2017, 12:39 p.m.) Review request for Ambari, Attila Magyar, Robert Levas, and Sebastian Toader. Bugs: AMBARI-19613 https://issues.apache.org/jira/browse/AMBARI-19613 Repository: ambari Description --- On secure namenode HA clusters the ZKFC component needs to access the zookeeper securely. On enabling security appropriate settings are configured to secure this connection. Diffs - ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/hadoop-env.xml c2f37c1 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/kerberos.json f30c9e4 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py 3270430 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/zkfc_slave.py f1891a5 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/templates/hdfs_jaas.conf.j2 PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py 783f811 ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/hadoop-env.xml 5be2b74 ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/configuration/hadoop-env.xml 24e0193 ambari-server/src/main/resources/stacks/HDP/2.4/services/HDFS/configuration/hadoop-env.xml 24e0193 ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/kerberos.json 9000e95 ambari-server/src/main/resources/stacks/HDP/3.0/hooks/before-ANY/scripts/params.py f70c8e9 ambari-server/src/main/resources/stacks/HDP/3.0/services/HDFS/configuration/hadoop-env.xml e680c1b Diff: https://reviews.apache.org/r/55680/diff/ Testing (updated) --- Testing done manually: Created an unsecure NN HA cluster * checked the configuration entry: ha.zookeeper.acl - doesn't exist * checked the hadoop-env.sh - doesn't contain the variable export HADOOP_ZKFC_OPTS * checked the hdfs_jaas.conf - doesn't exist * connected to zookeeper, listed znode acls - no limitations set Kerberized the NN HA cluster * checked the configuration entry: ha.zookeeper.acl - set to sasl:nn:cdrwa * checked the hadoop-env.sh - contains the variable export HADOOP_ZKFC_OPTS with proper value, points to the correct jaas file * checked the hdfs_jaas.conf - OK * connected to zookeeper, listed znode acls - set as required (/hadoop-ha/mycluster/ActiveStandbyElectorLock) Disabled Kerberos on the NN HA cluster * checked the configuration entry: ha.zookeeper.acl - removed * checked the hadoop-env.sh - doesn't contain the variable export HADOOP_ZKFC_OPTS * connected to zookeeper, listed znode acls - set as required (/hadoop-ha/mycluster/ActiveStandbyElectorLock) Unit tests: Successfully ran on local machine / unrelated test failed though. Thanks, Laszlo Puskas
Re: Review Request 55513: Use common property for principal name prefix to help with customization of unique principal names
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55513/#review161720
---
Ship it!
Ship It!
- Laszlo Puskas
On Jan. 13, 2017, 6:14 p.m., Robert Levas wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55513/
> ---
>
> (Updated Jan. 13, 2017, 6:14 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Attila Magyar, Eugene
> Chekanskiy, Laszlo Puskas, and Sebastian Toader.
>
>
> Bugs: AMBARI-19430
> https://issues.apache.org/jira/browse/AMBARI-19430
>
>
> Repository: ambari
>
>
> Description
> ---
>
> Use common property for principal name prefix to help with customization of
> unique principal names.
>
> All _headless_ Kerberos identities have a non-unique principal name (across
> clusters). To help this issue, the cluster name is appended to these
> principal names by adding "-${cluster-name|toLower()}" after the principal
> name component. If the user wants to change this convention, they will need
> to find all _headless_ principals and make the change. On top of that, when
> adding new components, they will need to remember to make the change to new
> _headless_ principal names.
>
> A better solution is to provide a _global_ property named "principal_suffix"
> and use that in each _headless_ principal name. By default the value for this
> property will be
>
> ```
> principal_suffix="-${cluster_name|toLower()}"
> ```
>
> If the user would like not use a prefix (in the event there is only a single
> cluster connecting to the KDC), the value can be changed to
>
> ```
> principal_suffix=""
> ```
>
> Finally if the user would like to use some other randomizer, they can set the
> value to something else. For example
>
> ```
> principal_suffix="_12345"
> ```
>
> The property is set in the Kerberos descriptor's "properties" block. For
> example:
>
> ```
> {
> "properties": {
> "realm": "${kerberos-env/realm}",
> ...,
> "principal_suffix": "-${cluster_name|toLower()}"
> },
> "identities": [
> ...,
> {
> "name": "smokeuser",
> "principal": {
> "value": "${cluster-env/smokeuser}${principal_suffix}@${realm}",
> "type": "user",
> "configuration": "cluster-env/smokeuser_principal_name",
> "local_username": "${cluster-env/smokeuser}"
> },
> ...
> }
> ],
> "services": [
> {
> ```
>
>
> Diffs
> -
>
>
> ambari-funtest/src/test/resources/stacks/HDP/2.0.8/services/HDFS/kerberos.json
> d53205d
>
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/kerberos.json
> caef123
>
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/kerberos.json
> 636d36e
>
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/kerberos.json
> f30c9e4
>
> ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/kerberos.json
> 1dd801b
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/kerberos.json
> fa6af33
> ambari-server/src/main/resources/common-services/SPARK/1.4.1/kerberos.json
> e7f78cd
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/kerberos.json
> 20e1dc0
> ambari-server/src/main/resources/common-services/STORM/0.9.1/kerberos.json
> fcfe524
> ambari-server/src/main/resources/common-services/STORM/1.0.1/kerberos.json
> 3068226
>
> ambari-server/src/main/resources/common-services/ZEPPELIN/0.6.0.2.5/kerberos.json
> 7c4c04c
> ambari-server/src/main/resources/stacks/HDP/2.0.6/kerberos.json 9579d0f
>
> ambari-server/src/main/resources/stacks/HDP/2.3.ECS/services/ECS/kerberos.json
> 9668354
>
> ambari-server/src/main/resources/stacks/HDP/2.3.ECS/services/HBASE/kerberos.json
> 20b10f7
>
> ambari-server/src/main/resources/stacks/HDP/2.3.GlusterFS/services/ACCUMULO/kerberos.json
> 678a2b5
>
> ambari-server/src/main/resources/stacks/HDP/2.3/services/ACCUMULO/kerberos.json
> 0fec0ab
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HBASE/kerberos.json
> 9ed40ef
> ambari-server/
Re: Review Request 55643: Blueprint installation should accept quick link profile
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55643/#review162065 --- ambari-server/src/main/java/org/apache/ambari/server/utils/DefaultTimeSource.java (line 29) <https://reviews.apache.org/r/55643/#comment233314> Why is this abstarction needed? Couldn't we simply use Calendar instead? - Laszlo Puskas On Jan. 17, 2017, 9:30 p.m., Balázs Bence Sári wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/55643/ > --- > > (Updated Jan. 17, 2017, 9:30 p.m.) > > > Review request for Ambari, Attila Doroszlai, Laszlo Puskas, Oliver Szabo, > Robert Nettleton, Sandor Magyari, Sumit Mohanty, and Sebastian Toader. > > > Bugs: AMBARI-19597 > https://issues.apache.org/jira/browse/AMBARI-19597 > > > Repository: ambari > > > Description > --- > > Specifying the quick links profile should be supported in blueprint based > cluster installation. First implementation will address the possibility of > adding the profile to the cluster creation template. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java > 5e8c803 > > ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java > b7c9e85 > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterResourceProvider.java > cb30f2d > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ProvisionClusterRequest.java > a35da86 > > ambari-server/src/main/java/org/apache/ambari/server/state/quicklinksprofile/AcceptAllFilter.java > d784a22 > > ambari-server/src/main/java/org/apache/ambari/server/state/quicklinksprofile/Component.java > a1267df > > ambari-server/src/main/java/org/apache/ambari/server/state/quicklinksprofile/Filter.java > c551830 > > ambari-server/src/main/java/org/apache/ambari/server/state/quicklinksprofile/QuickLinksProfile.java > c9ac6b4 > > ambari-server/src/main/java/org/apache/ambari/server/state/quicklinksprofile/QuickLinksProfileBuilder.java > PRE-CREATION > > ambari-server/src/main/java/org/apache/ambari/server/state/quicklinksprofile/QuickLinksProfileEvaluationException.java > 26819e1 > > ambari-server/src/main/java/org/apache/ambari/server/state/quicklinksprofile/QuickLinksProfileParser.java > a3ae677 > > ambari-server/src/main/java/org/apache/ambari/server/state/quicklinksprofile/Service.java > 7724852 > > ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java > 7db07a0 > > ambari-server/src/main/java/org/apache/ambari/server/utils/DefaultTimeSource.java > PRE-CREATION > ambari-server/src/main/java/org/apache/ambari/server/utils/TimeSource.java > PRE-CREATION > > ambari-server/src/test/java/org/apache/ambari/server/agent/AgentResourceTest.java > 17b1e27 > > ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ProvisionClusterRequestTest.java > 2cf478a > > ambari-server/src/test/java/org/apache/ambari/server/controller/internal/QuickLinkArtifactResourceProviderTest.java > 8c723c9 > > ambari-server/src/test/java/org/apache/ambari/server/state/quicklinksprofile/QuickLinksProfileBuilderTest.java > PRE-CREATION > > ambari-server/src/test/java/org/apache/ambari/server/topology/TopologyManagerTest.java > 7e6e5a3 > > ambari-server/src/test/java/org/apache/ambari/server/utils/StageUtilsTest.java > 5c77831 > > Diff: https://reviews.apache.org/r/55643/diff/ > > > Testing > --- > > - Wrote new unit tests > - Run the unit test suite for ambari-server. No failures. > - Covered the following cases by manual testing: >-- Installing a cluster via blueprint, quick links profile in the cluster > creation template, profile saved the first time >-- Installing a cluster via blueprint, quick links profile in the cluster > creation template, there was an existing quick links profile which was > overwritten during cluster installation >-- Installing a cluster via blueprint, no quick links profile in the > cluster creation template > > > Thanks, > > Balázs Bence Sári > >
Re: Review Request 55680: On secure NN HA clusters ZKFC connects to zookeeper securely
> On Jan. 18, 2017, 3:57 p.m., Attila Magyar wrote: > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/templates/hdfs_jaas.conf.j2, > > line 24 > > <https://reviews.apache.org/r/55680/diff/1/?file=1607730#file1607730line24> > > > > is this path always the same? > > Robert Levas wrote: > That path should not be hard-coded. It could change. Thanks, will fix it. - Laszlo --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55680/#review162123 ----------- On Jan. 18, 2017, 3:58 p.m., Laszlo Puskas wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/55680/ > --- > > (Updated Jan. 18, 2017, 3:58 p.m.) > > > Review request for Ambari, Attila Magyar, Robert Levas, and Sebastian Toader. > > > Bugs: AMBARI-19613 > https://issues.apache.org/jira/browse/AMBARI-19613 > > > Repository: ambari > > > Description > --- > > On secure namenode HA clusters the ZKFC component needs to access the > zookeeper securely. > On enabling security appropriate settings are configured to secure this > connection. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/hadoop-env.xml > c2f37c1 > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/kerberos.json > f30c9e4 > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py > 3270430 > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/zkfc_slave.py > f1891a5 > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/templates/hdfs_jaas.conf.j2 > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py > 783f811 > > ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/hadoop-env.xml > 5be2b74 > > ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/configuration/hadoop-env.xml > 24e0193 > > ambari-server/src/main/resources/stacks/HDP/2.4/services/HDFS/configuration/hadoop-env.xml > 24e0193 > ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/kerberos.json > 9000e95 > > Diff: https://reviews.apache.org/r/55680/diff/ > > > Testing > --- > > Testing done manually: > > Created an unsecure NN HA cluster > > * checked the configuration entry: ha.zookeeper.acl - doesn't exist > * checked the hadoop-env.sh - doesn't contain the variable export > HADOOP_ZKFC_OPTS > * checked the hdfs_jaas.conf - doesn't exist > * connected to zookeeper, listed znode acls - no limitations set > > Kerberized the NN HA cluster > > * checked the configuration entry: ha.zookeeper.acl - set to sasl:nn:cdrwa > * checked the hadoop-env.sh - contains the variable export HADOOP_ZKFC_OPTS > with proper value, points to the correct jaas file > * checked the hdfs_jaas.conf - OK > > Disabled Kerberos on the NN HA cluster > > * checked the configuration entry: ha.zookeeper.acl - removed > * checked the hadoop-env.sh - doesn't contain the variable export > HADOOP_ZKFC_OPTS > > Unit tests running. > > > Thanks, > > Laszlo Puskas > >
Re: Review Request 55680: On secure NN HA clusters ZKFC connects to zookeeper securely
> On Jan. 18, 2017, 4:12 p.m., Robert Levas wrote:
> > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/templates/hdfs_jaas.conf.j2,
> > line 24
> > <https://reviews.apache.org/r/55680/diff/1/?file=1607730#file1607730line24>
> >
> > This should be
> >
> > ```
> > keytab="{{nn_keytab}}"
> > ```
Thanks!
- Laszlo
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55680/#review162125
-------
On Jan. 18, 2017, 3:58 p.m., Laszlo Puskas wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55680/
> ---
>
> (Updated Jan. 18, 2017, 3:58 p.m.)
>
>
> Review request for Ambari, Attila Magyar, Robert Levas, and Sebastian Toader.
>
>
> Bugs: AMBARI-19613
> https://issues.apache.org/jira/browse/AMBARI-19613
>
>
> Repository: ambari
>
>
> Description
> ---
>
> On secure namenode HA clusters the ZKFC component needs to access the
> zookeeper securely.
> On enabling security appropriate settings are configured to secure this
> connection.
>
>
> Diffs
> -
>
>
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/hadoop-env.xml
> c2f37c1
>
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/kerberos.json
> f30c9e4
>
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py
> 3270430
>
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/zkfc_slave.py
> f1891a5
>
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/templates/hdfs_jaas.conf.j2
> PRE-CREATION
>
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py
> 783f811
>
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/hadoop-env.xml
> 5be2b74
>
> ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/configuration/hadoop-env.xml
> 24e0193
>
> ambari-server/src/main/resources/stacks/HDP/2.4/services/HDFS/configuration/hadoop-env.xml
> 24e0193
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/kerberos.json
> 9000e95
>
> Diff: https://reviews.apache.org/r/55680/diff/
>
>
> Testing
> ---
>
> Testing done manually:
>
> Created an unsecure NN HA cluster
>
> * checked the configuration entry: ha.zookeeper.acl - doesn't exist
> * checked the hadoop-env.sh - doesn't contain the variable export
> HADOOP_ZKFC_OPTS
> * checked the hdfs_jaas.conf - doesn't exist
> * connected to zookeeper, listed znode acls - no limitations set
>
> Kerberized the NN HA cluster
>
> * checked the configuration entry: ha.zookeeper.acl - set to sasl:nn:cdrwa
> * checked the hadoop-env.sh - contains the variable export HADOOP_ZKFC_OPTS
> with proper value, points to the correct jaas file
> * checked the hdfs_jaas.conf - OK
>
> Disabled Kerberos on the NN HA cluster
>
> * checked the configuration entry: ha.zookeeper.acl - removed
> * checked the hadoop-env.sh - doesn't contain the variable export
> HADOOP_ZKFC_OPTS
>
> Unit tests running.
>
>
> Thanks,
>
> Laszlo Puskas
>
>
Re: Review Request 55680: On secure NN HA clusters ZKFC connects to zookeeper securely
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55680/ --- (Updated Jan. 18, 2017, 3:58 p.m.) Review request for Ambari, Attila Magyar, Robert Levas, and Sebastian Toader. Bugs: AMBARI-19613 https://issues.apache.org/jira/browse/AMBARI-19613 Repository: ambari Description --- On secure namenode HA clusters the ZKFC component needs to access the zookeeper securely. On enabling security appropriate settings are configured to secure this connection. Diffs - ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/hadoop-env.xml c2f37c1 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/kerberos.json f30c9e4 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py 3270430 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/zkfc_slave.py f1891a5 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/templates/hdfs_jaas.conf.j2 PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py 783f811 ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/hadoop-env.xml 5be2b74 ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/configuration/hadoop-env.xml 24e0193 ambari-server/src/main/resources/stacks/HDP/2.4/services/HDFS/configuration/hadoop-env.xml 24e0193 ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/kerberos.json 9000e95 Diff: https://reviews.apache.org/r/55680/diff/ Testing (updated) --- Testing done manually: Created an unsecure NN HA cluster * checked the configuration entry: ha.zookeeper.acl - doesn't exist * checked the hadoop-env.sh - doesn't contain the variable export HADOOP_ZKFC_OPTS * checked the hdfs_jaas.conf - doesn't exist * connected to zookeeper, listed znode acls - no limitations set Kerberized the NN HA cluster * checked the configuration entry: ha.zookeeper.acl - set to sasl:nn:cdrwa * checked the hadoop-env.sh - contains the variable export HADOOP_ZKFC_OPTS with proper value, points to the correct jaas file * checked the hdfs_jaas.conf - OK Disabled Kerberos on the NN HA cluster * checked the configuration entry: ha.zookeeper.acl - removed * checked the hadoop-env.sh - doesn't contain the variable export HADOOP_ZKFC_OPTS Unit tests running. Thanks, Laszlo Puskas
Re: Review Request 55680: On secure NN HA clusters ZKFC connects to zookeeper securely
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55680/ --- (Updated Jan. 18, 2017, 5:17 p.m.) Review request for Ambari, Attila Magyar, Robert Levas, and Sebastian Toader. Changes --- Added changes to the stack 3.0 Bugs: AMBARI-19613 https://issues.apache.org/jira/browse/AMBARI-19613 Repository: ambari Description --- On secure namenode HA clusters the ZKFC component needs to access the zookeeper securely. On enabling security appropriate settings are configured to secure this connection. Diffs (updated) - ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/hadoop-env.xml c2f37c1 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/kerberos.json f30c9e4 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py 3270430 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/zkfc_slave.py f1891a5 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/templates/hdfs_jaas.conf.j2 PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py 783f811 ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/hadoop-env.xml 5be2b74 ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/configuration/hadoop-env.xml 24e0193 ambari-server/src/main/resources/stacks/HDP/2.4/services/HDFS/configuration/hadoop-env.xml 24e0193 ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/kerberos.json 9000e95 ambari-server/src/main/resources/stacks/HDP/3.0/hooks/before-ANY/scripts/params.py f70c8e9 ambari-server/src/main/resources/stacks/HDP/3.0/services/HDFS/configuration/hadoop-env.xml e680c1b Diff: https://reviews.apache.org/r/55680/diff/ Testing --- Testing done manually: Created an unsecure NN HA cluster * checked the configuration entry: ha.zookeeper.acl - doesn't exist * checked the hadoop-env.sh - doesn't contain the variable export HADOOP_ZKFC_OPTS * checked the hdfs_jaas.conf - doesn't exist * connected to zookeeper, listed znode acls - no limitations set Kerberized the NN HA cluster * checked the configuration entry: ha.zookeeper.acl - set to sasl:nn:cdrwa * checked the hadoop-env.sh - contains the variable export HADOOP_ZKFC_OPTS with proper value, points to the correct jaas file * checked the hdfs_jaas.conf - OK Disabled Kerberos on the NN HA cluster * checked the configuration entry: ha.zookeeper.acl - removed * checked the hadoop-env.sh - doesn't contain the variable export HADOOP_ZKFC_OPTS Unit tests running. Thanks, Laszlo Puskas
Re: Review Request 55680: On secure NN HA clusters ZKFC connects to zookeeper securely
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55680/ --- (Updated Jan. 18, 2017, 5:18 p.m.) Review request for Ambari, Attila Magyar, Robert Levas, and Sebastian Toader. Bugs: AMBARI-19613 https://issues.apache.org/jira/browse/AMBARI-19613 Repository: ambari Description --- On secure namenode HA clusters the ZKFC component needs to access the zookeeper securely. On enabling security appropriate settings are configured to secure this connection. Diffs - ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/hadoop-env.xml c2f37c1 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/kerberos.json f30c9e4 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py 3270430 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/zkfc_slave.py f1891a5 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/templates/hdfs_jaas.conf.j2 PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py 783f811 ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/hadoop-env.xml 5be2b74 ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/configuration/hadoop-env.xml 24e0193 ambari-server/src/main/resources/stacks/HDP/2.4/services/HDFS/configuration/hadoop-env.xml 24e0193 ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/kerberos.json 9000e95 ambari-server/src/main/resources/stacks/HDP/3.0/hooks/before-ANY/scripts/params.py f70c8e9 ambari-server/src/main/resources/stacks/HDP/3.0/services/HDFS/configuration/hadoop-env.xml e680c1b Diff: https://reviews.apache.org/r/55680/diff/ Testing (updated) --- Testing done manually: Created an unsecure NN HA cluster * checked the configuration entry: ha.zookeeper.acl - doesn't exist * checked the hadoop-env.sh - doesn't contain the variable export HADOOP_ZKFC_OPTS * checked the hdfs_jaas.conf - doesn't exist * connected to zookeeper, listed znode acls - no limitations set Kerberized the NN HA cluster * checked the configuration entry: ha.zookeeper.acl - set to sasl:nn:cdrwa * checked the hadoop-env.sh - contains the variable export HADOOP_ZKFC_OPTS with proper value, points to the correct jaas file * checked the hdfs_jaas.conf - OK Disabled Kerberos on the NN HA cluster * checked the configuration entry: ha.zookeeper.acl - removed * checked the hadoop-env.sh - doesn't contain the variable export HADOOP_ZKFC_OPTS Unit tests: Successfully ran on local machine / unrelated test failed though. Thanks, Laszlo Puskas
Re: Review Request 55680: On secure NN HA clusters ZKFC connects to zookeeper securely
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55680/ --- (Updated Jan. 18, 2017, 4:35 p.m.) Review request for Ambari, Attila Magyar, Robert Levas, and Sebastian Toader. Bugs: AMBARI-19613 https://issues.apache.org/jira/browse/AMBARI-19613 Repository: ambari Description --- On secure namenode HA clusters the ZKFC component needs to access the zookeeper securely. On enabling security appropriate settings are configured to secure this connection. Diffs (updated) - ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/hadoop-env.xml c2f37c1 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/kerberos.json f30c9e4 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py 3270430 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/zkfc_slave.py f1891a5 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/templates/hdfs_jaas.conf.j2 PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py 783f811 ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/hadoop-env.xml 5be2b74 ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/configuration/hadoop-env.xml 24e0193 ambari-server/src/main/resources/stacks/HDP/2.4/services/HDFS/configuration/hadoop-env.xml 24e0193 ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/kerberos.json 9000e95 Diff: https://reviews.apache.org/r/55680/diff/ Testing --- Testing done manually: Created an unsecure NN HA cluster * checked the configuration entry: ha.zookeeper.acl - doesn't exist * checked the hadoop-env.sh - doesn't contain the variable export HADOOP_ZKFC_OPTS * checked the hdfs_jaas.conf - doesn't exist * connected to zookeeper, listed znode acls - no limitations set Kerberized the NN HA cluster * checked the configuration entry: ha.zookeeper.acl - set to sasl:nn:cdrwa * checked the hadoop-env.sh - contains the variable export HADOOP_ZKFC_OPTS with proper value, points to the correct jaas file * checked the hdfs_jaas.conf - OK Disabled Kerberos on the NN HA cluster * checked the configuration entry: ha.zookeeper.acl - removed * checked the hadoop-env.sh - doesn't contain the variable export HADOOP_ZKFC_OPTS Unit tests running. Thanks, Laszlo Puskas
Review Request 55680: On secure NN HA clusters ZKFC connects to zookeeper securely
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55680/ --- Review request for Ambari, Attila Magyar, Robert Levas, and Sebastian Toader. Bugs: AMBARI-19613 https://issues.apache.org/jira/browse/AMBARI-19613 Repository: ambari Description --- On secure namenode HA clusters the ZKFC component needs to access the zookeeper securely. On enabling security appropriate settings are configured to secure this connection. Diffs - ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/hadoop-env.xml c2f37c1 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/kerberos.json f30c9e4 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py 3270430 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/zkfc_slave.py f1891a5 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/templates/hdfs_jaas.conf.j2 PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py 783f811 ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/hadoop-env.xml 5be2b74 ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/configuration/hadoop-env.xml 24e0193 ambari-server/src/main/resources/stacks/HDP/2.4/services/HDFS/configuration/hadoop-env.xml 24e0193 ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/kerberos.json 9000e95 Diff: https://reviews.apache.org/r/55680/diff/ Testing --- Testing done manually: 1. Created an unsecure NN HA cluster * checked the configuration entry: ha.zookeeper.acl - doesn't exist * checked the hadoop-env.sh - doesn't contain the variable export HADOOP_ZKFC_OPTS * checked the hdfs_jaas.conf - doesn't exist * connected to zookeeper, listed znode acls - no limitations set 2. Kerberized the NN HA cluster * checked the configuration entry: ha.zookeeper.acl - set to sasl:nn:cdrwa * checked the hadoop-env.sh - contains the variable export HADOOP_ZKFC_OPTS with proper value, points to the correct jaas file * checked the hdfs_jaas.conf - OK 3. Disabled Kerberos on the NN HA cluster * checked the configuration entry: ha.zookeeper.acl - removed * checked the hadoop-env.sh - doesn't contain the variable export HADOOP_ZKFC_OPTS Unit tests running. Thanks, Laszlo Puskas
Re: Review Request 55680: On secure NN HA clusters ZKFC connects to zookeeper securely
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55680/ --- (Updated Jan. 20, 2017, 11:21 a.m.) Review request for Ambari, Attila Magyar, Robert Levas, and Sebastian Toader. Changes --- Added changes to hdp 3.0 Fixed tests. Bugs: AMBARI-19613 https://issues.apache.org/jira/browse/AMBARI-19613 Repository: ambari Description --- On secure namenode HA clusters the ZKFC component needs to access the zookeeper securely. On enabling security appropriate settings are configured to secure this connection. Diffs (updated) - ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/hadoop-env.xml c2f37c1 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/kerberos.json 1cf1603 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py 3270430 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/zkfc_slave.py f1891a5 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/templates/hdfs_jaas.conf.j2 PRE-CREATION ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/configuration/hadoop-env.xml 24032fa ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/kerberos.json 4fdffcf ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/utils.py f76935a ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/zkfc_slave.py f1891a5 ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/templates/hdfs_jaas.conf.j2 PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py 783f811 ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/hadoop-env.xml 5be2b74 ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/configuration/hadoop-env.xml 24e0193 ambari-server/src/main/resources/stacks/HDP/2.4/services/HDFS/configuration/hadoop-env.xml 24e0193 ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/kerberos.json 766a014 ambari-server/src/main/resources/stacks/HDP/3.0/hooks/before-ANY/scripts/params.py f70c8e9 ambari-server/src/main/resources/stacks/HDP/3.0/services/HDFS/configuration/hadoop-env.xml e680c1b ambari-server/src/test/python/stacks/2.0.6/HDFS/test_zkfc.py e952108 Diff: https://reviews.apache.org/r/55680/diff/ Testing (updated) --- Testing done manually: Created an unsecure NN HA cluster * checked the configuration entry: ha.zookeeper.acl - doesn't exist * checked the hadoop-env.sh - doesn't contain the variable export HADOOP_ZKFC_OPTS * checked the hdfs_jaas.conf - doesn't exist * connected to zookeeper, listed znode acls - no limitations set Kerberized the NN HA cluster * checked the configuration entry: ha.zookeeper.acl - set to sasl:nn:cdrwa * checked the hadoop-env.sh - contains the variable export HADOOP_ZKFC_OPTS with proper value, points to the correct jaas file * checked the hdfs_jaas.conf - OK * connected to zookeeper, listed znode acls - set as required (/hadoop-ha/mycluster/ActiveStandbyElectorLock) Disabled Kerberos on the NN HA cluster * checked the configuration entry: ha.zookeeper.acl - removed * checked the hadoop-env.sh - doesn't contain the variable export HADOOP_ZKFC_OPTS * connected to zookeeper, listed znode acls - set as required (/hadoop-ha/mycluster/ActiveStandbyElectorLock) Unit tests: Running in progress for trunk Thanks, Laszlo Puskas
Re: Review Request 55764: Ldap sync fails when there are special characters in distinguished names
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55764/#review162439 --- Ship it! Ship It! - Laszlo Puskas On Jan. 20, 2017, 4:31 p.m., Robert Levas wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/55764/ > --- > > (Updated Jan. 20, 2017, 4:31 p.m.) > > > Review request for Ambari, Eugene Chekanskiy, Jonathan Hurley, Laszlo Puskas, > Oliver Szabo, and Sebastian Toader. > > > Bugs: AMBARI-19632 > https://issues.apache.org/jira/browse/AMBARI-19632 > > > Repository: ambari > > > Description > --- > > Ldap sync fails when there are special characters in distinguished names. > > For example if there was a user with the distinguished name of > `OU=test/test,OU=users,DC=EXAMPLE,DC=COM` and that user was a member of a > synced group, then the lookup of the user using the membership attribute in > the group would fail due to the special character. > > The error would look something like > ``` > REASON: Caught exception running LDAP sync. Uncategorized exception occured > during LDAP processing; nested exception is javax.naming.NamingException: > [LDAP: error code 1 - 20D6: SvcErr: DSID-031007DB, problem 5012 > (DIR_ERROR), data 0 > ]; remaining name 'OU=test/test,OU=users,DC=EXAMPLE,DC=COM' > ``` > > #Solution > Update the library version for Spring LDAP > - `org.springframework.security/spring-security-ldap` to `4.0.4.RELEASE` > - `org.springframework.ldap/spring-ldap-core` to `2.0.4.RELEASE` > > Then use `LdapUtils.newLdapName` to convert a String representing a DN into a > `javax.naming.ldap.LdapName` and use that object in the search facility > executed in > `org.apache.ambari.server.security.ldap.AmbariLdapDataPopulator#getFilteredLdapUsers(java.lang.String, > org.springframework.ldap.filter.Filter)`. > > > Diffs > - > > ambari-project/pom.xml 16ea2af > > ambari-server/src/main/java/org/apache/ambari/server/security/authorization/AmbariLdapBindAuthenticator.java > b4ef889 > > ambari-server/src/main/java/org/apache/ambari/server/security/authorization/AmbariLdapUtils.java > a64ab3d > > ambari-server/src/main/java/org/apache/ambari/server/security/ldap/AmbariLdapDataPopulator.java > 2dccf11 > > ambari-server/src/test/java/org/apache/ambari/server/security/AmbariLdapUtilsTest.java > 5629913 > > ambari-server/src/test/java/org/apache/ambari/server/security/authorization/AmbariLdapBindAuthenticatorTest.java > fcaae37 > > ambari-server/src/test/java/org/apache/ambari/server/security/ldap/AmbariLdapDataPopulatorTest.java > 6143cf8 > > Diff: https://reviews.apache.org/r/55764/diff/ > > > Testing > --- > > manually test syncing LDAP and authentication. > > > Thanks, > > Robert Levas > >
Re: Review Request 55777: Update quicklink.json files in stack definitions with authenticated/sso features
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55777/#review162460
---
Ship it!
Ship It!
- Laszlo Puskas
On Jan. 20, 2017, 6:30 p.m., Balázs Bence Sári wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55777/
> ---
>
> (Updated Jan. 20, 2017, 6:30 p.m.)
>
>
> Review request for Ambari, Attila Doroszlai, Attila Magyar, Laszlo Puskas,
> Oliver Szabo, Sandor Magyari, and Sebastian Toader.
>
>
> Bugs: AMBARI-19655
> https://issues.apache.org/jira/browse/AMBARI-19655
>
>
> Repository: ambari
>
>
> Description
> ---
>
> Existing quicklink definitions need to be updated with authenticatedness and
> sso-ness information wherever applicable.
>
> E.g:
> "links": [
> {
> "name": "atlas_dashboard",
> "label": "Atlas Dashboard",
> "url": "%@://%@:%@/#!/search?user.name=%@",
> ...
> "attributes": [
> "authenticated",
> "sso"
> ],
> ...
> }
> ]
>
>
> Diffs
> -
>
>
> ambari-server/src/main/resources/common-services/ATLAS/0.7.0.2.5/quicklinks/quicklinks.json
> e86b665
>
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/quicklinks/quicklinks.json
> 152ff57
>
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/quicklinks/quicklinks.json
> b6f6a09
>
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/quicklinks/quicklinks.json
> ea8c378
>
> ambari-server/src/main/resources/common-services/RANGER/0.5.0/quicklinks/quicklinks.json
> 3689ba9
>
> ambari-server/src/main/resources/stacks/HDP/2.3/services/ATLAS/quicklinks/quicklinks.json
> ccc52bc
>
> Diff: https://reviews.apache.org/r/55777/diff/
>
>
> Testing
> ---
>
> Manually tested the change. Installed a cluster.
> - Verified that attributes a returned by the API.
> - Verified that attributes are respected when a quick links profile to show
> authenticated/sso links only is in effect
>
>
> Thanks,
>
> Balázs Bence Sári
>
>
Re: Review Request 50753: Cleared cached resources from ambari-server on host removal
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50753/ --- (Updated Aug. 5, 2016, 2:39 p.m.) Review request for Ambari, Jonathan Hurley, Robert Nettleton, Sandor Magyari, and Sebastian Toader. Bugs: AMBARI-18005 https://issues.apache.org/jira/browse/AMBARI-18005 Repository: ambari Description --- When a host is removed from the cluster and later from ambari there's a chance the agent registers back to the ambari server before the agent is stopped. Stopping the machine running the agent without the host being deleted again leads to an inconsistent state in the ambari-server due to cached state. Resolution: The cached resources get cleared on host delete event. Diffs - ambari-server/src/main/java/org/apache/ambari/server/state/host/HostImpl.java a757010 ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java 0190478 Diff: https://reviews.apache.org/r/50753/diff/ Testing (updated) --- Suceesfully tested. (Manually) Unit tests OK Thanks, Laszlo Puskas
Re: Review Request 57147: Added support for processing custom command script timeout
> On Feb. 28, 2017, 6:06 p.m., Alejandro Fernandez wrote: > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml, > > line 64 > > <https://reviews.apache.org/r/57147/diff/1/?file=1651734#file1651734line64> > > > > I don't think Ambari should run a command that takes 2 hours! > > Otherwise, the user will not be able to run other commands on that host. > > > > Instead, we need HDFS to return right away and indicate if a rebalance > > is already happening. > > Alejandro Fernandez wrote: > Please ask for input from the HDFS team on how to best do this. We asked them already, they confirmed, that apart of the command output it's not possible to track/retrieve the status of the balancer. So once we loose the connection with the command output we can't get information about the balancer. - Laszlo --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/57147/#review167110 ----------- On Feb. 28, 2017, 6:01 p.m., Laszlo Puskas wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/57147/ > --- > > (Updated Feb. 28, 2017, 6:01 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Robert Levas, Sandor Magyari, > and Sebastian Toader. > > > Bugs: AMBARI-20175 > https://issues.apache.org/jira/browse/AMBARI-20175 > > > Repository: ambari > > > Description > --- > > HDFS rebalance operation can last for a long time (hours/days) thus when > triggered from the UI the command may be timed out by the Ambari server. > This behavior may confuse users, making them to trigger the rebalancer again > which will fail with "another balancer is running" error. > > The patch provides support for setting a reasonably long timeout for the > rebalance custo action so that Ambari server doesnt time out the command. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java > 867ebff > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml > fd7f2f6 > > Diff: https://reviews.apache.org/r/57147/diff/ > > > Testing > --- > > Unit tests OK. > Manually tested on local dev-env. > > > Thanks, > > Laszlo Puskas > >
