Re: Review Request 61527: AMBARI-21649 : Update Atlas log4j to reflect latest configurations

[Mugdha Varadkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61527/#review182559
---


Ship it!




Ship It!

- Mugdha Varadkar


On Aug. 10, 2017, 5:42 a.m., Vishal Suvagia wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61527/
> ---
> 
> (Updated Aug. 10, 2017, 5:42 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jayush Luniya, 
> Madhan Neethiraj, Mugdha Varadkar, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-21649
> https://issues.apache.org/jira/browse/AMBARI-21649
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Need to update Atlas log4j configurations to reflect latest configurations as 
> per update in Atlas.
> 
> PS: Attached patch is for trunk, seperate patch for branch-2.6 is attached on 
> Apache JIRA.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.7.0.3.0/configuration/atlas-log4j.xml
>  bafd47d 
>   ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/config-upgrade.xml 
> ec6ee55 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml
>  e9490b2 
>   ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml 
> 840b17d 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/ATLAS/configuration/atlas-log4j.xml
>  9e3abcf 
>   ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 
> c2c1532 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml
>  86b3956 
>   ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml 
> b376fa7 
> 
> 
> Diff: https://reviews.apache.org/r/61527/diff/2/
> 
> 
> Testing
> ---
> 
> Verified with fresh install and upgrade scenarios on CentOS 6.
> 
> 
> Thanks,
> 
> Vishal Suvagia
> 
>

Re: Review Request 61525: AMBARI-21690 Updating get_stack_version pattern match

[Mugdha Varadkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61525/
---

(Updated Aug. 10, 2017, 6:29 a.m.)


Review request for Ambari, Alejandro Fernandez, Gautam Borad, and Jonathan 
Hurley.


Bugs: AMBARI-21690
https://issues.apache.org/jira/browse/AMBARI-21690


Repository: ambari


Description
---

Currently get_stack_version only accepts stack version with -buildnumber at the 
end. 
Need to change stack_version pattern match for stack_version without 
-buildnumber.

For cross stack upgrade from IOP to HDP, if user restart services after Ambari 
upgrade ie before upgrading to HDP stack 2.6, the get_stack_version returns 
None. This affects Ranger-related functionality which depends on stack_version 
for generating file path.

The problem occured for cluster with IOP - 4.2.0.0 with latest Ambari, which 
doesn't have -buildnumber at the end, so 4.2.0.0 is not matched and 
stack_version=None.


Diffs (updated)
-

  
ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py
 7274a59 


Diff: https://reviews.apache.org/r/61525/diff/2/

Changes: https://reviews.apache.org/r/61525/diff/1-2/


Testing
---

Tested:
1)Services restart after ambari upgrade with IOP 4.2.0.0
2)Fresh installs with HDP.
3)EU from IOP to HDP-2.6.x and HDP-2.6.x to HDP-2.6.y

Unit test report:
Ran 274 tests in 6.820s

OK
--
Total run:1162
Total errors:0
Total failures:0
OK


Thanks,

Mugdha Varadkar

Re: Review Request 61527: AMBARI-21649 : Update Atlas log4j to reflect latest configurations

[Vishal Suvagia via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61527/
---

(Updated Aug. 10, 2017, 5:42 a.m.)


Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jayush Luniya, 
Madhan Neethiraj, Mugdha Varadkar, and Sumit Mohanty.


Changes
---

Adding atlas-log4j changes to 3.0 version.


Bugs: AMBARI-21649
https://issues.apache.org/jira/browse/AMBARI-21649


Repository: ambari


Description
---

Need to update Atlas log4j configurations to reflect latest configurations as 
per update in Atlas.

PS: Attached patch is for trunk, seperate patch for branch-2.6 is attached on 
Apache JIRA.


Diffs (updated)
-

  
ambari-server/src/main/resources/common-services/ATLAS/0.7.0.3.0/configuration/atlas-log4j.xml
 bafd47d 
  ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/config-upgrade.xml 
ec6ee55 
  
ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml
 e9490b2 
  ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml 
840b17d 
  
ambari-server/src/main/resources/stacks/HDP/2.6/services/ATLAS/configuration/atlas-log4j.xml
 9e3abcf 
  ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 
c2c1532 
  
ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml
 86b3956 
  ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml 
b376fa7 


Diff: https://reviews.apache.org/r/61527/diff/2/

Changes: https://reviews.apache.org/r/61527/diff/1-2/


Testing
---

Verified with fresh install and upgrade scenarios on CentOS 6.


Thanks,

Vishal Suvagia

Re: Review Request 61539: Can't register multiple PATCH versions

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61539/#review182554
---


Fix it, then Ship it!





ambari-server/src/main/java/org/apache/ambari/server/orm/dao/RepositoryVersionDAO.java
Lines 244-247 (patched)


DaoUtils catches exceptions - you shouldn't need the try/catch.


- Jonathan Hurley


On Aug. 9, 2017, 4:38 p.m., Nate Cole wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61539/
> ---
> 
> (Updated Aug. 9, 2017, 4:38 p.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko and Jonathan Hurley.
> 
> 
> Bugs: AMBARI-21693
> https://issues.apache.org/jira/browse/AMBARI-21693
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix issues when installing PATCH/MAINT VDF and matching repos.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/VersionDefinitionResourceProvider.java
>  77c09d403c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/RepositoryVersionDAO.java
>  95e608a86b 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/RepositoryVersionEntity.java
>  df15d27df3 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/VersionDefinitionResourceProviderTest.java
>  a297d945ce 
> 
> 
> Diff: https://reviews.apache.org/r/61539/diff/1/
> 
> 
> Testing
> ---
> 
> Manual.  Automated:
> 
> Tests run: 4409, Failures: 0, Errors: 0, Skipped: 30
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 23:26.422s
> [INFO] Finished at: Wed Aug 09 16:38:00 EDT 2017
> [INFO] Final Memory: 70M/2037M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Nate Cole
> 
>

Re: Review Request 61525: AMBARI-21690 Updating get_stack_version pattern match

[Jonathan Hurley]

> On Aug. 9, 2017, 2:29 p.m., Alejandro Fernandez wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py
> > Line 88 (original), 88 (patched)
> > 
> >
> > Just do this instead,
> > 
> > [0-9]+.[0-9]+.[0-9]+.[0-9]+(-[0-9]+)?

pythex.org likes this regex too - +1 for this change.


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61525/#review182507
---


On Aug. 9, 2017, 9:07 a.m., Mugdha Varadkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61525/
> ---
> 
> (Updated Aug. 9, 2017, 9:07 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Gautam Borad, and Jonathan 
> Hurley.
> 
> 
> Bugs: AMBARI-21690
> https://issues.apache.org/jira/browse/AMBARI-21690
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently get_stack_version only accepts stack version with -buildnumber at 
> the end. 
> Need to change stack_version pattern match for stack_version without 
> -buildnumber.
> 
> For cross stack upgrade from IOP to HDP, if user restart services after 
> Ambari upgrade ie before upgrading to HDP stack 2.6, the get_stack_version 
> returns None. This affects Ranger-related functionality which depends on 
> stack_version for generating file path.
> 
> The problem occured for cluster with IOP - 4.2.0.0 with latest Ambari, which 
> doesn't have -buildnumber at the end, so 4.2.0.0 is not matched and 
> stack_version=None.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py
>  7274a59 
> 
> 
> Diff: https://reviews.apache.org/r/61525/diff/1/
> 
> 
> Testing
> ---
> 
> Tested:
> 1)Services restart after ambari upgrade with IOP 4.2.0.0
> 2)Fresh installs with HDP.
> 3)EU from IOP to HDP-2.6.x and HDP-2.6.x to HDP-2.6.y
> 
> Unit test report:
> Ran 274 tests in 6.820s
> 
> OK
> --
> Total run:1162
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Mugdha Varadkar
> 
>

Re: Review Request 61542: Problem in starting accumulo in upgraded cluster (IOP-HDP)

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61542/#review182550
---


Ship it!




Ship It!

- Jonathan Hurley


On Aug. 9, 2017, 7:26 p.m., Sid Wagle wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61542/
> ---
> 
> (Updated Aug. 9, 2017, 7:26 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmytro Sen, Jonathan Hurley, 
> and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-21695
> https://issues.apache.org/jira/browse/AMBARI-21695
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> org.apache.ambari.server.metadata.CachedRoleCommandOrderProvider#getRoleCommandOrder(org.apache.ambari.server.state.Cluster)
> 
> This method caches the RCO for the stack which is never cleared. Since the BI 
> stack does not have Accumulo service the RCO is not honored resulting in 
> ACCUMULO service not starting after the express upgrade when ACCUMULO is 
> added.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/metadata/CachedRoleCommandOrderProvider.java
>  49abc66 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/FinalizeUpgradeAction.java
>  d531460 
>   ambari-server/src/main/resources/scripts/configs.py 40a8324 
> 
> 
> Diff: https://reviews.apache.org/r/61542/diff/2/
> 
> 
> Testing
> ---
> 
> UpgradeActionTest unit tests are green.
> 
> 
> File Attachments
> 
> 
> AMBARI-21695-1.patch
>   
> https://reviews.apache.org/media/uploaded/files/2017/08/09/eadc72ba-a0ce-4b99-9f57-f5230a7d165b__AMBARI-21695-1.patch
> AMBARI-21695-2.patch
>   
> https://reviews.apache.org/media/uploaded/files/2017/08/09/addb12dc-661d-480c-9526-c5c13a7a5566__AMBARI-21695-2.patch
> 
> 
> Thanks,
> 
> Sid Wagle
> 
>

Re: Review Request 61542: Problem in starting accumulo in upgraded cluster (IOP-HDP)

[Sid Wagle]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61542/
---

(Updated Aug. 9, 2017, 11:26 p.m.)


Review request for Ambari, Alejandro Fernandez, Dmytro Sen, Jonathan Hurley, 
and Sumit Mohanty.


Changes
---

Incorporated comments from @jhurley.


Bugs: AMBARI-21695
https://issues.apache.org/jira/browse/AMBARI-21695


Repository: ambari


Description
---

org.apache.ambari.server.metadata.CachedRoleCommandOrderProvider#getRoleCommandOrder(org.apache.ambari.server.state.Cluster)

This method caches the RCO for the stack which is never cleared. Since the BI 
stack does not have Accumulo service the RCO is not honored resulting in 
ACCUMULO service not starting after the express upgrade when ACCUMULO is added.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/metadata/CachedRoleCommandOrderProvider.java
 49abc66 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/FinalizeUpgradeAction.java
 d531460 
  ambari-server/src/main/resources/scripts/configs.py 40a8324 


Diff: https://reviews.apache.org/r/61542/diff/2/


Testing
---

UpgradeActionTest unit tests are green.


File Attachments (updated)


AMBARI-21695-1.patch
  
https://reviews.apache.org/media/uploaded/files/2017/08/09/eadc72ba-a0ce-4b99-9f57-f5230a7d165b__AMBARI-21695-1.patch
AMBARI-21695-2.patch
  
https://reviews.apache.org/media/uploaded/files/2017/08/09/addb12dc-661d-480c-9526-c5c13a7a5566__AMBARI-21695-2.patch


Thanks,

Sid Wagle

Re: Review Request 61542: Problem in starting accumulo in upgraded cluster (IOP-HDP)

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61542/#review182548
---


Ship it!




We already have a class handling this event - StackUpgradeFinishListener - you 
can clear the RCO stuff in there. This prevents you from needing to add the 
event publisher to the cache. If you want to keep the code the way it is, you'd 
need to do this:

@Inject
VersionEventPublisher versionEventPublisher;

...
eventPublisher.register(this);

- Jonathan Hurley


On Aug. 9, 2017, 7:05 p.m., Sid Wagle wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61542/
> ---
> 
> (Updated Aug. 9, 2017, 7:05 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmytro Sen, Jonathan Hurley, 
> and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-21695
> https://issues.apache.org/jira/browse/AMBARI-21695
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> org.apache.ambari.server.metadata.CachedRoleCommandOrderProvider#getRoleCommandOrder(org.apache.ambari.server.state.Cluster)
> 
> This method caches the RCO for the stack which is never cleared. Since the BI 
> stack does not have Accumulo service the RCO is not honored resulting in 
> ACCUMULO service not starting after the express upgrade when ACCUMULO is 
> added.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/metadata/CachedRoleCommandOrderProvider.java
>  49abc66 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/FinalizeUpgradeAction.java
>  d531460 
>   ambari-server/src/main/resources/scripts/configs.py 40a8324 
> 
> 
> Diff: https://reviews.apache.org/r/61542/diff/2/
> 
> 
> Testing
> ---
> 
> UpgradeActionTest unit tests are green.
> 
> 
> File Attachments
> 
> 
> AMBARI-21695-1.patch
>   
> https://reviews.apache.org/media/uploaded/files/2017/08/09/eadc72ba-a0ce-4b99-9f57-f5230a7d165b__AMBARI-21695-1.patch
> 
> 
> Thanks,
> 
> Sid Wagle
> 
>

Re: Review Request 61542: Problem in starting accumulo in upgraded cluster (IOP-HDP)

[Sid Wagle]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61542/
---

(Updated Aug. 9, 2017, 11:05 p.m.)


Review request for Ambari, Alejandro Fernandez, Dmytro Sen, Jonathan Hurley, 
and Sumit Mohanty.


Changes
---

Moved to clearing cache on StackUpgradeFinishEvent


Bugs: AMBARI-21695
https://issues.apache.org/jira/browse/AMBARI-21695


Repository: ambari


Description
---

org.apache.ambari.server.metadata.CachedRoleCommandOrderProvider#getRoleCommandOrder(org.apache.ambari.server.state.Cluster)

This method caches the RCO for the stack which is never cleared. Since the BI 
stack does not have Accumulo service the RCO is not honored resulting in 
ACCUMULO service not starting after the express upgrade when ACCUMULO is added.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/metadata/CachedRoleCommandOrderProvider.java
 49abc66 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/FinalizeUpgradeAction.java
 d531460 
  ambari-server/src/main/resources/scripts/configs.py 40a8324 


Diff: https://reviews.apache.org/r/61542/diff/2/


Testing
---

UpgradeActionTest unit tests are green.


File Attachments (updated)


AMBARI-21695-1.patch
  
https://reviews.apache.org/media/uploaded/files/2017/08/09/eadc72ba-a0ce-4b99-9f57-f5230a7d165b__AMBARI-21695-1.patch


Thanks,

Sid Wagle

Re: Review Request 61542: Problem in starting accumulo in upgraded cluster (IOP-HDP)

[Sid Wagle]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61542/
---

(Updated Aug. 9, 2017, 10:15 p.m.)


Review request for Ambari, Alejandro Fernandez, Dmytro Sen, Jonathan Hurley, 
and Sumit Mohanty.


Changes
---

Added same thing on downgrade.


Bugs: AMBARI-21695
https://issues.apache.org/jira/browse/AMBARI-21695


Repository: ambari


Description
---

org.apache.ambari.server.metadata.CachedRoleCommandOrderProvider#getRoleCommandOrder(org.apache.ambari.server.state.Cluster)

This method caches the RCO for the stack which is never cleared. Since the BI 
stack does not have Accumulo service the RCO is not honored resulting in 
ACCUMULO service not starting after the express upgrade when ACCUMULO is added.


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/metadata/CachedRoleCommandOrderProvider.java
 49abc66 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/FinalizeUpgradeAction.java
 d531460 
  ambari-server/src/main/resources/scripts/configs.py 40a8324 


Diff: https://reviews.apache.org/r/61542/diff/2/

Changes: https://reviews.apache.org/r/61542/diff/1-2/


Testing
---

UpgradeActionTest unit tests are green.


Thanks,

Sid Wagle

Re: Review Request 61542: Problem in starting accumulo in upgraded cluster (IOP-HDP)

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61542/#review182532
---


Ship it!




Ship It!

- Alejandro Fernandez


On Aug. 9, 2017, 9:41 p.m., Sid Wagle wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61542/
> ---
> 
> (Updated Aug. 9, 2017, 9:41 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmytro Sen, Jonathan Hurley, 
> and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-21695
> https://issues.apache.org/jira/browse/AMBARI-21695
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> org.apache.ambari.server.metadata.CachedRoleCommandOrderProvider#getRoleCommandOrder(org.apache.ambari.server.state.Cluster)
> 
> This method caches the RCO for the stack which is never cleared. Since the BI 
> stack does not have Accumulo service the RCO is not honored resulting in 
> ACCUMULO service not starting after the express upgrade when ACCUMULO is 
> added.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/metadata/CachedRoleCommandOrderProvider.java
>  49abc66 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/FinalizeUpgradeAction.java
>  d531460 
>   ambari-server/src/main/resources/scripts/configs.py 40a8324 
> 
> 
> Diff: https://reviews.apache.org/r/61542/diff/1/
> 
> 
> Testing
> ---
> 
> UpgradeActionTest unit tests are green.
> 
> 
> Thanks,
> 
> Sid Wagle
> 
>

Review Request 61542: Problem in starting accumulo in upgraded cluster (IOP-HDP)

[Sid Wagle]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61542/
---

Review request for Ambari, Alejandro Fernandez, Dmytro Sen, Jonathan Hurley, 
and Sumit Mohanty.


Bugs: AMBARI-21695
https://issues.apache.org/jira/browse/AMBARI-21695


Repository: ambari


Description
---

org.apache.ambari.server.metadata.CachedRoleCommandOrderProvider#getRoleCommandOrder(org.apache.ambari.server.state.Cluster)

This method caches the RCO for the stack which is never cleared. Since the BI 
stack does not have Accumulo service the RCO is not honored resulting in 
ACCUMULO service not starting after the express upgrade when ACCUMULO is added.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/metadata/CachedRoleCommandOrderProvider.java
 49abc66 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/FinalizeUpgradeAction.java
 d531460 
  ambari-server/src/main/resources/scripts/configs.py 40a8324 


Diff: https://reviews.apache.org/r/61542/diff/1/


Testing
---

UpgradeActionTest unit tests are green.


Thanks,

Sid Wagle

Review Request 61539: Can't register multiple PATCH versions

[Nate Cole]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61539/
---

Review request for Ambari, Dmitro Lisnichenko and Jonathan Hurley.


Bugs: AMBARI-21693
https://issues.apache.org/jira/browse/AMBARI-21693


Repository: ambari


Description
---

Fix issues when installing PATHC/MAINT VDF and matching repos.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/VersionDefinitionResourceProvider.java
 77c09d403c 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/RepositoryVersionDAO.java
 95e608a86b 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/RepositoryVersionEntity.java
 df15d27df3 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/VersionDefinitionResourceProviderTest.java
 a297d945ce 


Diff: https://reviews.apache.org/r/61539/diff/1/


Testing
---

Manual.  Automated pending.


Thanks,

Nate Cole

Re: Review Request 61525: AMBARI-21690 Updating get_stack_version pattern match

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61525/#review182507
---




ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py
Line 88 (original), 88 (patched)


Just do this instead,

[0-9]+.[0-9]+.[0-9]+.[0-9]+(-[0-9]+)?


- Alejandro Fernandez


On Aug. 9, 2017, 1:07 p.m., Mugdha Varadkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61525/
> ---
> 
> (Updated Aug. 9, 2017, 1:07 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Gautam Borad, and Jonathan 
> Hurley.
> 
> 
> Bugs: AMBARI-21690
> https://issues.apache.org/jira/browse/AMBARI-21690
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently get_stack_version only accepts stack version with -buildnumber at 
> the end. 
> Need to change stack_version pattern match for stack_version without 
> -buildnumber.
> 
> For cross stack upgrade from IOP to HDP, if user restart services after 
> Ambari upgrade ie before upgrading to HDP stack 2.6, the get_stack_version 
> returns None. This affects Ranger-related functionality which depends on 
> stack_version for generating file path.
> 
> The problem occured for cluster with IOP - 4.2.0.0 with latest Ambari, which 
> doesn't have -buildnumber at the end, so 4.2.0.0 is not matched and 
> stack_version=None.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py
>  7274a59 
> 
> 
> Diff: https://reviews.apache.org/r/61525/diff/1/
> 
> 
> Testing
> ---
> 
> Tested:
> 1)Services restart after ambari upgrade with IOP 4.2.0.0
> 2)Fresh installs with HDP.
> 3)EU from IOP to HDP-2.6.x and HDP-2.6.x to HDP-2.6.y
> 
> Unit test report:
> Ran 274 tests in 6.820s
> 
> OK
> --
> Total run:1162
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Mugdha Varadkar
> 
>

Re: Review Request 61521: Fix raceconditions causing ambari-agent to hang/fail

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61521/#review182506
---


Ship it!




Ship It!

- Alejandro Fernandez


On Aug. 9, 2017, 10:35 a.m., Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61521/
> ---
> 
> (Updated Aug. 9, 2017, 10:35 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Myroslav Papirkovskyy, and 
> Sid Wagle.
> 
> 
> Bugs: AMBARI-21688
> https://issues.apache.org/jira/browse/AMBARI-21688
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix two raceconditions:  
> First causes agent to hang forever during initial connection:
> 
> 
> 
> INFO 2017-08-08 16:55:06,780 transport.py:329 - Starting receiver 
> loopINFO 2017-08-08 16:55:06,781 transport.py:271 - Sending frame: 'STOMP', 
> headers={'host': 0, 'accept-version': '1.2'}
> 
> 
> Second one causes heartbeating fail with:
> 
> 
> 
> ERROR 2017-08-08 10:12:08,328 HeartbeatThread.py:167 - Error in 
> responseId sequence - restarting
> INFO 2017-08-08 10:12:08,328 ExitHelper.py:56 - Performing cleanup before 
> exiting...
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/main/python/ambari_agent/CommandStatusDict.py d6cbdcc 
>   ambari-agent/src/main/python/ambari_agent/ComponentStatusExecutor.py 
> be3eb5b 
>   ambari-agent/src/main/python/ambari_agent/HeartbeatThread.py e2fe4af 
>   ambari-agent/src/main/python/ambari_agent/HostStatusReporter.py c45b64a 
>   ambari-agent/src/main/python/ambari_agent/InitializerModule.py 9b031f7 
>   
> ambari-agent/src/main/python/ambari_agent/listeners/ServerResponsesListener.py
>  6d23c37 
>   ambari-agent/src/main/python/ambari_agent/security.py 9509619 
>   ambari-common/src/main/python/ambari_stomp/adapter/websocket.py 220f399 
>   ambari-common/src/main/python/ambari_ws4py/websocket.py b5c1fd3 
> 
> 
> Diff: https://reviews.apache.org/r/61521/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>

Re: Review Request 61503: Service and Patch Upgrade Catalog Changes for 2.6.0

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61503/#review182486
---




ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
Lines 213 (patched)


I'm a little worried about this one. I thought the way you'd approach it 
was to make a query to get the ID of the one CURRENT repo and then use that ID 
directly via simple UPDATE/SET commands. 

The problem here seems to be twofold:
- There's a lot of added complexity per database for this call
- It doesn't really check to make sure you get the right ID

I'd think it would be simpler to do something like:

- Query for the ID of the repo to use for all set commands. If the CURRENT 
repo can't be found, throw an exception and fail the upgrade
- One you have the ID, now go and make the DDL structure withou constraints
- Set the ID on the new structures
- Add the constraints
- Drop the old stuff


- Jonathan Hurley


On Aug. 8, 2017, 1:47 p.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61503/
> ---
> 
> (Updated Aug. 8, 2017, 1:47 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Jonathan Hurley, and Nate Cole.
> 
> 
> Bugs: AMBARI-21169
> https://issues.apache.org/jira/browse/AMBARI-21169
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Implement the following upgrade catalog changes related to service/patch 
> upgrades:
> 
> h5. {{servicecomponentdesiredstate}}
> - Remove:  desired_stack_id BIGINT NOT NULL
> - Remove: desired_version VARCHAR(255) NOT NULL DEFAULT 'UNKNOWN'
> - Remove: FK on desired_stack_id (FK_scds_desired_stack_id)
> - Add: desired_repo_version_id BIGINT NOT NULL
> - Add: FK to repo_version_id (FK_scds_desired_repo_id)
> 
> h5. {{hostcomponentdesiredstate}}
> - Remove: desired_stack_id BIGINT NOT NULL
> - Remove: FK on desired_stack_id (FK_hcds_desired_stack_id)
> 
> h5. {{hostcomponentstate}}
> - Remove: current_stack_id BIGINT NOT NULL
> - Remove: FK on desired_stack_id (FK_hcs_current_stack_id)
> 
> h5. {{servicedesiredstate}}
> - Remove: desired_stack_id BIGINT NOT NULL
> - Add: desired_repo_version_id BIGINT NOT NULL
> - Add: FK  to repo_version_id (FK_repo_version_id)
> 
> h5. {{host_version}}
> - Change the {{UNIQUE}} constraint to allow for multiple {{CURRENT}} 
> repositories per host. Restriction should also include the 
> {{repo_version_id}} for uniqueness now.
> 
> h5. {{cluster_version}}
> - This table was removed.
> 
> h5. {{servicecomponent_version}}
> - Create this table and populate with data
> 
> h5. {{upgrade}}
> - Add orchestration VARCHAR(255) NOT NULL DEFAULT 'STANDARD'
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/java/org/apache/ambari/server/orm/DBAccessor.java 
> 03136984ba 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/DBAccessorImpl.java 
> 38d60e9a4a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/helpers/dbms/DbmsHelper.java
>  00b02f1033 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/helpers/dbms/GenericDbmsHelper.java
>  11cbcf922b 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/helpers/dbms/H2Helper.java
>  100c8659ef 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/helpers/dbms/MySqlHelper.java
>  ad26520e4a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/helpers/dbms/OracleHelper.java
>  73356d16e3 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/helpers/dbms/PostgresHelper.java
>  46d7315d91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/SchemaUpgradeHelper.java
>  35e5b1030f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/61503/diff/1/
> 
> 
> Testing
> ---
> 
> running tests on live cluster
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>

Re: Review Request 61500: Service Checks Will Run Multiple Times In Patch/Service Upgrades

[Nate Cole]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61500/
---

(Updated Aug. 9, 2017, 10:38 a.m.)


Review request for Ambari, Dmitro Lisnichenko and Jonathan Hurley.


Changes
---

Update for comments.  Post-loop the groups looking for ones to remove if a 
service check shouldn't run.


Bugs: AMBARI-21679
https://issues.apache.org/jira/browse/AMBARI-21679


Repository: ambari


Description
---

There are two issues with Service Checks and patches:

1. A service check grouping can be orchestrated one right after the other if 
the grouping in between them has no processing.
2. A service check grouping can be orchestrated before any components have been 
scheduled.


Diffs (updated)
-

  ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeHelper.java 
07012961af 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java
 3deb7c866e 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/Grouping.java
 fed5b77942 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ServiceCheckGrouping.java
 6a085f331f 
  
ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java
 24a3fa2ce2 
  
ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/upgrade_test_checks.xml
 f82b0258de 


Diff: https://reviews.apache.org/r/61500/diff/2/

Changes: https://reviews.apache.org/r/61500/diff/1-2/


Testing
---

Manual.  Automated:

Tests run: 4481, Failures: 0, Errors: 0, Skipped: 31

[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 21:58.183s
[INFO] Finished at: Tue Aug 08 11:36:30 EDT 2017
[INFO] Final Memory: 57M/637M
[INFO] 


Thanks,

Nate Cole

Review Request 61527: AMBARI-21649 : Update Atlas log4j to reflect latest configurations

[Vishal Suvagia via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61527/
---

Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jayush Luniya, 
Madhan Neethiraj, Mugdha Varadkar, and Sumit Mohanty.


Bugs: AMBARI-21649
https://issues.apache.org/jira/browse/AMBARI-21649


Repository: ambari


Description
---

Need to update Atlas log4j configurations to reflect latest configurations as 
per update in Atlas.

PS: Attached patch is for trunk, seperate patch for branch-2.6 is attached on 
Apache JIRA.


Diffs
-

  ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/config-upgrade.xml 
ec6ee55 
  
ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml
 e9490b2 
  ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml 
840b17d 
  
ambari-server/src/main/resources/stacks/HDP/2.6/services/ATLAS/configuration/atlas-log4j.xml
 9e3abcf 
  ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 
c2c1532 
  
ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml
 86b3956 
  ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml 
b376fa7 


Diff: https://reviews.apache.org/r/61527/diff/1/


Testing
---

Verified with fresh install and upgrade scenarios on CentOS 6.


Thanks,

Vishal Suvagia

Re: Review Request 61525: AMBARI-21690 Updating get_stack_version pattern match

[Mugdha Varadkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61525/
---

(Updated Aug. 9, 2017, 1:07 p.m.)


Review request for Ambari, Alejandro Fernandez, Gautam Borad, and Jonathan 
Hurley.


Bugs: AMBARI-21690
https://issues.apache.org/jira/browse/AMBARI-21690


Repository: ambari


Description
---

Currently get_stack_version only accepts stack version with -buildnumber at the 
end. 
Need to change stack_version pattern match for stack_version without 
-buildnumber.

For cross stack upgrade from IOP to HDP, if user restart services after Ambari 
upgrade ie before upgrading to HDP stack 2.6, the get_stack_version returns 
None. This affects Ranger-related functionality which depends on stack_version 
for generating file path.

The problem occured for cluster with IOP - 4.2.0.0 with latest Ambari, which 
doesn't have -buildnumber at the end, so 4.2.0.0 is not matched and 
stack_version=None.


Diffs
-

  
ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py
 7274a59 


Diff: https://reviews.apache.org/r/61525/diff/1/


Testing
---

Tested:
1)Services restart after ambari upgrade with IOP 4.2.0.0
2)Fresh installs with HDP.
3)EU from IOP to HDP-2.6.x and HDP-2.6.x to HDP-2.6.y

Unit test report:
Ran 274 tests in 6.820s

OK
--
Total run:1162
Total errors:0
Total failures:0
OK


Thanks,

Mugdha Varadkar

Review Request 61525: AMBARI-21690 Updating get_stack_version pattern match

[Mugdha Varadkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61525/
---

Review request for Ambari, Alejandro Fernandez, Gautam Borad, and Jonathan 
Hurley.


Repository: ambari


Description
---

Currently get_stack_version only accepts stack version with -buildnumber at the 
end. 
Need to change stack_version pattern match for stack_version without 
-buildnumber.

For cross stack upgrade from IOP to HDP, if user restart services after Ambari 
upgrade ie before upgrading to HDP stack 2.6, the get_stack_version returns 
None. This affects Ranger-related functionality which depends on stack_version 
for generating file path.

The problem occured for cluster with IOP - 4.2.0.0 with latest Ambari, which 
doesn't have -buildnumber at the end, so 4.2.0.0 is not matched and 
stack_version=None.


Diffs
-

  
ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py
 7274a59 


Diff: https://reviews.apache.org/r/61525/diff/1/


Testing
---

Tested:
1)Services restart after ambari upgrade with IOP 4.2.0.0
2)Fresh installs with HDP.
3)EU from IOP to HDP-2.6.x and HDP-2.6.x to HDP-2.6.y

Unit test report:
Ran 274 tests in 6.820s

OK
--
Total run:1162
Total errors:0
Total failures:0
OK


Thanks,

Mugdha Varadkar

Review Request 61521: Fix raceconditions causing ambari-agent to hang/fail

[Andrew Onischuk]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61521/
---

Review request for Ambari, Alejandro Fernandez, Myroslav Papirkovskyy, and Sid 
Wagle.


Bugs: AMBARI-21688
https://issues.apache.org/jira/browse/AMBARI-21688


Repository: ambari


Description
---

Fix two raceconditions:  
First causes agent to hang forever during initial connection:



INFO 2017-08-08 16:55:06,780 transport.py:329 - Starting receiver loopINFO 
2017-08-08 16:55:06,781 transport.py:271 - Sending frame: 'STOMP', 
headers={'host': 0, 'accept-version': '1.2'}


Second one causes heartbeating fail with:



ERROR 2017-08-08 10:12:08,328 HeartbeatThread.py:167 - Error in responseId 
sequence - restarting
INFO 2017-08-08 10:12:08,328 ExitHelper.py:56 - Performing cleanup before 
exiting...


Diffs
-

  ambari-agent/src/main/python/ambari_agent/CommandStatusDict.py d6cbdcc 
  ambari-agent/src/main/python/ambari_agent/ComponentStatusExecutor.py be3eb5b 
  ambari-agent/src/main/python/ambari_agent/HeartbeatThread.py e2fe4af 
  ambari-agent/src/main/python/ambari_agent/HostStatusReporter.py c45b64a 
  ambari-agent/src/main/python/ambari_agent/InitializerModule.py 9b031f7 
  
ambari-agent/src/main/python/ambari_agent/listeners/ServerResponsesListener.py 
6d23c37 
  ambari-agent/src/main/python/ambari_agent/security.py 9509619 
  ambari-common/src/main/python/ambari_stomp/adapter/websocket.py 220f399 
  ambari-common/src/main/python/ambari_ws4py/websocket.py b5c1fd3 


Diff: https://reviews.apache.org/r/61521/diff/1/


Testing
---

mvn clean test


Thanks,

Andrew Onischuk