Re: Review Request 54264: Removing Tasks From host_role_command Causes Upgrades To Show As PENDING

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54264/#review157825
---


Ship it!




Ship It!

- Alejandro Fernandez


On Dec. 2, 2016, 2:43 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54264/
> ---
> 
> (Updated Dec. 2, 2016, 2:43 p.m.)
> 
> 
> Review request for Ambari, Nate Cole, Robert Levas, Robert Nettleton, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-19055
> https://issues.apache.org/jira/browse/AMBARI-19055
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> It may be necessary to remove entries from the {{host_role_command}} table if 
> the size of the table has grown excessively large in order to reduce the 
> query times for "IN_PROGRESS" requests.
> 
> However, if you remove all tasks for an upgrade, but you leave the stages, 
> request, and upgrade items, the upgrade will appear as though it is now 
> {{PENDING}}.
> 
> {noformat:title=Before Removal}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/upgrades/12;,
>   "Upgrade": {
> ...
> "request_status": "COMPLETED",
> "skip_failures": false,
> "skip_service_check_failures": false,
> "start_time": 1480517560950,
> "suspended": false,
> "to_version": "2.5.2.0-67",
> "type": "INTERNAL_REQUEST",
> "upgrade_type": "NON_ROLLING"
>   },
>   "upgrade_groups": [
>   ...
> {noformat}
> 
> {noformat:title=After Removal}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/upgrades/12;,
>   "Upgrade": {
> ...
> "request_status": "PENDING",
> "skip_failures": false,
> "skip_service_check_failures": false,
> "start_time": 1480517560950,
> "suspended": false,
> "to_version": "2.5.2.0-67",
> "type": "INTERNAL_REQUEST",
> "upgrade_type": "NON_ROLLING"
>   },
>   "upgrade_groups": [
>   ...
> {noformat}
> 
> The actual request is showing as {{COMPLETED}} correctly:
> {noformat}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/requests/12;,
>   "Requests": {
> "aborted_task_count": 0,
> "cluster_name": "c1",
> "completed_task_count": 0,
> "create_time": 1480517560897,
> "end_time": 1480517643350,
> "exclusive": false,
> "failed_task_count": 0,
> "id": 12,
> "inputs": null,
> "operation_level": null,
> "progress_percent": 100,
> "queued_task_count": 0,
> "request_context": "Upgrading to 2.5.2.0-67",
> "request_schedule": null,
> "request_status": "COMPLETED",
> "resource_filters": [],
> "start_time": 1480517560950,
> "task_count": 0,
> "timed_out_task_count": 0,
> "type": "INTERNAL_REQUEST"
>   },
>   "tasks": [],
>   ...
> {noformat}
> 
> STR:
> - Perform a simple upgrade of a ZK-only cluster
> - Find out the ID of the upgrade request (say its 12)
> - Remove tasks
> {code}
> DELETE FROM execution_command WHERE task_id IN (SELECT task_id FROM 
> host_role_command WHERE request_id = 12)
> 
> DELETE FROM host_role_command WHERE request_id = 12
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/CalculatedStatus.java
>  3a86aef 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RequestResourceProvider.java
>  8c1bc57 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StageResourceProvider.java
>  59dd9d9 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/CalculatedStatusTest.java
>  6f592cd 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RequestResourceProviderTest.java
>  5dfc74d 
> 
> Diff: https://reviews.apache.org/r/54264/diff/
> 
> 
> Testing
> ---
> 
> Tests run: 4785, Failures: 0, Errors: 0, Skipped: 37
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 26:38 min
> [INFO] Finished at: 2016-12-01T16:51:11-05:00
> [INFO] Final Memory: 57M/704M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 54264: Removing Tasks From host_role_command Causes Upgrades To Show As PENDING

[Jonathan Hurley]

> On Dec. 2, 2016, 1:26 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/CalculatedStatus.java,
> >  line 57
> > 
> >
> > I think this will work most of the time. However, if users are deleting 
> > x records out of the host_role_command table, then there's still a chance 
> > that an RU/EU will still have tasks that aren't deleted, and hence the 
> > upgrade will still show up as PENDING.
> > 
> > In my opinion, we need a better way of deleting old records. What do 
> > you think?
> > 
> > It would be ideal if the user can be prompted to delete data older than 
> > x days. If an EU/RU falls in that category, then we find the requestIDs and 
> > then delete all tasks that belong to those requestIDs. This way, there is 
> > no "partial deletion" of data, which can be problematic.
> 
> Nate Cole wrote:
> Seems to me that people shouldn't be removing records out of the DB that 
> can orphan requests like this.  We have a way to clean records, it just 
> doesn't look like it's implemented for R/S/T (only for Alerts for some 
> reason).
> 
> We can even make it such that only COMPLETED requests are fully purged.  
> Who cares about the history that all DN were successfully restarted three 
> months ago?
> 
> Alejandro Fernandez wrote:
> I believe we have a PreCheck that finds the last Upgrade and makes sure 
> that it completed (not sure if it checks the request status or the last 3 
> tasks during Finalize).
> Either way, I don't think users should leave orphan records. Our story 
> for cleaning up history shouldn't be through manually deleting records. I 
> would prefer if this could be done through an API and eventually UI.

The problem is that we're facing this issue now - with databases that have been 
manually purged. The commands being run are done so to include all tasks within 
a given time period as long as they don't spill over - they run queries before 
to figure out if their query will cut requests in half or if they'll clean them 
out cleanly.

That's a different Jira - this was to address the fact that a 0-task request is 
COMPLETED, not PENDING (unless its logical)


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54264/#review15
---


On Dec. 2, 2016, 9:43 a.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54264/
> ---
> 
> (Updated Dec. 2, 2016, 9:43 a.m.)
> 
> 
> Review request for Ambari, Nate Cole, Robert Levas, Robert Nettleton, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-19055
> https://issues.apache.org/jira/browse/AMBARI-19055
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> It may be necessary to remove entries from the {{host_role_command}} table if 
> the size of the table has grown excessively large in order to reduce the 
> query times for "IN_PROGRESS" requests.
> 
> However, if you remove all tasks for an upgrade, but you leave the stages, 
> request, and upgrade items, the upgrade will appear as though it is now 
> {{PENDING}}.
> 
> {noformat:title=Before Removal}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/upgrades/12;,
>   "Upgrade": {
> ...
> "request_status": "COMPLETED",
> "skip_failures": false,
> "skip_service_check_failures": false,
> "start_time": 1480517560950,
> "suspended": false,
> "to_version": "2.5.2.0-67",
> "type": "INTERNAL_REQUEST",
> "upgrade_type": "NON_ROLLING"
>   },
>   "upgrade_groups": [
>   ...
> {noformat}
> 
> {noformat:title=After Removal}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/upgrades/12;,
>   "Upgrade": {
> ...
> "request_status": "PENDING",
> "skip_failures": false,
> "skip_service_check_failures": false,
> "start_time": 1480517560950,
> "suspended": false,
> "to_version": "2.5.2.0-67",
> "type": "INTERNAL_REQUEST",
> "upgrade_type": "NON_ROLLING"
>   },
>   "upgrade_groups": [
>   ...
> {noformat}
> 
> The actual request is showing as {{COMPLETED}} correctly:
> {noformat}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/requests/12;,
>   "Requests": {
> "aborted_task_count": 0,
> "cluster_name": "c1",
> "completed_task_count": 0,
> "create_time": 1480517560897,
> "end_time": 1480517643350,
> "exclusive": false,
> "failed_task_count": 0,
> "id": 12,
> "inputs": null,
> "operation_level": null,
> "progress_percent": 100,
> "queued_task_count": 0,
> "request_context": "Upgrading to 2.5.2.0-67",
> "request_schedule": null,
> 

Re: Review Request 54264: Removing Tasks From host_role_command Causes Upgrades To Show As PENDING

[Alejandro Fernandez]

> On Dec. 2, 2016, 6:26 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/CalculatedStatus.java,
> >  line 57
> > 
> >
> > I think this will work most of the time. However, if users are deleting 
> > x records out of the host_role_command table, then there's still a chance 
> > that an RU/EU will still have tasks that aren't deleted, and hence the 
> > upgrade will still show up as PENDING.
> > 
> > In my opinion, we need a better way of deleting old records. What do 
> > you think?
> > 
> > It would be ideal if the user can be prompted to delete data older than 
> > x days. If an EU/RU falls in that category, then we find the requestIDs and 
> > then delete all tasks that belong to those requestIDs. This way, there is 
> > no "partial deletion" of data, which can be problematic.
> 
> Nate Cole wrote:
> Seems to me that people shouldn't be removing records out of the DB that 
> can orphan requests like this.  We have a way to clean records, it just 
> doesn't look like it's implemented for R/S/T (only for Alerts for some 
> reason).
> 
> We can even make it such that only COMPLETED requests are fully purged.  
> Who cares about the history that all DN were successfully restarted three 
> months ago?

I believe we have a PreCheck that finds the last Upgrade and makes sure that it 
completed (not sure if it checks the request status or the last 3 tasks during 
Finalize).
Either way, I don't think users should leave orphan records. Our story for 
cleaning up history shouldn't be through manually deleting records. I would 
prefer if this could be done through an API and eventually UI.


- Alejandro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54264/#review15
---


On Dec. 2, 2016, 2:43 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54264/
> ---
> 
> (Updated Dec. 2, 2016, 2:43 p.m.)
> 
> 
> Review request for Ambari, Nate Cole, Robert Levas, Robert Nettleton, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-19055
> https://issues.apache.org/jira/browse/AMBARI-19055
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> It may be necessary to remove entries from the {{host_role_command}} table if 
> the size of the table has grown excessively large in order to reduce the 
> query times for "IN_PROGRESS" requests.
> 
> However, if you remove all tasks for an upgrade, but you leave the stages, 
> request, and upgrade items, the upgrade will appear as though it is now 
> {{PENDING}}.
> 
> {noformat:title=Before Removal}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/upgrades/12;,
>   "Upgrade": {
> ...
> "request_status": "COMPLETED",
> "skip_failures": false,
> "skip_service_check_failures": false,
> "start_time": 1480517560950,
> "suspended": false,
> "to_version": "2.5.2.0-67",
> "type": "INTERNAL_REQUEST",
> "upgrade_type": "NON_ROLLING"
>   },
>   "upgrade_groups": [
>   ...
> {noformat}
> 
> {noformat:title=After Removal}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/upgrades/12;,
>   "Upgrade": {
> ...
> "request_status": "PENDING",
> "skip_failures": false,
> "skip_service_check_failures": false,
> "start_time": 1480517560950,
> "suspended": false,
> "to_version": "2.5.2.0-67",
> "type": "INTERNAL_REQUEST",
> "upgrade_type": "NON_ROLLING"
>   },
>   "upgrade_groups": [
>   ...
> {noformat}
> 
> The actual request is showing as {{COMPLETED}} correctly:
> {noformat}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/requests/12;,
>   "Requests": {
> "aborted_task_count": 0,
> "cluster_name": "c1",
> "completed_task_count": 0,
> "create_time": 1480517560897,
> "end_time": 1480517643350,
> "exclusive": false,
> "failed_task_count": 0,
> "id": 12,
> "inputs": null,
> "operation_level": null,
> "progress_percent": 100,
> "queued_task_count": 0,
> "request_context": "Upgrading to 2.5.2.0-67",
> "request_schedule": null,
> "request_status": "COMPLETED",
> "resource_filters": [],
> "start_time": 1480517560950,
> "task_count": 0,
> "timed_out_task_count": 0,
> "type": "INTERNAL_REQUEST"
>   },
>   "tasks": [],
>   ...
> {noformat}
> 
> STR:
> - Perform a simple upgrade of a ZK-only cluster
> - Find out the ID of the upgrade request (say its 12)
> - Remove tasks
> {code}
> DELETE FROM execution_command WHERE task_id IN (SELECT task_id FROM 
> host_role_command WHERE request_id = 12)
> 
> DELETE FROM 

Re: Review Request 54264: Removing Tasks From host_role_command Causes Upgrades To Show As PENDING

[Nate Cole]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54264/#review157786
---


Ship it!




Ship It!

- Nate Cole


On Dec. 2, 2016, 9:43 a.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54264/
> ---
> 
> (Updated Dec. 2, 2016, 9:43 a.m.)
> 
> 
> Review request for Ambari, Nate Cole, Robert Levas, Robert Nettleton, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-19055
> https://issues.apache.org/jira/browse/AMBARI-19055
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> It may be necessary to remove entries from the {{host_role_command}} table if 
> the size of the table has grown excessively large in order to reduce the 
> query times for "IN_PROGRESS" requests.
> 
> However, if you remove all tasks for an upgrade, but you leave the stages, 
> request, and upgrade items, the upgrade will appear as though it is now 
> {{PENDING}}.
> 
> {noformat:title=Before Removal}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/upgrades/12;,
>   "Upgrade": {
> ...
> "request_status": "COMPLETED",
> "skip_failures": false,
> "skip_service_check_failures": false,
> "start_time": 1480517560950,
> "suspended": false,
> "to_version": "2.5.2.0-67",
> "type": "INTERNAL_REQUEST",
> "upgrade_type": "NON_ROLLING"
>   },
>   "upgrade_groups": [
>   ...
> {noformat}
> 
> {noformat:title=After Removal}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/upgrades/12;,
>   "Upgrade": {
> ...
> "request_status": "PENDING",
> "skip_failures": false,
> "skip_service_check_failures": false,
> "start_time": 1480517560950,
> "suspended": false,
> "to_version": "2.5.2.0-67",
> "type": "INTERNAL_REQUEST",
> "upgrade_type": "NON_ROLLING"
>   },
>   "upgrade_groups": [
>   ...
> {noformat}
> 
> The actual request is showing as {{COMPLETED}} correctly:
> {noformat}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/requests/12;,
>   "Requests": {
> "aborted_task_count": 0,
> "cluster_name": "c1",
> "completed_task_count": 0,
> "create_time": 1480517560897,
> "end_time": 1480517643350,
> "exclusive": false,
> "failed_task_count": 0,
> "id": 12,
> "inputs": null,
> "operation_level": null,
> "progress_percent": 100,
> "queued_task_count": 0,
> "request_context": "Upgrading to 2.5.2.0-67",
> "request_schedule": null,
> "request_status": "COMPLETED",
> "resource_filters": [],
> "start_time": 1480517560950,
> "task_count": 0,
> "timed_out_task_count": 0,
> "type": "INTERNAL_REQUEST"
>   },
>   "tasks": [],
>   ...
> {noformat}
> 
> STR:
> - Perform a simple upgrade of a ZK-only cluster
> - Find out the ID of the upgrade request (say its 12)
> - Remove tasks
> {code}
> DELETE FROM execution_command WHERE task_id IN (SELECT task_id FROM 
> host_role_command WHERE request_id = 12)
> 
> DELETE FROM host_role_command WHERE request_id = 12
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/CalculatedStatus.java
>  3a86aef 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RequestResourceProvider.java
>  8c1bc57 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StageResourceProvider.java
>  59dd9d9 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/CalculatedStatusTest.java
>  6f592cd 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RequestResourceProviderTest.java
>  5dfc74d 
> 
> Diff: https://reviews.apache.org/r/54264/diff/
> 
> 
> Testing
> ---
> 
> Tests run: 4785, Failures: 0, Errors: 0, Skipped: 37
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 26:38 min
> [INFO] Finished at: 2016-12-01T16:51:11-05:00
> [INFO] Final Memory: 57M/704M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 54264: Removing Tasks From host_role_command Causes Upgrades To Show As PENDING

[Nate Cole]

> On Dec. 2, 2016, 1:26 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/CalculatedStatus.java,
> >  line 57
> > 
> >
> > I think this will work most of the time. However, if users are deleting 
> > x records out of the host_role_command table, then there's still a chance 
> > that an RU/EU will still have tasks that aren't deleted, and hence the 
> > upgrade will still show up as PENDING.
> > 
> > In my opinion, we need a better way of deleting old records. What do 
> > you think?
> > 
> > It would be ideal if the user can be prompted to delete data older than 
> > x days. If an EU/RU falls in that category, then we find the requestIDs and 
> > then delete all tasks that belong to those requestIDs. This way, there is 
> > no "partial deletion" of data, which can be problematic.

Seems to me that people shouldn't be removing records out of the DB that can 
orphan requests like this.  We have a way to clean records, it just doesn't 
look like it's implemented for R/S/T (only for Alerts for some reason).

We can even make it such that only COMPLETED requests are fully purged.  Who 
cares about the history that all DN were successfully restarted three months 
ago?


- Nate


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54264/#review15
---


On Dec. 2, 2016, 9:43 a.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54264/
> ---
> 
> (Updated Dec. 2, 2016, 9:43 a.m.)
> 
> 
> Review request for Ambari, Nate Cole, Robert Levas, Robert Nettleton, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-19055
> https://issues.apache.org/jira/browse/AMBARI-19055
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> It may be necessary to remove entries from the {{host_role_command}} table if 
> the size of the table has grown excessively large in order to reduce the 
> query times for "IN_PROGRESS" requests.
> 
> However, if you remove all tasks for an upgrade, but you leave the stages, 
> request, and upgrade items, the upgrade will appear as though it is now 
> {{PENDING}}.
> 
> {noformat:title=Before Removal}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/upgrades/12;,
>   "Upgrade": {
> ...
> "request_status": "COMPLETED",
> "skip_failures": false,
> "skip_service_check_failures": false,
> "start_time": 1480517560950,
> "suspended": false,
> "to_version": "2.5.2.0-67",
> "type": "INTERNAL_REQUEST",
> "upgrade_type": "NON_ROLLING"
>   },
>   "upgrade_groups": [
>   ...
> {noformat}
> 
> {noformat:title=After Removal}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/upgrades/12;,
>   "Upgrade": {
> ...
> "request_status": "PENDING",
> "skip_failures": false,
> "skip_service_check_failures": false,
> "start_time": 1480517560950,
> "suspended": false,
> "to_version": "2.5.2.0-67",
> "type": "INTERNAL_REQUEST",
> "upgrade_type": "NON_ROLLING"
>   },
>   "upgrade_groups": [
>   ...
> {noformat}
> 
> The actual request is showing as {{COMPLETED}} correctly:
> {noformat}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/requests/12;,
>   "Requests": {
> "aborted_task_count": 0,
> "cluster_name": "c1",
> "completed_task_count": 0,
> "create_time": 1480517560897,
> "end_time": 1480517643350,
> "exclusive": false,
> "failed_task_count": 0,
> "id": 12,
> "inputs": null,
> "operation_level": null,
> "progress_percent": 100,
> "queued_task_count": 0,
> "request_context": "Upgrading to 2.5.2.0-67",
> "request_schedule": null,
> "request_status": "COMPLETED",
> "resource_filters": [],
> "start_time": 1480517560950,
> "task_count": 0,
> "timed_out_task_count": 0,
> "type": "INTERNAL_REQUEST"
>   },
>   "tasks": [],
>   ...
> {noformat}
> 
> STR:
> - Perform a simple upgrade of a ZK-only cluster
> - Find out the ID of the upgrade request (say its 12)
> - Remove tasks
> {code}
> DELETE FROM execution_command WHERE task_id IN (SELECT task_id FROM 
> host_role_command WHERE request_id = 12)
> 
> DELETE FROM host_role_command WHERE request_id = 12
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/CalculatedStatus.java
>  3a86aef 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RequestResourceProvider.java
>  8c1bc57 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StageResourceProvider.java
>  59dd9d9 
>   
> 

Re: Review Request 54264: Removing Tasks From host_role_command Causes Upgrades To Show As PENDING

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54264/#review15
---




ambari-server/src/main/java/org/apache/ambari/server/controller/internal/CalculatedStatus.java
 (line 57)


I think this will work most of the time. However, if users are deleting x 
records out of the host_role_command table, then there's still a chance that an 
RU/EU will still have tasks that aren't deleted, and hence the upgrade will 
still show up as PENDING.

In my opinion, we need a better way of deleting old records. What do you 
think?

It would be ideal if the user can be prompted to delete data older than x 
days. If an EU/RU falls in that category, then we find the requestIDs and then 
delete all tasks that belong to those requestIDs. This way, there is no 
"partial deletion" of data, which can be problematic.


- Alejandro Fernandez


On Dec. 2, 2016, 2:43 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54264/
> ---
> 
> (Updated Dec. 2, 2016, 2:43 p.m.)
> 
> 
> Review request for Ambari, Nate Cole, Robert Levas, Robert Nettleton, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-19055
> https://issues.apache.org/jira/browse/AMBARI-19055
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> It may be necessary to remove entries from the {{host_role_command}} table if 
> the size of the table has grown excessively large in order to reduce the 
> query times for "IN_PROGRESS" requests.
> 
> However, if you remove all tasks for an upgrade, but you leave the stages, 
> request, and upgrade items, the upgrade will appear as though it is now 
> {{PENDING}}.
> 
> {noformat:title=Before Removal}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/upgrades/12;,
>   "Upgrade": {
> ...
> "request_status": "COMPLETED",
> "skip_failures": false,
> "skip_service_check_failures": false,
> "start_time": 1480517560950,
> "suspended": false,
> "to_version": "2.5.2.0-67",
> "type": "INTERNAL_REQUEST",
> "upgrade_type": "NON_ROLLING"
>   },
>   "upgrade_groups": [
>   ...
> {noformat}
> 
> {noformat:title=After Removal}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/upgrades/12;,
>   "Upgrade": {
> ...
> "request_status": "PENDING",
> "skip_failures": false,
> "skip_service_check_failures": false,
> "start_time": 1480517560950,
> "suspended": false,
> "to_version": "2.5.2.0-67",
> "type": "INTERNAL_REQUEST",
> "upgrade_type": "NON_ROLLING"
>   },
>   "upgrade_groups": [
>   ...
> {noformat}
> 
> The actual request is showing as {{COMPLETED}} correctly:
> {noformat}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/requests/12;,
>   "Requests": {
> "aborted_task_count": 0,
> "cluster_name": "c1",
> "completed_task_count": 0,
> "create_time": 1480517560897,
> "end_time": 1480517643350,
> "exclusive": false,
> "failed_task_count": 0,
> "id": 12,
> "inputs": null,
> "operation_level": null,
> "progress_percent": 100,
> "queued_task_count": 0,
> "request_context": "Upgrading to 2.5.2.0-67",
> "request_schedule": null,
> "request_status": "COMPLETED",
> "resource_filters": [],
> "start_time": 1480517560950,
> "task_count": 0,
> "timed_out_task_count": 0,
> "type": "INTERNAL_REQUEST"
>   },
>   "tasks": [],
>   ...
> {noformat}
> 
> STR:
> - Perform a simple upgrade of a ZK-only cluster
> - Find out the ID of the upgrade request (say its 12)
> - Remove tasks
> {code}
> DELETE FROM execution_command WHERE task_id IN (SELECT task_id FROM 
> host_role_command WHERE request_id = 12)
> 
> DELETE FROM host_role_command WHERE request_id = 12
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/CalculatedStatus.java
>  3a86aef 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RequestResourceProvider.java
>  8c1bc57 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StageResourceProvider.java
>  59dd9d9 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/CalculatedStatusTest.java
>  6f592cd 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RequestResourceProviderTest.java
>  5dfc74d 
> 
> Diff: https://reviews.apache.org/r/54264/diff/
> 
> 
> Testing
> ---
> 
> Tests run: 4785, Failures: 0, Errors: 0, Skipped: 37
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 

Re: Review Request 54264: Removing Tasks From host_role_command Causes Upgrades To Show As PENDING

[Robert Levas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54264/#review157760
---


Ship it!




Ship It!

- Robert Levas


On Dec. 2, 2016, 9:43 a.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54264/
> ---
> 
> (Updated Dec. 2, 2016, 9:43 a.m.)
> 
> 
> Review request for Ambari, Nate Cole, Robert Levas, Robert Nettleton, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-19055
> https://issues.apache.org/jira/browse/AMBARI-19055
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> It may be necessary to remove entries from the {{host_role_command}} table if 
> the size of the table has grown excessively large in order to reduce the 
> query times for "IN_PROGRESS" requests.
> 
> However, if you remove all tasks for an upgrade, but you leave the stages, 
> request, and upgrade items, the upgrade will appear as though it is now 
> {{PENDING}}.
> 
> {noformat:title=Before Removal}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/upgrades/12;,
>   "Upgrade": {
> ...
> "request_status": "COMPLETED",
> "skip_failures": false,
> "skip_service_check_failures": false,
> "start_time": 1480517560950,
> "suspended": false,
> "to_version": "2.5.2.0-67",
> "type": "INTERNAL_REQUEST",
> "upgrade_type": "NON_ROLLING"
>   },
>   "upgrade_groups": [
>   ...
> {noformat}
> 
> {noformat:title=After Removal}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/upgrades/12;,
>   "Upgrade": {
> ...
> "request_status": "PENDING",
> "skip_failures": false,
> "skip_service_check_failures": false,
> "start_time": 1480517560950,
> "suspended": false,
> "to_version": "2.5.2.0-67",
> "type": "INTERNAL_REQUEST",
> "upgrade_type": "NON_ROLLING"
>   },
>   "upgrade_groups": [
>   ...
> {noformat}
> 
> The actual request is showing as {{COMPLETED}} correctly:
> {noformat}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/requests/12;,
>   "Requests": {
> "aborted_task_count": 0,
> "cluster_name": "c1",
> "completed_task_count": 0,
> "create_time": 1480517560897,
> "end_time": 1480517643350,
> "exclusive": false,
> "failed_task_count": 0,
> "id": 12,
> "inputs": null,
> "operation_level": null,
> "progress_percent": 100,
> "queued_task_count": 0,
> "request_context": "Upgrading to 2.5.2.0-67",
> "request_schedule": null,
> "request_status": "COMPLETED",
> "resource_filters": [],
> "start_time": 1480517560950,
> "task_count": 0,
> "timed_out_task_count": 0,
> "type": "INTERNAL_REQUEST"
>   },
>   "tasks": [],
>   ...
> {noformat}
> 
> STR:
> - Perform a simple upgrade of a ZK-only cluster
> - Find out the ID of the upgrade request (say its 12)
> - Remove tasks
> {code}
> DELETE FROM execution_command WHERE task_id IN (SELECT task_id FROM 
> host_role_command WHERE request_id = 12)
> 
> DELETE FROM host_role_command WHERE request_id = 12
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/CalculatedStatus.java
>  3a86aef 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RequestResourceProvider.java
>  8c1bc57 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StageResourceProvider.java
>  59dd9d9 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/CalculatedStatusTest.java
>  6f592cd 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RequestResourceProviderTest.java
>  5dfc74d 
> 
> Diff: https://reviews.apache.org/r/54264/diff/
> 
> 
> Testing
> ---
> 
> Tests run: 4785, Failures: 0, Errors: 0, Skipped: 37
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 26:38 min
> [INFO] Finished at: 2016-12-01T16:51:11-05:00
> [INFO] Final Memory: 57M/704M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 54264: Removing Tasks From host_role_command Causes Upgrades To Show As PENDING

[Jonathan Hurley]

> On Dec. 2, 2016, 7:42 a.m., Robert Levas wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/CalculatedStatus.java,
> >  line 93
> > 
> >
> > Shouldn't this return a previously instantiated constant (`private 
> > static final ...)`?

Yes, yes it should ... I'll change this one and the other one.


> On Dec. 2, 2016, 7:42 a.m., Robert Levas wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RequestResourceProvider.java,
> >  line 753
> > 
> >
> > Wouldn't it be _cheaper_ to put this in an `else` block rather and 
> > perform the calculation only to overwrite the result if it is a logical 
> > topolpgy request?

Yep, I'll make that change.


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54264/#review157733
---


On Dec. 1, 2016, 5:45 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54264/
> ---
> 
> (Updated Dec. 1, 2016, 5:45 p.m.)
> 
> 
> Review request for Ambari, Nate Cole, Robert Levas, Robert Nettleton, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-19055
> https://issues.apache.org/jira/browse/AMBARI-19055
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> It may be necessary to remove entries from the {{host_role_command}} table if 
> the size of the table has grown excessively large in order to reduce the 
> query times for "IN_PROGRESS" requests.
> 
> However, if you remove all tasks for an upgrade, but you leave the stages, 
> request, and upgrade items, the upgrade will appear as though it is now 
> {{PENDING}}.
> 
> {noformat:title=Before Removal}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/upgrades/12;,
>   "Upgrade": {
> ...
> "request_status": "COMPLETED",
> "skip_failures": false,
> "skip_service_check_failures": false,
> "start_time": 1480517560950,
> "suspended": false,
> "to_version": "2.5.2.0-67",
> "type": "INTERNAL_REQUEST",
> "upgrade_type": "NON_ROLLING"
>   },
>   "upgrade_groups": [
>   ...
> {noformat}
> 
> {noformat:title=After Removal}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/upgrades/12;,
>   "Upgrade": {
> ...
> "request_status": "PENDING",
> "skip_failures": false,
> "skip_service_check_failures": false,
> "start_time": 1480517560950,
> "suspended": false,
> "to_version": "2.5.2.0-67",
> "type": "INTERNAL_REQUEST",
> "upgrade_type": "NON_ROLLING"
>   },
>   "upgrade_groups": [
>   ...
> {noformat}
> 
> The actual request is showing as {{COMPLETED}} correctly:
> {noformat}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/requests/12;,
>   "Requests": {
> "aborted_task_count": 0,
> "cluster_name": "c1",
> "completed_task_count": 0,
> "create_time": 1480517560897,
> "end_time": 1480517643350,
> "exclusive": false,
> "failed_task_count": 0,
> "id": 12,
> "inputs": null,
> "operation_level": null,
> "progress_percent": 100,
> "queued_task_count": 0,
> "request_context": "Upgrading to 2.5.2.0-67",
> "request_schedule": null,
> "request_status": "COMPLETED",
> "resource_filters": [],
> "start_time": 1480517560950,
> "task_count": 0,
> "timed_out_task_count": 0,
> "type": "INTERNAL_REQUEST"
>   },
>   "tasks": [],
>   ...
> {noformat}
> 
> STR:
> - Perform a simple upgrade of a ZK-only cluster
> - Find out the ID of the upgrade request (say its 12)
> - Remove tasks
> {code}
> DELETE FROM execution_command WHERE task_id IN (SELECT task_id FROM 
> host_role_command WHERE request_id = 12)
> 
> DELETE FROM host_role_command WHERE request_id = 12
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/CalculatedStatus.java
>  3a86aef 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RequestResourceProvider.java
>  8c1bc57 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/CalculatedStatusTest.java
>  6f592cd 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RequestResourceProviderTest.java
>  5dfc74d 
> 
> Diff: https://reviews.apache.org/r/54264/diff/
> 
> 
> Testing
> ---
> 
> Tests run: 4785, Failures: 0, Errors: 0, Skipped: 37
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 26:38 

Re: Review Request 54264: Removing Tasks From host_role_command Causes Upgrades To Show As PENDING

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54264/
---

(Updated Dec. 2, 2016, 9:43 a.m.)


Review request for Ambari, Nate Cole, Robert Levas, Robert Nettleton, and 
Sebastian Toader.


Changes
---

Updated with reviewer's comments.


Bugs: AMBARI-19055
https://issues.apache.org/jira/browse/AMBARI-19055


Repository: ambari


Description
---

It may be necessary to remove entries from the {{host_role_command}} table if 
the size of the table has grown excessively large in order to reduce the query 
times for "IN_PROGRESS" requests.

However, if you remove all tasks for an upgrade, but you leave the stages, 
request, and upgrade items, the upgrade will appear as though it is now 
{{PENDING}}.

{noformat:title=Before Removal}
{
  "href": "http://localhost:8080/api/v1/clusters/c1/upgrades/12;,
  "Upgrade": {
...
"request_status": "COMPLETED",
"skip_failures": false,
"skip_service_check_failures": false,
"start_time": 1480517560950,
"suspended": false,
"to_version": "2.5.2.0-67",
"type": "INTERNAL_REQUEST",
"upgrade_type": "NON_ROLLING"
  },
  "upgrade_groups": [
  ...
{noformat}

{noformat:title=After Removal}
{
  "href": "http://localhost:8080/api/v1/clusters/c1/upgrades/12;,
  "Upgrade": {
...
"request_status": "PENDING",
"skip_failures": false,
"skip_service_check_failures": false,
"start_time": 1480517560950,
"suspended": false,
"to_version": "2.5.2.0-67",
"type": "INTERNAL_REQUEST",
"upgrade_type": "NON_ROLLING"
  },
  "upgrade_groups": [
  ...
{noformat}

The actual request is showing as {{COMPLETED}} correctly:
{noformat}
{
  "href": "http://localhost:8080/api/v1/clusters/c1/requests/12;,
  "Requests": {
"aborted_task_count": 0,
"cluster_name": "c1",
"completed_task_count": 0,
"create_time": 1480517560897,
"end_time": 1480517643350,
"exclusive": false,
"failed_task_count": 0,
"id": 12,
"inputs": null,
"operation_level": null,
"progress_percent": 100,
"queued_task_count": 0,
"request_context": "Upgrading to 2.5.2.0-67",
"request_schedule": null,
"request_status": "COMPLETED",
"resource_filters": [],
"start_time": 1480517560950,
"task_count": 0,
"timed_out_task_count": 0,
"type": "INTERNAL_REQUEST"
  },
  "tasks": [],
  ...
{noformat}

STR:
- Perform a simple upgrade of a ZK-only cluster
- Find out the ID of the upgrade request (say its 12)
- Remove tasks
{code}
DELETE FROM execution_command WHERE task_id IN (SELECT task_id FROM 
host_role_command WHERE request_id = 12)

DELETE FROM host_role_command WHERE request_id = 12
{code}


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/CalculatedStatus.java
 3a86aef 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RequestResourceProvider.java
 8c1bc57 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StageResourceProvider.java
 59dd9d9 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/CalculatedStatusTest.java
 6f592cd 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RequestResourceProviderTest.java
 5dfc74d 

Diff: https://reviews.apache.org/r/54264/diff/


Testing
---

Tests run: 4785, Failures: 0, Errors: 0, Skipped: 37

[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 26:38 min
[INFO] Finished at: 2016-12-01T16:51:11-05:00
[INFO] Final Memory: 57M/704M
[INFO] 


Thanks,

Jonathan Hurley

Re: Review Request 54264: Removing Tasks From host_role_command Causes Upgrades To Show As PENDING

[Robert Nettleton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54264/#review157749
---


Ship it!




Ship It!

- Robert Nettleton


On Dec. 1, 2016, 10:45 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54264/
> ---
> 
> (Updated Dec. 1, 2016, 10:45 p.m.)
> 
> 
> Review request for Ambari, Nate Cole, Robert Levas, Robert Nettleton, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-19055
> https://issues.apache.org/jira/browse/AMBARI-19055
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> It may be necessary to remove entries from the {{host_role_command}} table if 
> the size of the table has grown excessively large in order to reduce the 
> query times for "IN_PROGRESS" requests.
> 
> However, if you remove all tasks for an upgrade, but you leave the stages, 
> request, and upgrade items, the upgrade will appear as though it is now 
> {{PENDING}}.
> 
> {noformat:title=Before Removal}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/upgrades/12;,
>   "Upgrade": {
> ...
> "request_status": "COMPLETED",
> "skip_failures": false,
> "skip_service_check_failures": false,
> "start_time": 1480517560950,
> "suspended": false,
> "to_version": "2.5.2.0-67",
> "type": "INTERNAL_REQUEST",
> "upgrade_type": "NON_ROLLING"
>   },
>   "upgrade_groups": [
>   ...
> {noformat}
> 
> {noformat:title=After Removal}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/upgrades/12;,
>   "Upgrade": {
> ...
> "request_status": "PENDING",
> "skip_failures": false,
> "skip_service_check_failures": false,
> "start_time": 1480517560950,
> "suspended": false,
> "to_version": "2.5.2.0-67",
> "type": "INTERNAL_REQUEST",
> "upgrade_type": "NON_ROLLING"
>   },
>   "upgrade_groups": [
>   ...
> {noformat}
> 
> The actual request is showing as {{COMPLETED}} correctly:
> {noformat}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/requests/12;,
>   "Requests": {
> "aborted_task_count": 0,
> "cluster_name": "c1",
> "completed_task_count": 0,
> "create_time": 1480517560897,
> "end_time": 1480517643350,
> "exclusive": false,
> "failed_task_count": 0,
> "id": 12,
> "inputs": null,
> "operation_level": null,
> "progress_percent": 100,
> "queued_task_count": 0,
> "request_context": "Upgrading to 2.5.2.0-67",
> "request_schedule": null,
> "request_status": "COMPLETED",
> "resource_filters": [],
> "start_time": 1480517560950,
> "task_count": 0,
> "timed_out_task_count": 0,
> "type": "INTERNAL_REQUEST"
>   },
>   "tasks": [],
>   ...
> {noformat}
> 
> STR:
> - Perform a simple upgrade of a ZK-only cluster
> - Find out the ID of the upgrade request (say its 12)
> - Remove tasks
> {code}
> DELETE FROM execution_command WHERE task_id IN (SELECT task_id FROM 
> host_role_command WHERE request_id = 12)
> 
> DELETE FROM host_role_command WHERE request_id = 12
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/CalculatedStatus.java
>  3a86aef 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RequestResourceProvider.java
>  8c1bc57 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/CalculatedStatusTest.java
>  6f592cd 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RequestResourceProviderTest.java
>  5dfc74d 
> 
> Diff: https://reviews.apache.org/r/54264/diff/
> 
> 
> Testing
> ---
> 
> Tests run: 4785, Failures: 0, Errors: 0, Skipped: 37
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 26:38 min
> [INFO] Finished at: 2016-12-01T16:51:11-05:00
> [INFO] Final Memory: 57M/704M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 54264: Removing Tasks From host_role_command Causes Upgrades To Show As PENDING

[Robert Levas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54264/#review157733
---


Fix it, then Ship it!




Ship It!


ambari-server/src/main/java/org/apache/ambari/server/controller/internal/CalculatedStatus.java
 (line 93)


Shouldn't this return a previously instantiated constant (`private static 
final ...)`?



ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RequestResourceProvider.java
 (line 751)


Wouldn't it be _cheaper_ to put this in an `else` block rather and perform 
the calculation only to overwrite the result if it is a logical topolpgy 
request?


- Robert Levas


On Dec. 1, 2016, 5:45 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54264/
> ---
> 
> (Updated Dec. 1, 2016, 5:45 p.m.)
> 
> 
> Review request for Ambari, Nate Cole, Robert Levas, Robert Nettleton, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-19055
> https://issues.apache.org/jira/browse/AMBARI-19055
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> It may be necessary to remove entries from the {{host_role_command}} table if 
> the size of the table has grown excessively large in order to reduce the 
> query times for "IN_PROGRESS" requests.
> 
> However, if you remove all tasks for an upgrade, but you leave the stages, 
> request, and upgrade items, the upgrade will appear as though it is now 
> {{PENDING}}.
> 
> {noformat:title=Before Removal}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/upgrades/12;,
>   "Upgrade": {
> ...
> "request_status": "COMPLETED",
> "skip_failures": false,
> "skip_service_check_failures": false,
> "start_time": 1480517560950,
> "suspended": false,
> "to_version": "2.5.2.0-67",
> "type": "INTERNAL_REQUEST",
> "upgrade_type": "NON_ROLLING"
>   },
>   "upgrade_groups": [
>   ...
> {noformat}
> 
> {noformat:title=After Removal}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/upgrades/12;,
>   "Upgrade": {
> ...
> "request_status": "PENDING",
> "skip_failures": false,
> "skip_service_check_failures": false,
> "start_time": 1480517560950,
> "suspended": false,
> "to_version": "2.5.2.0-67",
> "type": "INTERNAL_REQUEST",
> "upgrade_type": "NON_ROLLING"
>   },
>   "upgrade_groups": [
>   ...
> {noformat}
> 
> The actual request is showing as {{COMPLETED}} correctly:
> {noformat}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/requests/12;,
>   "Requests": {
> "aborted_task_count": 0,
> "cluster_name": "c1",
> "completed_task_count": 0,
> "create_time": 1480517560897,
> "end_time": 1480517643350,
> "exclusive": false,
> "failed_task_count": 0,
> "id": 12,
> "inputs": null,
> "operation_level": null,
> "progress_percent": 100,
> "queued_task_count": 0,
> "request_context": "Upgrading to 2.5.2.0-67",
> "request_schedule": null,
> "request_status": "COMPLETED",
> "resource_filters": [],
> "start_time": 1480517560950,
> "task_count": 0,
> "timed_out_task_count": 0,
> "type": "INTERNAL_REQUEST"
>   },
>   "tasks": [],
>   ...
> {noformat}
> 
> STR:
> - Perform a simple upgrade of a ZK-only cluster
> - Find out the ID of the upgrade request (say its 12)
> - Remove tasks
> {code}
> DELETE FROM execution_command WHERE task_id IN (SELECT task_id FROM 
> host_role_command WHERE request_id = 12)
> 
> DELETE FROM host_role_command WHERE request_id = 12
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/CalculatedStatus.java
>  3a86aef 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RequestResourceProvider.java
>  8c1bc57 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/CalculatedStatusTest.java
>  6f592cd 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RequestResourceProviderTest.java
>  5dfc74d 
> 
> Diff: https://reviews.apache.org/r/54264/diff/
> 
> 
> Testing
> ---
> 
> Tests run: 4785, Failures: 0, Errors: 0, Skipped: 37
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 26:38 min
> [INFO] Finished at: 2016-12-01T16:51:11-05:00
> [INFO] Final Memory: 57M/704M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 54264: Removing Tasks From host_role_command Causes Upgrades To Show As PENDING

[Sebastian Toader]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54264/#review157734
---


Ship it!




Ship It!

- Sebastian Toader


On Dec. 1, 2016, 11:45 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54264/
> ---
> 
> (Updated Dec. 1, 2016, 11:45 p.m.)
> 
> 
> Review request for Ambari, Nate Cole, Robert Levas, Robert Nettleton, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-19055
> https://issues.apache.org/jira/browse/AMBARI-19055
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> It may be necessary to remove entries from the {{host_role_command}} table if 
> the size of the table has grown excessively large in order to reduce the 
> query times for "IN_PROGRESS" requests.
> 
> However, if you remove all tasks for an upgrade, but you leave the stages, 
> request, and upgrade items, the upgrade will appear as though it is now 
> {{PENDING}}.
> 
> {noformat:title=Before Removal}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/upgrades/12;,
>   "Upgrade": {
> ...
> "request_status": "COMPLETED",
> "skip_failures": false,
> "skip_service_check_failures": false,
> "start_time": 1480517560950,
> "suspended": false,
> "to_version": "2.5.2.0-67",
> "type": "INTERNAL_REQUEST",
> "upgrade_type": "NON_ROLLING"
>   },
>   "upgrade_groups": [
>   ...
> {noformat}
> 
> {noformat:title=After Removal}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/upgrades/12;,
>   "Upgrade": {
> ...
> "request_status": "PENDING",
> "skip_failures": false,
> "skip_service_check_failures": false,
> "start_time": 1480517560950,
> "suspended": false,
> "to_version": "2.5.2.0-67",
> "type": "INTERNAL_REQUEST",
> "upgrade_type": "NON_ROLLING"
>   },
>   "upgrade_groups": [
>   ...
> {noformat}
> 
> The actual request is showing as {{COMPLETED}} correctly:
> {noformat}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/requests/12;,
>   "Requests": {
> "aborted_task_count": 0,
> "cluster_name": "c1",
> "completed_task_count": 0,
> "create_time": 1480517560897,
> "end_time": 1480517643350,
> "exclusive": false,
> "failed_task_count": 0,
> "id": 12,
> "inputs": null,
> "operation_level": null,
> "progress_percent": 100,
> "queued_task_count": 0,
> "request_context": "Upgrading to 2.5.2.0-67",
> "request_schedule": null,
> "request_status": "COMPLETED",
> "resource_filters": [],
> "start_time": 1480517560950,
> "task_count": 0,
> "timed_out_task_count": 0,
> "type": "INTERNAL_REQUEST"
>   },
>   "tasks": [],
>   ...
> {noformat}
> 
> STR:
> - Perform a simple upgrade of a ZK-only cluster
> - Find out the ID of the upgrade request (say its 12)
> - Remove tasks
> {code}
> DELETE FROM execution_command WHERE task_id IN (SELECT task_id FROM 
> host_role_command WHERE request_id = 12)
> 
> DELETE FROM host_role_command WHERE request_id = 12
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/CalculatedStatus.java
>  3a86aef 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RequestResourceProvider.java
>  8c1bc57 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/CalculatedStatusTest.java
>  6f592cd 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RequestResourceProviderTest.java
>  5dfc74d 
> 
> Diff: https://reviews.apache.org/r/54264/diff/
> 
> 
> Testing
> ---
> 
> Tests run: 4785, Failures: 0, Errors: 0, Skipped: 37
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 26:38 min
> [INFO] Finished at: 2016-12-01T16:51:11-05:00
> [INFO] Final Memory: 57M/704M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 54264: Removing Tasks From host_role_command Causes Upgrades To Show As PENDING

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54264/#review157663
---




ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RequestResourceProvider.java
 (lines 747 - 757)


https://issues.apache.org/jira/browse/AMBARI-15643 needs "logical" requests 
created by blueprints to be in the PENDING state when there are no tasks.

This is because as hosts join and register, tasks are scheduled on-the-fly.

All other times, if there are no tasks, we should consider the request as 
COMPLETED.

So the check here is to see if there are no tasks but this is a logical 
request - if so, then it can be considered as PENDING.


- Jonathan Hurley


On Dec. 1, 2016, 5:45 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54264/
> ---
> 
> (Updated Dec. 1, 2016, 5:45 p.m.)
> 
> 
> Review request for Ambari, Nate Cole, Robert Levas, Robert Nettleton, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-19055
> https://issues.apache.org/jira/browse/AMBARI-19055
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> It may be necessary to remove entries from the {{host_role_command}} table if 
> the size of the table has grown excessively large in order to reduce the 
> query times for "IN_PROGRESS" requests.
> 
> However, if you remove all tasks for an upgrade, but you leave the stages, 
> request, and upgrade items, the upgrade will appear as though it is now 
> {{PENDING}}.
> 
> {noformat:title=Before Removal}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/upgrades/12;,
>   "Upgrade": {
> ...
> "request_status": "COMPLETED",
> "skip_failures": false,
> "skip_service_check_failures": false,
> "start_time": 1480517560950,
> "suspended": false,
> "to_version": "2.5.2.0-67",
> "type": "INTERNAL_REQUEST",
> "upgrade_type": "NON_ROLLING"
>   },
>   "upgrade_groups": [
>   ...
> {noformat}
> 
> {noformat:title=After Removal}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/upgrades/12;,
>   "Upgrade": {
> ...
> "request_status": "PENDING",
> "skip_failures": false,
> "skip_service_check_failures": false,
> "start_time": 1480517560950,
> "suspended": false,
> "to_version": "2.5.2.0-67",
> "type": "INTERNAL_REQUEST",
> "upgrade_type": "NON_ROLLING"
>   },
>   "upgrade_groups": [
>   ...
> {noformat}
> 
> The actual request is showing as {{COMPLETED}} correctly:
> {noformat}
> {
>   "href": "http://localhost:8080/api/v1/clusters/c1/requests/12;,
>   "Requests": {
> "aborted_task_count": 0,
> "cluster_name": "c1",
> "completed_task_count": 0,
> "create_time": 1480517560897,
> "end_time": 1480517643350,
> "exclusive": false,
> "failed_task_count": 0,
> "id": 12,
> "inputs": null,
> "operation_level": null,
> "progress_percent": 100,
> "queued_task_count": 0,
> "request_context": "Upgrading to 2.5.2.0-67",
> "request_schedule": null,
> "request_status": "COMPLETED",
> "resource_filters": [],
> "start_time": 1480517560950,
> "task_count": 0,
> "timed_out_task_count": 0,
> "type": "INTERNAL_REQUEST"
>   },
>   "tasks": [],
>   ...
> {noformat}
> 
> STR:
> - Perform a simple upgrade of a ZK-only cluster
> - Find out the ID of the upgrade request (say its 12)
> - Remove tasks
> {code}
> DELETE FROM execution_command WHERE task_id IN (SELECT task_id FROM 
> host_role_command WHERE request_id = 12)
> 
> DELETE FROM host_role_command WHERE request_id = 12
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/CalculatedStatus.java
>  3a86aef 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RequestResourceProvider.java
>  8c1bc57 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/CalculatedStatusTest.java
>  6f592cd 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RequestResourceProviderTest.java
>  5dfc74d 
> 
> Diff: https://reviews.apache.org/r/54264/diff/
> 
> 
> Testing
> ---
> 
> Tests run: 4785, Failures: 0, Errors: 0, Skipped: 37
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 26:38 min
> [INFO] Finished at: 2016-12-01T16:51:11-05:00
> [INFO] Final Memory: 57M/704M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 54264: Removing Tasks From host_role_command Causes Upgrades To Show As PENDING

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54264/
---

(Updated Dec. 1, 2016, 5:45 p.m.)


Review request for Ambari, Nate Cole and Robert Levas.


Changes
---

Updated unit tests to account for logical (topology) requests


Bugs: AMBARI-19055
https://issues.apache.org/jira/browse/AMBARI-19055


Repository: ambari


Description
---

It may be necessary to remove entries from the {{host_role_command}} table if 
the size of the table has grown excessively large in order to reduce the query 
times for "IN_PROGRESS" requests.

However, if you remove all tasks for an upgrade, but you leave the stages, 
request, and upgrade items, the upgrade will appear as though it is now 
{{PENDING}}.

{noformat:title=Before Removal}
{
  "href": "http://localhost:8080/api/v1/clusters/c1/upgrades/12;,
  "Upgrade": {
...
"request_status": "COMPLETED",
"skip_failures": false,
"skip_service_check_failures": false,
"start_time": 1480517560950,
"suspended": false,
"to_version": "2.5.2.0-67",
"type": "INTERNAL_REQUEST",
"upgrade_type": "NON_ROLLING"
  },
  "upgrade_groups": [
  ...
{noformat}

{noformat:title=After Removal}
{
  "href": "http://localhost:8080/api/v1/clusters/c1/upgrades/12;,
  "Upgrade": {
...
"request_status": "PENDING",
"skip_failures": false,
"skip_service_check_failures": false,
"start_time": 1480517560950,
"suspended": false,
"to_version": "2.5.2.0-67",
"type": "INTERNAL_REQUEST",
"upgrade_type": "NON_ROLLING"
  },
  "upgrade_groups": [
  ...
{noformat}

The actual request is showing as {{COMPLETED}} correctly:
{noformat}
{
  "href": "http://localhost:8080/api/v1/clusters/c1/requests/12;,
  "Requests": {
"aborted_task_count": 0,
"cluster_name": "c1",
"completed_task_count": 0,
"create_time": 1480517560897,
"end_time": 1480517643350,
"exclusive": false,
"failed_task_count": 0,
"id": 12,
"inputs": null,
"operation_level": null,
"progress_percent": 100,
"queued_task_count": 0,
"request_context": "Upgrading to 2.5.2.0-67",
"request_schedule": null,
"request_status": "COMPLETED",
"resource_filters": [],
"start_time": 1480517560950,
"task_count": 0,
"timed_out_task_count": 0,
"type": "INTERNAL_REQUEST"
  },
  "tasks": [],
  ...
{noformat}

STR:
- Perform a simple upgrade of a ZK-only cluster
- Find out the ID of the upgrade request (say its 12)
- Remove tasks
{code}
DELETE FROM execution_command WHERE task_id IN (SELECT task_id FROM 
host_role_command WHERE request_id = 12)

DELETE FROM host_role_command WHERE request_id = 12
{code}


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/CalculatedStatus.java
 3a86aef 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RequestResourceProvider.java
 8c1bc57 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/CalculatedStatusTest.java
 6f592cd 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RequestResourceProviderTest.java
 5dfc74d 

Diff: https://reviews.apache.org/r/54264/diff/


Testing (updated)
---

Tests run: 4785, Failures: 0, Errors: 0, Skipped: 37

[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 26:38 min
[INFO] Finished at: 2016-12-01T16:51:11-05:00
[INFO] Final Memory: 57M/704M
[INFO] 


Thanks,

Jonathan Hurley