Re: Review Request 50198: HAWQ fails to execute queries in low-memory environment

2016-07-19 Thread Matt 


> On July 19, 2016, 2:50 p.m., Lav Jain wrote:
> > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py,
> >  line 392
> > 
> >
> > Should config-name be set to hawq_rm_memory_limit_perseg?

In YARN mode, hawq_rm_memory_limit_perseg is invisible. 
We would like to show the users the error that the memory is low regardless of 
which mode it is on. Hence we show the error on hawq_global_rm_type property, 
which is always visible.


- Matt


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50198/#review142847
---


On July 19, 2016, 4:34 p.m., Matt wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50198/
> ---
> 
> (Updated July 19, 2016, 4:34 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov and Lav Jain.
> 
> 
> Bugs: AMBARI-17720
> https://issues.apache.org/jira/browse/AMBARI-17720
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When a user is running on vagrant or single node with low system memory, 
> configurations are not sufficient to allocate containers for "default" 6 
> vsegs per node especially during HASH dist table query or external table 
> query.
> 
> Stack advisor should adjust the config values to ensure service checks and 
> queries (including Random tables, HASH dist tables, PXF ext tables) do not 
> fail.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/configuration/hawq-site.xml
>  0bddb07 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  dffe57d 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> 4089b75 
>   ambari-server/src/test/python/common-services/HAWQ/test_service_advisor.py 
> f8a9468 
>   ambari-server/src/test/python/common-services/PXF/test_service_advisor.py 
> 7510e5f 
> 
> Diff: https://reviews.apache.org/r/50198/diff/
> 
> 
> Testing
> ---
> 
> Tested on Live cluster
> 
> Added unit tests:
> ```
> test_hawq_master_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_master_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_hawq_segment_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_segment_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_hawq_standby_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_standby_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_missing_configs (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_exception_registration_status 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... No 
> handlers could be found for logger "ambari_alerts"
> ok
> test_missing_configs 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_missing_slave_file 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_successful_registration_status 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_unsuccessful_empty_db_registration_status 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_unsuccessful_registration_status 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_unsuccessful_registration_status_plural 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_missing_configs (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_no_standby_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_none_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_not_configured_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_not_synchronized_state (test_alert_sync_status.TestAlertSyncStatus) ... 
> ok
> test_synchronized_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_synchronizing_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_unknown_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_configure_default (test_hawqmaster.TestHawqMaster) ... ok
> test_install_default (test_hawqmaster.TestHawqMaster) ... ok
> test_remove_hawq_standby (test_hawqmaster.TestHawqMaster)
> Run custom command Remove HAWQ Standby ... 2016-07-19 12:12:33,657 - Removing 
> HAWQ Standby Master ...
> ok
> test_resync_hawq_standby (test_hawqmaster.TestHawqMaster)
> Run custom command Resync HAWQ Standby ... 2016-07-19 12:12:33,660 - HAWQ 
> Standby Master Re-Sync started in fast mode...
> ok
> test_run_hawq_check_case1

Re: Review Request 50198: HAWQ fails to execute queries in low-memory environment

2016-07-19 Thread Matt 


> On July 19, 2016, 1:59 p.m., Alexander Denissov wrote:
> > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py,
> >  line 213
> > 
> >
> > this property has no impact on the logic inside this if, maybe move it 
> > out of the if statement ?

Moved it to another if statement


- Matt


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50198/#review142831
---


On July 19, 2016, 4:34 p.m., Matt wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50198/
> ---
> 
> (Updated July 19, 2016, 4:34 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov and Lav Jain.
> 
> 
> Bugs: AMBARI-17720
> https://issues.apache.org/jira/browse/AMBARI-17720
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When a user is running on vagrant or single node with low system memory, 
> configurations are not sufficient to allocate containers for "default" 6 
> vsegs per node especially during HASH dist table query or external table 
> query.
> 
> Stack advisor should adjust the config values to ensure service checks and 
> queries (including Random tables, HASH dist tables, PXF ext tables) do not 
> fail.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/configuration/hawq-site.xml
>  0bddb07 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  dffe57d 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> 4089b75 
>   ambari-server/src/test/python/common-services/HAWQ/test_service_advisor.py 
> f8a9468 
>   ambari-server/src/test/python/common-services/PXF/test_service_advisor.py 
> 7510e5f 
> 
> Diff: https://reviews.apache.org/r/50198/diff/
> 
> 
> Testing
> ---
> 
> Tested on Live cluster
> 
> Added unit tests:
> ```
> test_hawq_master_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_master_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_hawq_segment_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_segment_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_hawq_standby_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_standby_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_missing_configs (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_exception_registration_status 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... No 
> handlers could be found for logger "ambari_alerts"
> ok
> test_missing_configs 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_missing_slave_file 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_successful_registration_status 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_unsuccessful_empty_db_registration_status 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_unsuccessful_registration_status 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_unsuccessful_registration_status_plural 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_missing_configs (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_no_standby_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_none_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_not_configured_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_not_synchronized_state (test_alert_sync_status.TestAlertSyncStatus) ... 
> ok
> test_synchronized_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_synchronizing_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_unknown_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_configure_default (test_hawqmaster.TestHawqMaster) ... ok
> test_install_default (test_hawqmaster.TestHawqMaster) ... ok
> test_remove_hawq_standby (test_hawqmaster.TestHawqMaster)
> Run custom command Remove HAWQ Standby ... 2016-07-19 12:12:33,657 - Removing 
> HAWQ Standby Master ...
> ok
> test_resync_hawq_standby (test_hawqmaster.TestHawqMaster)
> Run custom command Resync HAWQ Standby ... 2016-07-19 12:12:33,660 - HAWQ 
> Standby Master Re-Sync started in fast mode...
> ok
> test_run_hawq_check_case1 (test_hawqmaster.TestHawqMaster)
> Running HAWQ Check Case 1: Non HDFS-HA, Standalone Resource Management, Not 
> Kerberized ... 2016-07-19 12:12:33,663 - Executing

Re: Review Request 50198: HAWQ fails to execute queries in low-memory environment

2016-07-19 Thread Matt 


> On July 19, 2016, 2:50 p.m., Lav Jain wrote:
> > ambari-server/src/test/python/common-services/HAWQ/test_service_advisor.py, 
> > line 378
> > 
> >
> > Should line 378 be hosts["items"][2] instead of hosts["items"][3]? Same 
> > for line 390.

They should all be 0,1,3
2 does not have HAWQ components.


- Matt


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50198/#review142847
---


On July 19, 2016, 4:34 p.m., Matt wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50198/
> ---
> 
> (Updated July 19, 2016, 4:34 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov and Lav Jain.
> 
> 
> Bugs: AMBARI-17720
> https://issues.apache.org/jira/browse/AMBARI-17720
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When a user is running on vagrant or single node with low system memory, 
> configurations are not sufficient to allocate containers for "default" 6 
> vsegs per node especially during HASH dist table query or external table 
> query.
> 
> Stack advisor should adjust the config values to ensure service checks and 
> queries (including Random tables, HASH dist tables, PXF ext tables) do not 
> fail.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/configuration/hawq-site.xml
>  0bddb07 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  dffe57d 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> 4089b75 
>   ambari-server/src/test/python/common-services/HAWQ/test_service_advisor.py 
> f8a9468 
>   ambari-server/src/test/python/common-services/PXF/test_service_advisor.py 
> 7510e5f 
> 
> Diff: https://reviews.apache.org/r/50198/diff/
> 
> 
> Testing
> ---
> 
> Tested on Live cluster
> 
> Added unit tests:
> ```
> test_hawq_master_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_master_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_hawq_segment_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_segment_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_hawq_standby_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_standby_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_missing_configs (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_exception_registration_status 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... No 
> handlers could be found for logger "ambari_alerts"
> ok
> test_missing_configs 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_missing_slave_file 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_successful_registration_status 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_unsuccessful_empty_db_registration_status 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_unsuccessful_registration_status 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_unsuccessful_registration_status_plural 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_missing_configs (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_no_standby_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_none_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_not_configured_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_not_synchronized_state (test_alert_sync_status.TestAlertSyncStatus) ... 
> ok
> test_synchronized_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_synchronizing_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_unknown_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_configure_default (test_hawqmaster.TestHawqMaster) ... ok
> test_install_default (test_hawqmaster.TestHawqMaster) ... ok
> test_remove_hawq_standby (test_hawqmaster.TestHawqMaster)
> Run custom command Remove HAWQ Standby ... 2016-07-19 12:12:33,657 - Removing 
> HAWQ Standby Master ...
> ok
> test_resync_hawq_standby (test_hawqmaster.TestHawqMaster)
> Run custom command Resync HAWQ Standby ... 2016-07-19 12:12:33,660 - HAWQ 
> Standby Master Re-Sync started in fast mode...
> ok
> test_run_hawq_check_case1 (test_hawqmaster.TestHawqMaster)
> Running HAWQ Check Case 1: Non HDFS-HA, Standalone Resource Management, Not 
> Kerberized ... 2016-07-19 12:12:33,663 - Executing

Re: Review Request 50198: HAWQ fails to execute queries in low-memory environment

2016-07-19 Thread Matt 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50198/
---

(Updated July 19, 2016, 4:34 p.m.)


Review request for Ambari, Alexander Denissov and Lav Jain.


Bugs: AMBARI-17720
https://issues.apache.org/jira/browse/AMBARI-17720


Repository: ambari


Description
---

When a user is running on vagrant or single node with low system memory, 
configurations are not sufficient to allocate containers for "default" 6 vsegs 
per node especially during HASH dist table query or external table query.

Stack advisor should adjust the config values to ensure service checks and 
queries (including Random tables, HASH dist tables, PXF ext tables) do not fail.


Diffs (updated)
-

  
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/configuration/hawq-site.xml
 0bddb07 
  
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py 
dffe57d 
  ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
4089b75 
  ambari-server/src/test/python/common-services/HAWQ/test_service_advisor.py 
f8a9468 
  ambari-server/src/test/python/common-services/PXF/test_service_advisor.py 
7510e5f 

Diff: https://reviews.apache.org/r/50198/diff/


Testing
---

Tested on Live cluster

Added unit tests:
```
test_hawq_master_critical 
(test_alert_component_status.TestAlertComponentStatus) ... ok
test_hawq_master_ok (test_alert_component_status.TestAlertComponentStatus) ... 
ok
test_hawq_segment_critical 
(test_alert_component_status.TestAlertComponentStatus) ... ok
test_hawq_segment_ok (test_alert_component_status.TestAlertComponentStatus) ... 
ok
test_hawq_standby_critical 
(test_alert_component_status.TestAlertComponentStatus) ... ok
test_hawq_standby_ok (test_alert_component_status.TestAlertComponentStatus) ... 
ok
test_missing_configs (test_alert_component_status.TestAlertComponentStatus) ... 
ok
test_exception_registration_status 
(test_alert_segment_registration_status.TestAlertRegistrationStatus) ... No 
handlers could be found for logger "ambari_alerts"
ok
test_missing_configs 
(test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
test_missing_slave_file 
(test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
test_successful_registration_status 
(test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
test_unsuccessful_empty_db_registration_status 
(test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
test_unsuccessful_registration_status 
(test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
test_unsuccessful_registration_status_plural 
(test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
test_missing_configs (test_alert_sync_status.TestAlertSyncStatus) ... ok
test_no_standby_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
test_none_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
test_not_configured_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
test_not_synchronized_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
test_synchronized_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
test_synchronizing_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
test_unknown_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
test_configure_default (test_hawqmaster.TestHawqMaster) ... ok
test_install_default (test_hawqmaster.TestHawqMaster) ... ok
test_remove_hawq_standby (test_hawqmaster.TestHawqMaster)
Run custom command Remove HAWQ Standby ... 2016-07-19 12:12:33,657 - Removing 
HAWQ Standby Master ...
ok
test_resync_hawq_standby (test_hawqmaster.TestHawqMaster)
Run custom command Resync HAWQ Standby ... 2016-07-19 12:12:33,660 - HAWQ 
Standby Master Re-Sync started in fast mode...
ok
test_run_hawq_check_case1 (test_hawqmaster.TestHawqMaster)
Running HAWQ Check Case 1: Non HDFS-HA, Standalone Resource Management, Not 
Kerberized ... 2016-07-19 12:12:33,663 - Executing HAWQ Check ...
ok
test_run_hawq_check_case10 (test_hawqmaster.TestHawqMaster)
Running HAWQ Check Case 10: HDFS-HA, YARN Resource Management Non YARN_HA, 
Kerberized ... 2016-07-19 12:12:33,669 - Executing HAWQ Check ...
ok
test_run_hawq_check_case11 (test_hawqmaster.TestHawqMaster)
Running HAWQ Check Case 11: HDFS-HA, YARN Resource Management YARN_HA, Not 
Kerberized ... 2016-07-19 12:12:33,674 - Executing HAWQ Check ...
ok
test_run_hawq_check_case12 (test_hawqmaster.TestHawqMaster)
Running HAWQ Check Case 12: HDFS-HA, YARN Resource Management YARN_HA, 
Kerberized ... 2016-07-19 12:12:33,680 - Executing HAWQ Check ...
ok
test_run_hawq_check_case2 (test_hawqmaster.TestHawqMaster)
Running HAWQ Check Case 2: Non HDFS-HA, Standalone Resource Management, 
Kerberized ... 2016-07-19 12:12:33,686 - Executing HAWQ Check ...
ok
test_run_hawq_check_case3

Re: Review Request 50198: HAWQ fails to execute queries in low-memory environment

2016-07-19 Thread Lav Jain 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50198/#review142847
---




ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py 
(line 392)


Should config-name be set to hawq_rm_memory_limit_perseg?



ambari-server/src/test/python/common-services/HAWQ/test_service_advisor.py 
(line 378)


Should line 378 be hosts["items"][2] instead of hosts["items"][3]? Same for 
line 390.


- Lav Jain


On July 19, 2016, 7:15 p.m., Matt wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50198/
> ---
> 
> (Updated July 19, 2016, 7:15 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov and Lav Jain.
> 
> 
> Bugs: AMBARI-17720
> https://issues.apache.org/jira/browse/AMBARI-17720
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When a user is running on vagrant or single node with low system memory, 
> configurations are not sufficient to allocate containers for "default" 6 
> vsegs per node especially during HASH dist table query or external table 
> query.
> 
> Stack advisor should adjust the config values to ensure service checks and 
> queries (including Random tables, HASH dist tables, PXF ext tables) do not 
> fail.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/configuration/hawq-site.xml
>  0bddb07 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  dffe57d 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> 4089b75 
>   ambari-server/src/test/python/common-services/HAWQ/test_service_advisor.py 
> f8a9468 
>   ambari-server/src/test/python/common-services/PXF/test_service_advisor.py 
> 7510e5f 
> 
> Diff: https://reviews.apache.org/r/50198/diff/
> 
> 
> Testing
> ---
> 
> Tested on Live cluster
> 
> Added unit tests:
> ```
> test_hawq_master_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_master_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_hawq_segment_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_segment_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_hawq_standby_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_standby_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_missing_configs (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_exception_registration_status 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... No 
> handlers could be found for logger "ambari_alerts"
> ok
> test_missing_configs 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_missing_slave_file 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_successful_registration_status 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_unsuccessful_empty_db_registration_status 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_unsuccessful_registration_status 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_unsuccessful_registration_status_plural 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_missing_configs (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_no_standby_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_none_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_not_configured_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_not_synchronized_state (test_alert_sync_status.TestAlertSyncStatus) ... 
> ok
> test_synchronized_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_synchronizing_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_unknown_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_configure_default (test_hawqmaster.TestHawqMaster) ... ok
> test_install_default (test_hawqmaster.TestHawqMaster) ... ok
> test_remove_hawq_standby (test_hawqmaster.TestHawqMaster)
> Run custom command Remove HAWQ Standby ... 2016-07-19 12:12:33,657 - Removing 
> HAWQ Standby Master ...
> ok
> test_resync_hawq_standby (test_hawqmaster.TestHawqMaster)
> Run custom command Resync HAWQ Standby ... 2016-07-19 12:12:33,660 - HAWQ 
> Standby Master Re-Sync started in fast mode...
> ok
> test_run_hawq_check_case1 (test_hawqmaster.TestHawqMaster)
> Running HAWQ Check Case 1: Non HDFS-HA, Standalone Resource

Re: Review Request 50198: HAWQ fails to execute queries in low-memory environment

2016-07-19 Thread Lav Jain 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50198/#review142849
---


Ship it!




Ship It!

- Lav Jain


On July 19, 2016, 7:15 p.m., Matt wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50198/
> ---
> 
> (Updated July 19, 2016, 7:15 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov and Lav Jain.
> 
> 
> Bugs: AMBARI-17720
> https://issues.apache.org/jira/browse/AMBARI-17720
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When a user is running on vagrant or single node with low system memory, 
> configurations are not sufficient to allocate containers for "default" 6 
> vsegs per node especially during HASH dist table query or external table 
> query.
> 
> Stack advisor should adjust the config values to ensure service checks and 
> queries (including Random tables, HASH dist tables, PXF ext tables) do not 
> fail.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/configuration/hawq-site.xml
>  0bddb07 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  dffe57d 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> 4089b75 
>   ambari-server/src/test/python/common-services/HAWQ/test_service_advisor.py 
> f8a9468 
>   ambari-server/src/test/python/common-services/PXF/test_service_advisor.py 
> 7510e5f 
> 
> Diff: https://reviews.apache.org/r/50198/diff/
> 
> 
> Testing
> ---
> 
> Tested on Live cluster
> 
> Added unit tests:
> ```
> test_hawq_master_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_master_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_hawq_segment_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_segment_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_hawq_standby_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_standby_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_missing_configs (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_exception_registration_status 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... No 
> handlers could be found for logger "ambari_alerts"
> ok
> test_missing_configs 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_missing_slave_file 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_successful_registration_status 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_unsuccessful_empty_db_registration_status 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_unsuccessful_registration_status 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_unsuccessful_registration_status_plural 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_missing_configs (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_no_standby_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_none_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_not_configured_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_not_synchronized_state (test_alert_sync_status.TestAlertSyncStatus) ... 
> ok
> test_synchronized_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_synchronizing_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_unknown_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_configure_default (test_hawqmaster.TestHawqMaster) ... ok
> test_install_default (test_hawqmaster.TestHawqMaster) ... ok
> test_remove_hawq_standby (test_hawqmaster.TestHawqMaster)
> Run custom command Remove HAWQ Standby ... 2016-07-19 12:12:33,657 - Removing 
> HAWQ Standby Master ...
> ok
> test_resync_hawq_standby (test_hawqmaster.TestHawqMaster)
> Run custom command Resync HAWQ Standby ... 2016-07-19 12:12:33,660 - HAWQ 
> Standby Master Re-Sync started in fast mode...
> ok
> test_run_hawq_check_case1 (test_hawqmaster.TestHawqMaster)
> Running HAWQ Check Case 1: Non HDFS-HA, Standalone Resource Management, Not 
> Kerberized ... 2016-07-19 12:12:33,663 - Executing HAWQ Check ...
> ok
> test_run_hawq_check_case10 (test_hawqmaster.TestHawqMaster)
> Running HAWQ Check Case 10: HDFS-HA, YARN Resource Management Non YARN_HA, 
> Kerberized ... 2016-07-19 12:12:33,669 - Executing HAWQ Check ...
> ok
> test_run_hawq_check_case11 (test_hawqmaster.TestHawqMaster)
> Running HAWQ Check Case 11: HDFS-HA, YARN Resource Management

Re: Review Request 50198: HAWQ fails to execute queries in low-memory environment

2016-07-19 Thread Alexander Denissov 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50198/#review142834
---


Ship it!




Ship It!

- Alexander Denissov


On July 19, 2016, 7:15 p.m., Matt wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50198/
> ---
> 
> (Updated July 19, 2016, 7:15 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov and Lav Jain.
> 
> 
> Bugs: AMBARI-17720
> https://issues.apache.org/jira/browse/AMBARI-17720
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When a user is running on vagrant or single node with low system memory, 
> configurations are not sufficient to allocate containers for "default" 6 
> vsegs per node especially during HASH dist table query or external table 
> query.
> 
> Stack advisor should adjust the config values to ensure service checks and 
> queries (including Random tables, HASH dist tables, PXF ext tables) do not 
> fail.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/configuration/hawq-site.xml
>  0bddb07 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  dffe57d 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> 4089b75 
>   ambari-server/src/test/python/common-services/HAWQ/test_service_advisor.py 
> f8a9468 
>   ambari-server/src/test/python/common-services/PXF/test_service_advisor.py 
> 7510e5f 
> 
> Diff: https://reviews.apache.org/r/50198/diff/
> 
> 
> Testing
> ---
> 
> Tested on Live cluster
> 
> Added unit tests:
> ```
> test_hawq_master_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_master_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_hawq_segment_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_segment_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_hawq_standby_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_standby_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_missing_configs (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_exception_registration_status 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... No 
> handlers could be found for logger "ambari_alerts"
> ok
> test_missing_configs 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_missing_slave_file 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_successful_registration_status 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_unsuccessful_empty_db_registration_status 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_unsuccessful_registration_status 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_unsuccessful_registration_status_plural 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_missing_configs (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_no_standby_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_none_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_not_configured_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_not_synchronized_state (test_alert_sync_status.TestAlertSyncStatus) ... 
> ok
> test_synchronized_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_synchronizing_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_unknown_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_configure_default (test_hawqmaster.TestHawqMaster) ... ok
> test_install_default (test_hawqmaster.TestHawqMaster) ... ok
> test_remove_hawq_standby (test_hawqmaster.TestHawqMaster)
> Run custom command Remove HAWQ Standby ... 2016-07-19 12:12:33,657 - Removing 
> HAWQ Standby Master ...
> ok
> test_resync_hawq_standby (test_hawqmaster.TestHawqMaster)
> Run custom command Resync HAWQ Standby ... 2016-07-19 12:12:33,660 - HAWQ 
> Standby Master Re-Sync started in fast mode...
> ok
> test_run_hawq_check_case1 (test_hawqmaster.TestHawqMaster)
> Running HAWQ Check Case 1: Non HDFS-HA, Standalone Resource Management, Not 
> Kerberized ... 2016-07-19 12:12:33,663 - Executing HAWQ Check ...
> ok
> test_run_hawq_check_case10 (test_hawqmaster.TestHawqMaster)
> Running HAWQ Check Case 10: HDFS-HA, YARN Resource Management Non YARN_HA, 
> Kerberized ... 2016-07-19 12:12:33,669 - Executing HAWQ Check ...
> ok
> test_run_hawq_check_case11 (test_hawqmaster.TestHawqMaster)
> Running HAWQ Check Case 11: HDFS-HA, YARN Resource

Re: Review Request 50198: HAWQ fails to execute queries in low-memory environment

2016-07-19 Thread Alexander Denissov 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50198/#review142831
---




ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py 
(line 213)


this property has no impact on the logic inside this if, maybe move it out 
of the if statement ?


- Alexander Denissov


On July 19, 2016, 7:15 p.m., Matt wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50198/
> ---
> 
> (Updated July 19, 2016, 7:15 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov and Lav Jain.
> 
> 
> Bugs: AMBARI-17720
> https://issues.apache.org/jira/browse/AMBARI-17720
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When a user is running on vagrant or single node with low system memory, 
> configurations are not sufficient to allocate containers for "default" 6 
> vsegs per node especially during HASH dist table query or external table 
> query.
> 
> Stack advisor should adjust the config values to ensure service checks and 
> queries (including Random tables, HASH dist tables, PXF ext tables) do not 
> fail.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/configuration/hawq-site.xml
>  0bddb07 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  dffe57d 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> 4089b75 
>   ambari-server/src/test/python/common-services/HAWQ/test_service_advisor.py 
> f8a9468 
>   ambari-server/src/test/python/common-services/PXF/test_service_advisor.py 
> 7510e5f 
> 
> Diff: https://reviews.apache.org/r/50198/diff/
> 
> 
> Testing
> ---
> 
> Tested on Live cluster
> 
> Added unit tests:
> ```
> test_hawq_master_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_master_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_hawq_segment_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_segment_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_hawq_standby_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_standby_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_missing_configs (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_exception_registration_status 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... No 
> handlers could be found for logger "ambari_alerts"
> ok
> test_missing_configs 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_missing_slave_file 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_successful_registration_status 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_unsuccessful_empty_db_registration_status 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_unsuccessful_registration_status 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_unsuccessful_registration_status_plural 
> (test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
> test_missing_configs (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_no_standby_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_none_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_not_configured_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_not_synchronized_state (test_alert_sync_status.TestAlertSyncStatus) ... 
> ok
> test_synchronized_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_synchronizing_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_unknown_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
> test_configure_default (test_hawqmaster.TestHawqMaster) ... ok
> test_install_default (test_hawqmaster.TestHawqMaster) ... ok
> test_remove_hawq_standby (test_hawqmaster.TestHawqMaster)
> Run custom command Remove HAWQ Standby ... 2016-07-19 12:12:33,657 - Removing 
> HAWQ Standby Master ...
> ok
> test_resync_hawq_standby (test_hawqmaster.TestHawqMaster)
> Run custom command Resync HAWQ Standby ... 2016-07-19 12:12:33,660 - HAWQ 
> Standby Master Re-Sync started in fast mode...
> ok
> test_run_hawq_check_case1 (test_hawqmaster.TestHawqMaster)
> Running HAWQ Check Case 1: Non HDFS-HA, Standalone Resource Management, Not 
> Kerberized ... 2016-07-19 12:12:33,663 - Executing HAWQ Check ...
> ok
> test_run_hawq_check_case10 (test_hawqmaster.TestHawqMaster)
> Running HAWQ Check Case 10: HDFS-HA,

Re: Review Request 50198: HAWQ fails to execute queries in low-memory environment

2016-07-19 Thread Matt 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50198/
---

(Updated July 19, 2016, 12:15 p.m.)


Review request for Ambari, Alexander Denissov and Lav Jain.


Changes
---

Remove unused import


Bugs: AMBARI-17720
https://issues.apache.org/jira/browse/AMBARI-17720


Repository: ambari


Description
---

When a user is running on vagrant or single node with low system memory, 
configurations are not sufficient to allocate containers for "default" 6 vsegs 
per node especially during HASH dist table query or external table query.

Stack advisor should adjust the config values to ensure service checks and 
queries (including Random tables, HASH dist tables, PXF ext tables) do not fail.


Diffs (updated)
-

  
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/configuration/hawq-site.xml
 0bddb07 
  
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py 
dffe57d 
  ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
4089b75 
  ambari-server/src/test/python/common-services/HAWQ/test_service_advisor.py 
f8a9468 
  ambari-server/src/test/python/common-services/PXF/test_service_advisor.py 
7510e5f 

Diff: https://reviews.apache.org/r/50198/diff/


Testing
---

Tested on Live cluster

Added unit tests:
```
test_hawq_master_critical 
(test_alert_component_status.TestAlertComponentStatus) ... ok
test_hawq_master_ok (test_alert_component_status.TestAlertComponentStatus) ... 
ok
test_hawq_segment_critical 
(test_alert_component_status.TestAlertComponentStatus) ... ok
test_hawq_segment_ok (test_alert_component_status.TestAlertComponentStatus) ... 
ok
test_hawq_standby_critical 
(test_alert_component_status.TestAlertComponentStatus) ... ok
test_hawq_standby_ok (test_alert_component_status.TestAlertComponentStatus) ... 
ok
test_missing_configs (test_alert_component_status.TestAlertComponentStatus) ... 
ok
test_exception_registration_status 
(test_alert_segment_registration_status.TestAlertRegistrationStatus) ... No 
handlers could be found for logger "ambari_alerts"
ok
test_missing_configs 
(test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
test_missing_slave_file 
(test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
test_successful_registration_status 
(test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
test_unsuccessful_empty_db_registration_status 
(test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
test_unsuccessful_registration_status 
(test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
test_unsuccessful_registration_status_plural 
(test_alert_segment_registration_status.TestAlertRegistrationStatus) ... ok
test_missing_configs (test_alert_sync_status.TestAlertSyncStatus) ... ok
test_no_standby_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
test_none_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
test_not_configured_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
test_not_synchronized_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
test_synchronized_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
test_synchronizing_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
test_unknown_state (test_alert_sync_status.TestAlertSyncStatus) ... ok
test_configure_default (test_hawqmaster.TestHawqMaster) ... ok
test_install_default (test_hawqmaster.TestHawqMaster) ... ok
test_remove_hawq_standby (test_hawqmaster.TestHawqMaster)
Run custom command Remove HAWQ Standby ... 2016-07-19 12:12:33,657 - Removing 
HAWQ Standby Master ...
ok
test_resync_hawq_standby (test_hawqmaster.TestHawqMaster)
Run custom command Resync HAWQ Standby ... 2016-07-19 12:12:33,660 - HAWQ 
Standby Master Re-Sync started in fast mode...
ok
test_run_hawq_check_case1 (test_hawqmaster.TestHawqMaster)
Running HAWQ Check Case 1: Non HDFS-HA, Standalone Resource Management, Not 
Kerberized ... 2016-07-19 12:12:33,663 - Executing HAWQ Check ...
ok
test_run_hawq_check_case10 (test_hawqmaster.TestHawqMaster)
Running HAWQ Check Case 10: HDFS-HA, YARN Resource Management Non YARN_HA, 
Kerberized ... 2016-07-19 12:12:33,669 - Executing HAWQ Check ...
ok
test_run_hawq_check_case11 (test_hawqmaster.TestHawqMaster)
Running HAWQ Check Case 11: HDFS-HA, YARN Resource Management YARN_HA, Not 
Kerberized ... 2016-07-19 12:12:33,674 - Executing HAWQ Check ...
ok
test_run_hawq_check_case12 (test_hawqmaster.TestHawqMaster)
Running HAWQ Check Case 12: HDFS-HA, YARN Resource Management YARN_HA, 
Kerberized ... 2016-07-19 12:12:33,680 - Executing HAWQ Check ...
ok
test_run_hawq_check_case2 (test_hawqmaster.TestHawqMaster)
Running HAWQ Check Case 2: Non HDFS-HA, Standalone Resource Management, 
Kerberized ... 2016-07-19 12:12:33,686 - Executing HAWQ Check ...
ok