Re: Review Request 48643: Ambari uses too small a window for region server shutdown

2016-06-13 Thread Dmitro Lisnichenko

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48643/#review137306
---


Ship it!




Ship It!

- Dmitro Lisnichenko


On June 13, 2016, 6:21 p.m., Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48643/
> ---
> 
> (Updated June 13, 2016, 6:21 p.m.)
> 
> 
> Review request for Ambari and Dmitro Lisnichenko.
> 
> 
> Bugs: AMBARI-16914
> https://issues.apache.org/jira/browse/AMBARI-16914
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Ambari seems to issue a formal shutdown to a Region server but quickly (30
> seconds) follows it up with SIGKILL. On a full loaded HBase system with about
> 200 regions per region server and active transaction flow, there is no way a
> RS can stop in 30 seconds. This has caused many issues in production including
> a memstore corruption. Why not use the shutdown script that comes with HBase?
> 
> 
> 
> 2016-05-24 15:36:19,191 - 
> Execute['/usr/hdp/current/hbase-regionserver/bin/hbase-daemon.sh --config 
> /usr/hdp/current/hbase-regionserver/conf stop regionserver']
> {'only_if': 'ambari-sudo.sh -H -E test -f 
> /var/run/hbase/hbase-hbase-regionserver.pid && ps -p `ambari-sudo.sh -H -E 
> cat /var/run/hbase/hbase-hbase-regionserver.pid` >/dev/null 2>&1', 
> 'on_timeout': '! ( ambari-sudo.sh -H -E test -f 
> /var/run/hbase/hbase-hbase-regionserver.pid && ps -p `ambari-sudo.sh -H -E 
> cat /var/run/hbase/hbase-hbase-regionserver.pid` >/dev/null 2>&1 ) || 
> ambari-sudo.sh -H -E kill -9 `ambari-sudo.sh -H -E cat 
> /var/run/hbase/hbase-hbase-regionserver.pid`', 'timeout': 30, 'user': 'hbase'}
> 
> 2016-05-24 15:36:50,982 - Executing '! ( ambari-sudo.sh -H -E test -f 
> /var/run/hbase/hbase-hbase-regionserver.pid && ps -p `ambari-sudo.sh -H -E 
> cat /var/run/hbase/hbase-hbase-regionserver.pid` >/dev/null 2>&1 ) || 
> ambari-sudo.sh -H -E kill -9 `ambari-sudo.sh -H -E cat 
> /var/run/hbase/hbase-hbase-regionserver.pid`'. Reason: Execution of 
> 'ambari-sudo.sh su hbase -l -s /bin/bash -c 'export 
> PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent'"'"'
>  ; /usr/hdp/current/hbase-regionserver/bin/hbase-daemon.sh --config 
> /usr/hdp/current/hbase-regionserver/conf stop regionserver'' was killed due 
> timeout after 30 seconds
> 2016-05-24 15:36:51,053 - 
> File['/var/run/hbase/hbase-hbase-regionserver.pid']
> {'action': ['delete']}
> 
> 2016-05-24 15:36:51,054 - Deleting 
> File['/var/run/hbase/hbase-hbase-regionserver.pid'
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-env.xml
>  b40923a 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/hbase_service.py
>  4d0d7f3 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params_linux.py
>  21b491d 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/configuration/hbase-env.xml
>  eaee3cf 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_service.py
>  a6904f6 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py
>  05bad1c 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/phoenix_service.py
>  0a42cda 
>   
> ambari-server/src/test/python/stacks/2.0.6/HBASE/test_phoenix_queryserver.py 
> ac8b153 
>   ambari-server/src/test/python/stacks/2.0.6/configs/default.json 04aa828 
>   ambari-server/src/test/python/stacks/2.0.6/configs/secured.json 02f982e 
> 
> Diff: https://reviews.apache.org/r/48643/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>



Review Request 48643: Ambari uses too small a window for region server shutdown

2016-06-13 Thread Andrew Onischuk

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48643/
---

Review request for Ambari and Dmitro Lisnichenko.


Bugs: AMBARI-16914
https://issues.apache.org/jira/browse/AMBARI-16914


Repository: ambari


Description
---

Ambari seems to issue a formal shutdown to a Region server but quickly (30
seconds) follows it up with SIGKILL. On a full loaded HBase system with about
200 regions per region server and active transaction flow, there is no way a
RS can stop in 30 seconds. This has caused many issues in production including
a memstore corruption. Why not use the shutdown script that comes with HBase?



2016-05-24 15:36:19,191 - 
Execute['/usr/hdp/current/hbase-regionserver/bin/hbase-daemon.sh --config 
/usr/hdp/current/hbase-regionserver/conf stop regionserver']
{'only_if': 'ambari-sudo.sh -H -E test -f 
/var/run/hbase/hbase-hbase-regionserver.pid && ps -p `ambari-sudo.sh -H -E cat 
/var/run/hbase/hbase-hbase-regionserver.pid` >/dev/null 2>&1', 'on_timeout': '! 
( ambari-sudo.sh -H -E test -f /var/run/hbase/hbase-hbase-regionserver.pid && 
ps -p `ambari-sudo.sh -H -E cat /var/run/hbase/hbase-hbase-regionserver.pid` 
>/dev/null 2>&1 ) || ambari-sudo.sh -H -E kill -9 `ambari-sudo.sh -H -E cat 
/var/run/hbase/hbase-hbase-regionserver.pid`', 'timeout': 30, 'user': 'hbase'}

2016-05-24 15:36:50,982 - Executing '! ( ambari-sudo.sh -H -E test -f 
/var/run/hbase/hbase-hbase-regionserver.pid && ps -p `ambari-sudo.sh -H -E cat 
/var/run/hbase/hbase-hbase-regionserver.pid` >/dev/null 2>&1 ) || 
ambari-sudo.sh -H -E kill -9 `ambari-sudo.sh -H -E cat 
/var/run/hbase/hbase-hbase-regionserver.pid`'. Reason: Execution of 
'ambari-sudo.sh su hbase -l -s /bin/bash -c 'export 
PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent'"'"'
 ; /usr/hdp/current/hbase-regionserver/bin/hbase-daemon.sh --config 
/usr/hdp/current/hbase-regionserver/conf stop regionserver'' was killed due 
timeout after 30 seconds
2016-05-24 15:36:51,053 - 
File['/var/run/hbase/hbase-hbase-regionserver.pid']
{'action': ['delete']}

2016-05-24 15:36:51,054 - Deleting 
File['/var/run/hbase/hbase-hbase-regionserver.pid'


Diffs
-

  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-env.xml
 b40923a 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/hbase_service.py
 4d0d7f3 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params_linux.py
 21b491d 
  
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/configuration/hbase-env.xml
 eaee3cf 
  
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_service.py
 a6904f6 
  
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py
 05bad1c 
  
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/phoenix_service.py
 0a42cda 
  ambari-server/src/test/python/stacks/2.0.6/HBASE/test_phoenix_queryserver.py 
ac8b153 
  ambari-server/src/test/python/stacks/2.0.6/configs/default.json 04aa828 
  ambari-server/src/test/python/stacks/2.0.6/configs/secured.json 02f982e 

Diff: https://reviews.apache.org/r/48643/diff/


Testing
---

mvn clean test


Thanks,

Andrew Onischuk