Re: Review Request 51724: Add Kerberos HTTP SPNEGO authentication support to Ambari Metrics Monitor

2016-10-24 Thread Qin Liu 


> On Sept. 22, 2016, 7:54 p.m., Robert Levas wrote:
> > ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/spnego_kerberos_auth.py,
> >  line 33
> > 
> >
> > See `curl_krb_request`
> 
> Dmytro Sen wrote:
> I think, we don't want to do a separate HTTP connection for every call, 
> now metrics monitor use the same cached connection + only GET is implemented 
> in current curl_krb_request, we need POST for emmitting metrics.

Hi Dmytro, thanks for jumping in to move it forward!


- Qin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51724/#review150057
---


On Sept. 22, 2016, 12:44 p.m., Qin Liu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51724/
> ---
> 
> (Updated Sept. 22, 2016, 12:44 p.m.)
> 
> 
> Review request for Ambari, Di Li, Dmytro Sen, and Sid Wagle.
> 
> 
> Bugs: AMBARI-17898
> https://issues.apache.org/jira/browse/AMBARI-17898
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> ()This is a subtask of AMBARI-14384 "Ambari Metrics doesn't use SPNEGO to 
> authenticate".
> 
> In a Kerberos enabled cluster with SPNEGO enabled on Hadoop APIs, Ambari 
> Metrics Collector (in AMS distributed mode) web-console will be Kerberos HTTP 
> SPNEGO enabled too. But Ambari Metrics Monitor, a client of Ambari Metrics 
> Collector, currently does not support Kerberos HTTP SPNEGO authentication.
> 
> /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out:
> 2015-12-15 13:26:30,663 [INFO] emitter.py:101 - server: 
> http://metrics-collector:6188/ws/v1/timeline/metrics
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:84 - Error sending metrics to 
> server. HTTP Error 401: Authentication required
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:90 - Retrying after 5 ...
> 
> 
> Diffs
> -
> 
>   ambari-metrics/ambari-metrics-assembly/pom.xml d73a0af 
>   ambari-metrics/ambari-metrics-host-monitoring/conf/unix/metric_monitor.ini 
> e98c65c 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/config_reader.py
>  3ca3a31 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/emitter.py 
> 050af16 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/faked_kerberos.py
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/spnego_kerberos_auth.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/kerberos.json
>  51f541f 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
>  6934924 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/templates/metric_monitor.ini.j2
>  3823912 
> 
> Diff: https://reviews.apache.org/r/51724/diff/
> 
> 
> Testing
> ---
> 
> 1. passed existing unittest cases.
> 2. manually tested with HDP-2.5.0.0/trunk 
>1) Non http spnego enabled clusters
>2) http spnego enabled cluster - tested the trunk's monitor code against a 
> http spnego enabled cluster 
>   http spnego enabled cluster is currently broken in trunk: the 
> distributed metrics collector's http server currently will not started with 
> http spnego enabled. so I had to manually test the trunk's monitor code with 
> http spnego enabled collector on a branch2.4 cluster. Need to mention that 
> the testing should be valid because the code change area is monitor code and 
> the trunk and branch2.4's collectors are same.
> 
> 
> Thanks,
> 
> Qin Liu
> 
>

Re: Review Request 51724: Add Kerberos HTTP SPNEGO authentication support to Ambari Metrics Monitor

2016-10-12 Thread Dmytro Sen 


> On Сен. 22, 2016, 7:54 п.п., Robert Levas wrote:
> > ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/spnego_kerberos_auth.py,
> >  line 33
> > 
> >
> > See `curl_krb_request`

I think, we don't want to do a separate HTTP connection for every call, now 
metrics monitor use the same cached connection + only GET is implemented in 
current curl_krb_request, we need POST for emmitting metrics.


- Dmytro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51724/#review150057
---


On Сен. 22, 2016, 12:44 п.п., Qin Liu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51724/
> ---
> 
> (Updated Сен. 22, 2016, 12:44 п.п.)
> 
> 
> Review request for Ambari, Di Li, Dmytro Sen, and Sid Wagle.
> 
> 
> Bugs: AMBARI-17898
> https://issues.apache.org/jira/browse/AMBARI-17898
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> ()This is a subtask of AMBARI-14384 "Ambari Metrics doesn't use SPNEGO to 
> authenticate".
> 
> In a Kerberos enabled cluster with SPNEGO enabled on Hadoop APIs, Ambari 
> Metrics Collector (in AMS distributed mode) web-console will be Kerberos HTTP 
> SPNEGO enabled too. But Ambari Metrics Monitor, a client of Ambari Metrics 
> Collector, currently does not support Kerberos HTTP SPNEGO authentication.
> 
> /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out:
> 2015-12-15 13:26:30,663 [INFO] emitter.py:101 - server: 
> http://metrics-collector:6188/ws/v1/timeline/metrics
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:84 - Error sending metrics to 
> server. HTTP Error 401: Authentication required
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:90 - Retrying after 5 ...
> 
> 
> Diffs
> -
> 
>   ambari-metrics/ambari-metrics-assembly/pom.xml d73a0af 
>   ambari-metrics/ambari-metrics-host-monitoring/conf/unix/metric_monitor.ini 
> e98c65c 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/config_reader.py
>  3ca3a31 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/emitter.py 
> 050af16 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/faked_kerberos.py
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/spnego_kerberos_auth.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/kerberos.json
>  51f541f 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
>  6934924 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/templates/metric_monitor.ini.j2
>  3823912 
> 
> Diff: https://reviews.apache.org/r/51724/diff/
> 
> 
> Testing
> ---
> 
> 1. passed existing unittest cases.
> 2. manually tested with HDP-2.5.0.0/trunk 
>1) Non http spnego enabled clusters
>2) http spnego enabled cluster - tested the trunk's monitor code against a 
> http spnego enabled cluster 
>   http spnego enabled cluster is currently broken in trunk: the 
> distributed metrics collector's http server currently will not started with 
> http spnego enabled. so I had to manually test the trunk's monitor code with 
> http spnego enabled collector on a branch2.4 cluster. Need to mention that 
> the testing should be valid because the code change area is monitor code and 
> the trunk and branch2.4's collectors are same.
> 
> 
> Thanks,
> 
> Qin Liu
> 
>

Re: Review Request 51724: Add Kerberos HTTP SPNEGO authentication support to Ambari Metrics Monitor

2016-09-23 Thread Qin Liu 


> On Sept. 8, 2016, 6:13 p.m., Sid Wagle wrote:
> > ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/spnego_kerberos_auth.py,
> >  line 25
> > 
> >
> > I do not see a "import kerberos" anywhere in Ambari deployed cluster 
> > under: /usr/lib/python2.6/site-packages/
> > 
> > Where does this dependency get fulfiled ?
> 
> Qin Liu wrote:
> It is in /usr/lib/python2.6/site-packages/ipalib/plugins on all of my 
> clusters. I will include ipa-python in ambari-metrics-assembly/pom.xml. 
> Thanks Sid!
> 
> Sid Wagle wrote:
> This seems to be ipa only dependency, does addinig it as component build 
> dep make sense?
> What about MIT kereberos?
> 
> Qin Liu wrote:
> Hi Sid,
> Yes, ipa is the only dependency for metrics monitor code. I think it 
> makes sense to include ipa in ambari-metrics component build deployment.
> MIT kerberos is outside of this JIRA.
> 
> Sid Wagle wrote:
> In that case we should also log error on the import if module not found. 
> Additionally, open a Jira for getting this to work with MIT Kerberos so 
> someone can take it up. As far as this patch is concerened we are good 
> contingent on the small logging change and making sure we have a Jira to 
> track MIT Kerberos won't work the same way.
> 
> Qin Liu wrote:
> Hi Sid,
> I updated the code to log error on the import if kerberos module is not 
> found. On a deployed cluster, the exception should never occur because 
> kerberos module should always be included in ipa lib. The gss API in kerberos 
> module is used to authenticate using kerberos. The kerberos to be used is MIT 
> kerberos because an existing MIT KDC should be chosen at cluster kerberos 
> enabling.

Hi Sid, 
Forgot to mention that there is no extra work required on the AMS side to have 
things work with MIT kerberos because MIT kerberos already exists. To enable 
MIT kerberos on a cluster, just need to install MIT KDC on a host that can be 
reached by the ambari server.


- Qin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51724/#review148224
---


On Sept. 22, 2016, 12:44 p.m., Qin Liu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51724/
> ---
> 
> (Updated Sept. 22, 2016, 12:44 p.m.)
> 
> 
> Review request for Ambari, Di Li, Dmytro Sen, and Sid Wagle.
> 
> 
> Bugs: AMBARI-17898
> https://issues.apache.org/jira/browse/AMBARI-17898
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> ()This is a subtask of AMBARI-14384 "Ambari Metrics doesn't use SPNEGO to 
> authenticate".
> 
> In a Kerberos enabled cluster with SPNEGO enabled on Hadoop APIs, Ambari 
> Metrics Collector (in AMS distributed mode) web-console will be Kerberos HTTP 
> SPNEGO enabled too. But Ambari Metrics Monitor, a client of Ambari Metrics 
> Collector, currently does not support Kerberos HTTP SPNEGO authentication.
> 
> /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out:
> 2015-12-15 13:26:30,663 [INFO] emitter.py:101 - server: 
> http://metrics-collector:6188/ws/v1/timeline/metrics
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:84 - Error sending metrics to 
> server. HTTP Error 401: Authentication required
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:90 - Retrying after 5 ...
> 
> 
> Diffs
> -
> 
>   ambari-metrics/ambari-metrics-assembly/pom.xml d73a0af 
>   ambari-metrics/ambari-metrics-host-monitoring/conf/unix/metric_monitor.ini 
> e98c65c 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/config_reader.py
>  3ca3a31 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/emitter.py 
> 050af16 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/faked_kerberos.py
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/spnego_kerberos_auth.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/kerberos.json
>  51f541f 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
>  6934924 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/templates/metric_monitor.ini.j2
>  3823912 
> 
> Diff: https://reviews.apache.org/r/51724/diff/
> 
> 
> Testing
> ---
> 
> 1. passed existing unittest cases.
> 2. manually tested with HDP-2.5.0.0/trunk 
>1) Non http spnego enabled clusters
>2) http spnego enabled cluster - tested the trunk's monitor code against a 
> http spnego enabled cluster 
>   http spnego enabled cluster is currently broken in trunk: the 
> distributed metrics

Re: Review Request 51724: Add Kerberos HTTP SPNEGO authentication support to Ambari Metrics Monitor

2016-09-22 Thread Qin Liu 


> On Sept. 22, 2016, 7:54 p.m., Robert Levas wrote:
> > ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/emitter.py,
> >  line 121
> > 
> >
> > Possibly use `curl_krb_request` instead.
> 
> Di Li wrote:
> Qin, please look at 'from 
> resource_management.libraries.functions.curl_krb_request import 
> curl_krb_request', see if you can reuse APIs from the existing Ambari lib

Sure, will investigate it. Thanks Robert and Di!


> On Sept. 22, 2016, 7:54 p.m., Robert Levas wrote:
> > ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/faked_kerberos.py,
> >  line 26
> > 
> >
> > The file name should be `krberr.py` rather than `faked_kerberos.py`

Hi Robert, will fix it. Thanks!


- Qin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51724/#review150057
---


On Sept. 22, 2016, 12:44 p.m., Qin Liu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51724/
> ---
> 
> (Updated Sept. 22, 2016, 12:44 p.m.)
> 
> 
> Review request for Ambari, Di Li, Dmytro Sen, and Sid Wagle.
> 
> 
> Bugs: AMBARI-17898
> https://issues.apache.org/jira/browse/AMBARI-17898
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> ()This is a subtask of AMBARI-14384 "Ambari Metrics doesn't use SPNEGO to 
> authenticate".
> 
> In a Kerberos enabled cluster with SPNEGO enabled on Hadoop APIs, Ambari 
> Metrics Collector (in AMS distributed mode) web-console will be Kerberos HTTP 
> SPNEGO enabled too. But Ambari Metrics Monitor, a client of Ambari Metrics 
> Collector, currently does not support Kerberos HTTP SPNEGO authentication.
> 
> /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out:
> 2015-12-15 13:26:30,663 [INFO] emitter.py:101 - server: 
> http://metrics-collector:6188/ws/v1/timeline/metrics
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:84 - Error sending metrics to 
> server. HTTP Error 401: Authentication required
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:90 - Retrying after 5 ...
> 
> 
> Diffs
> -
> 
>   ambari-metrics/ambari-metrics-assembly/pom.xml d73a0af 
>   ambari-metrics/ambari-metrics-host-monitoring/conf/unix/metric_monitor.ini 
> e98c65c 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/config_reader.py
>  3ca3a31 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/emitter.py 
> 050af16 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/faked_kerberos.py
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/spnego_kerberos_auth.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/kerberos.json
>  51f541f 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
>  6934924 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/templates/metric_monitor.ini.j2
>  3823912 
> 
> Diff: https://reviews.apache.org/r/51724/diff/
> 
> 
> Testing
> ---
> 
> 1. passed existing unittest cases.
> 2. manually tested with HDP-2.5.0.0/trunk 
>1) Non http spnego enabled clusters
>2) http spnego enabled cluster - tested the trunk's monitor code against a 
> http spnego enabled cluster 
>   http spnego enabled cluster is currently broken in trunk: the 
> distributed metrics collector's http server currently will not started with 
> http spnego enabled. so I had to manually test the trunk's monitor code with 
> http spnego enabled collector on a branch2.4 cluster. Need to mention that 
> the testing should be valid because the code change area is monitor code and 
> the trunk and branch2.4's collectors are same.
> 
> 
> Thanks,
> 
> Qin Liu
> 
>

Re: Review Request 51724: Add Kerberos HTTP SPNEGO authentication support to Ambari Metrics Monitor

2016-09-22 Thread Di Li 


> On Sept. 22, 2016, 7:54 p.m., Robert Levas wrote:
> > ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/emitter.py,
> >  line 121
> > 
> >
> > Possibly use `curl_krb_request` instead.

Qin, please look at 'from 
resource_management.libraries.functions.curl_krb_request import 
curl_krb_request', see if you can reuse APIs from the existing Ambari lib


- Di


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51724/#review150057
---


On Sept. 22, 2016, 12:44 p.m., Qin Liu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51724/
> ---
> 
> (Updated Sept. 22, 2016, 12:44 p.m.)
> 
> 
> Review request for Ambari, Di Li, Dmytro Sen, and Sid Wagle.
> 
> 
> Bugs: AMBARI-17898
> https://issues.apache.org/jira/browse/AMBARI-17898
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> ()This is a subtask of AMBARI-14384 "Ambari Metrics doesn't use SPNEGO to 
> authenticate".
> 
> In a Kerberos enabled cluster with SPNEGO enabled on Hadoop APIs, Ambari 
> Metrics Collector (in AMS distributed mode) web-console will be Kerberos HTTP 
> SPNEGO enabled too. But Ambari Metrics Monitor, a client of Ambari Metrics 
> Collector, currently does not support Kerberos HTTP SPNEGO authentication.
> 
> /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out:
> 2015-12-15 13:26:30,663 [INFO] emitter.py:101 - server: 
> http://metrics-collector:6188/ws/v1/timeline/metrics
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:84 - Error sending metrics to 
> server. HTTP Error 401: Authentication required
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:90 - Retrying after 5 ...
> 
> 
> Diffs
> -
> 
>   ambari-metrics/ambari-metrics-assembly/pom.xml d73a0af 
>   ambari-metrics/ambari-metrics-host-monitoring/conf/unix/metric_monitor.ini 
> e98c65c 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/config_reader.py
>  3ca3a31 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/emitter.py 
> 050af16 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/faked_kerberos.py
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/spnego_kerberos_auth.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/kerberos.json
>  51f541f 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
>  6934924 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/templates/metric_monitor.ini.j2
>  3823912 
> 
> Diff: https://reviews.apache.org/r/51724/diff/
> 
> 
> Testing
> ---
> 
> 1. passed existing unittest cases.
> 2. manually tested with HDP-2.5.0.0/trunk 
>1) Non http spnego enabled clusters
>2) http spnego enabled cluster - tested the trunk's monitor code against a 
> http spnego enabled cluster 
>   http spnego enabled cluster is currently broken in trunk: the 
> distributed metrics collector's http server currently will not started with 
> http spnego enabled. so I had to manually test the trunk's monitor code with 
> http spnego enabled collector on a branch2.4 cluster. Need to mention that 
> the testing should be valid because the code change area is monitor code and 
> the trunk and branch2.4's collectors are same.
> 
> 
> Thanks,
> 
> Qin Liu
> 
>

Re: Review Request 51724: Add Kerberos HTTP SPNEGO authentication support to Ambari Metrics Monitor

2016-09-22 Thread Tim Thorpe 


> On Sept. 22, 2016, 7:54 p.m., Robert Levas wrote:
> > ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/config_reader.py,
> >  lines 119-120
> > 
> >
> > These shoud not be hardcoded. The aths to the tools (kinit and klist), 
> > the principal name, and the keytab file should be found in the config data.

Hi Robert, lines 119 and 120 are both part of the config_content string which 
provides default values for the configuration.  These are only used if the 
config file doesn't exist:

   if os.path.exists(CONFIG_FILE_PATH):
 self.config.read(CONFIG_FILE_PATH)
   else:
 self.config.readfp(StringIO.StringIO(config_content))


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51724/#review150057
---


On Sept. 22, 2016, 12:44 p.m., Qin Liu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51724/
> ---
> 
> (Updated Sept. 22, 2016, 12:44 p.m.)
> 
> 
> Review request for Ambari, Di Li, Dmytro Sen, and Sid Wagle.
> 
> 
> Bugs: AMBARI-17898
> https://issues.apache.org/jira/browse/AMBARI-17898
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> ()This is a subtask of AMBARI-14384 "Ambari Metrics doesn't use SPNEGO to 
> authenticate".
> 
> In a Kerberos enabled cluster with SPNEGO enabled on Hadoop APIs, Ambari 
> Metrics Collector (in AMS distributed mode) web-console will be Kerberos HTTP 
> SPNEGO enabled too. But Ambari Metrics Monitor, a client of Ambari Metrics 
> Collector, currently does not support Kerberos HTTP SPNEGO authentication.
> 
> /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out:
> 2015-12-15 13:26:30,663 [INFO] emitter.py:101 - server: 
> http://metrics-collector:6188/ws/v1/timeline/metrics
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:84 - Error sending metrics to 
> server. HTTP Error 401: Authentication required
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:90 - Retrying after 5 ...
> 
> 
> Diffs
> -
> 
>   ambari-metrics/ambari-metrics-assembly/pom.xml d73a0af 
>   ambari-metrics/ambari-metrics-host-monitoring/conf/unix/metric_monitor.ini 
> e98c65c 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/config_reader.py
>  3ca3a31 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/emitter.py 
> 050af16 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/faked_kerberos.py
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/spnego_kerberos_auth.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/kerberos.json
>  51f541f 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
>  6934924 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/templates/metric_monitor.ini.j2
>  3823912 
> 
> Diff: https://reviews.apache.org/r/51724/diff/
> 
> 
> Testing
> ---
> 
> 1. passed existing unittest cases.
> 2. manually tested with HDP-2.5.0.0/trunk 
>1) Non http spnego enabled clusters
>2) http spnego enabled cluster - tested the trunk's monitor code against a 
> http spnego enabled cluster 
>   http spnego enabled cluster is currently broken in trunk: the 
> distributed metrics collector's http server currently will not started with 
> http spnego enabled. so I had to manually test the trunk's monitor code with 
> http spnego enabled collector on a branch2.4 cluster. Need to mention that 
> the testing should be valid because the code change area is monitor code and 
> the trunk and branch2.4's collectors are same.
> 
> 
> Thanks,
> 
> Qin Liu
> 
>

Re: Review Request 51724: Add Kerberos HTTP SPNEGO authentication support to Ambari Metrics Monitor

2016-09-22 Thread Robert Levas 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51724/#review150057
---




ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/config_reader.py
 (lines 119 - 120)


These shoud not be hardcoded. The aths to the tools (kinit and klist), the 
principal name, and the keytab file should be found in the config data.



ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/emitter.py 
(line 121)


Possibly use `curl_krb_request` instead.



ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/faked_kerberos.py
 (line 26)


The file name should be `krberr.py` rather than `faked_kerberos.py`



ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/spnego_kerberos_auth.py
 (line 33)


See `curl_krb_request`


- Robert Levas


On Sept. 22, 2016, 8:44 a.m., Qin Liu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51724/
> ---
> 
> (Updated Sept. 22, 2016, 8:44 a.m.)
> 
> 
> Review request for Ambari, Di Li, Dmytro Sen, and Sid Wagle.
> 
> 
> Bugs: AMBARI-17898
> https://issues.apache.org/jira/browse/AMBARI-17898
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> ()This is a subtask of AMBARI-14384 "Ambari Metrics doesn't use SPNEGO to 
> authenticate".
> 
> In a Kerberos enabled cluster with SPNEGO enabled on Hadoop APIs, Ambari 
> Metrics Collector (in AMS distributed mode) web-console will be Kerberos HTTP 
> SPNEGO enabled too. But Ambari Metrics Monitor, a client of Ambari Metrics 
> Collector, currently does not support Kerberos HTTP SPNEGO authentication.
> 
> /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out:
> 2015-12-15 13:26:30,663 [INFO] emitter.py:101 - server: 
> http://metrics-collector:6188/ws/v1/timeline/metrics
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:84 - Error sending metrics to 
> server. HTTP Error 401: Authentication required
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:90 - Retrying after 5 ...
> 
> 
> Diffs
> -
> 
>   ambari-metrics/ambari-metrics-assembly/pom.xml d73a0af 
>   ambari-metrics/ambari-metrics-host-monitoring/conf/unix/metric_monitor.ini 
> e98c65c 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/config_reader.py
>  3ca3a31 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/emitter.py 
> 050af16 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/faked_kerberos.py
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/spnego_kerberos_auth.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/kerberos.json
>  51f541f 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
>  6934924 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/templates/metric_monitor.ini.j2
>  3823912 
> 
> Diff: https://reviews.apache.org/r/51724/diff/
> 
> 
> Testing
> ---
> 
> 1. passed existing unittest cases.
> 2. manually tested with HDP-2.5.0.0/trunk 
>1) Non http spnego enabled clusters
>2) http spnego enabled cluster - tested the trunk's monitor code against a 
> http spnego enabled cluster 
>   http spnego enabled cluster is currently broken in trunk: the 
> distributed metrics collector's http server currently will not started with 
> http spnego enabled. so I had to manually test the trunk's monitor code with 
> http spnego enabled collector on a branch2.4 cluster. Need to mention that 
> the testing should be valid because the code change area is monitor code and 
> the trunk and branch2.4's collectors are same.
> 
> 
> Thanks,
> 
> Qin Liu
> 
>

Re: Review Request 51724: Add Kerberos HTTP SPNEGO authentication support to Ambari Metrics Monitor

2016-09-22 Thread Qin Liu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51724/
---

(Updated Sept. 22, 2016, 12:44 p.m.)


Review request for Ambari, Di Li, Dmytro Sen, and Sid Wagle.


Changes
---

rebase metric_monitor.ini and metric_monitor.ini.j2


Bugs: AMBARI-17898
https://issues.apache.org/jira/browse/AMBARI-17898


Repository: ambari


Description
---

()This is a subtask of AMBARI-14384 "Ambari Metrics doesn't use SPNEGO to 
authenticate".

In a Kerberos enabled cluster with SPNEGO enabled on Hadoop APIs, Ambari 
Metrics Collector (in AMS distributed mode) web-console will be Kerberos HTTP 
SPNEGO enabled too. But Ambari Metrics Monitor, a client of Ambari Metrics 
Collector, currently does not support Kerberos HTTP SPNEGO authentication.

/var/log/ambari-metrics-monitor/ambari-metrics-monitor.out:
2015-12-15 13:26:30,663 [INFO] emitter.py:101 - server: 
http://metrics-collector:6188/ws/v1/timeline/metrics
2015-12-15 13:26:30,671 [WARNING] emitter.py:84 - Error sending metrics to 
server. HTTP Error 401: Authentication required
2015-12-15 13:26:30,671 [WARNING] emitter.py:90 - Retrying after 5 ...


Diffs (updated)
-

  ambari-metrics/ambari-metrics-assembly/pom.xml d73a0af 
  ambari-metrics/ambari-metrics-host-monitoring/conf/unix/metric_monitor.ini 
e98c65c 
  
ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/config_reader.py
 3ca3a31 
  ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/emitter.py 
050af16 
  
ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/faked_kerberos.py
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/spnego_kerberos_auth.py
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/kerberos.json
 51f541f 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
 6934924 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/templates/metric_monitor.ini.j2
 3823912 

Diff: https://reviews.apache.org/r/51724/diff/


Testing
---

1. passed existing unittest cases.
2. manually tested with HDP-2.5.0.0/trunk 
   1) Non http spnego enabled clusters
   2) http spnego enabled cluster - tested the trunk's monitor code against a 
http spnego enabled cluster 
  http spnego enabled cluster is currently broken in trunk: the distributed 
metrics collector's http server currently will not started with http spnego 
enabled. so I had to manually test the trunk's monitor code with http spnego 
enabled collector on a branch2.4 cluster. Need to mention that the testing 
should be valid because the code change area is monitor code and the trunk and 
branch2.4's collectors are same.


Thanks,

Qin Liu

Re: Review Request 51724: Add Kerberos HTTP SPNEGO authentication support to Ambari Metrics Monitor

2016-09-22 Thread Qin Liu 


> On Sept. 19, 2016, 9:18 a.m., Robert Levas wrote:
> > I think this is incorrect.  The Metrics Monitor should authenticate with 
> > its own service principal, not the SPNEGO principal.  That is used for 
> > web-based services, like Ambari's web-based interface.  There should be an 
> > `ams/_HOST` principal for this, instead.

Thanks Robert! I replaced the spnego principal with ams principle in the new 
patch.


> On Sept. 19, 2016, 9:18 a.m., Robert Levas wrote:
> > ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/config_reader.py,
> >  line 112
> > 
> >
> > The path to the SPNEGO keytab file and the SPNEGO principal name must 
> > not be hard coded.  There should be a relevant config property for this and 
> > it should be set via the Kerberos descriptor.
> 
> Sid Wagle wrote:
> Ahh thanks Robert for pointing out the keytab, I missed that part. @Qin: 
> This should be easy to add by editing 
> common-services/AMBARI_METRICS/0.1.0/kerberos.json.

Thanks Sid! I added keytab/principal for the ams monitor to kerberos.json in 
the new patch.


In addition, for hardcode issue raised by Robert, the "kinit_cmd = 
/usr/bin/kinit -kt ..." in config_reader.py is just a comment/example (with new 
change, it will be kinit_cmd = /usr/bin/kinit -kt 
/etc/security/keytabs/ams.monitor.keytab amsmon/localhost). It won't take any 
effect. The real kinit_cmd will be obtained from the API in config_reader.py:
class Configuration:
...
  def get_kinit_cmd(self):
return self.get("emitter", "kinit_cmd")

and "kinit_cmd" will be obtained from params.py below in the new patch:
  kinit_cmd = '%s -kt %s %s' % (kinit_path_local, 
config['configurations']['ams-hbase-security-site']['ams.monitor.keytab'], 
config['configurations']['ams-hbase-security-site']['ams.monitor.principal'].replace('_HOST',_hostname_lowercase))


- Qin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51724/#review149443
---


On Sept. 22, 2016, 10:14 a.m., Qin Liu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51724/
> ---
> 
> (Updated Sept. 22, 2016, 10:14 a.m.)
> 
> 
> Review request for Ambari, Di Li, Dmytro Sen, and Sid Wagle.
> 
> 
> Bugs: AMBARI-17898
> https://issues.apache.org/jira/browse/AMBARI-17898
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> ()This is a subtask of AMBARI-14384 "Ambari Metrics doesn't use SPNEGO to 
> authenticate".
> 
> In a Kerberos enabled cluster with SPNEGO enabled on Hadoop APIs, Ambari 
> Metrics Collector (in AMS distributed mode) web-console will be Kerberos HTTP 
> SPNEGO enabled too. But Ambari Metrics Monitor, a client of Ambari Metrics 
> Collector, currently does not support Kerberos HTTP SPNEGO authentication.
> 
> /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out:
> 2015-12-15 13:26:30,663 [INFO] emitter.py:101 - server: 
> http://metrics-collector:6188/ws/v1/timeline/metrics
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:84 - Error sending metrics to 
> server. HTTP Error 401: Authentication required
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:90 - Retrying after 5 ...
> 
> 
> Diffs
> -
> 
>   ambari-metrics/ambari-metrics-assembly/pom.xml d73a0af 
>   ambari-metrics/ambari-metrics-host-monitoring/conf/unix/metric_monitor.ini 
> e98c65c 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/config_reader.py
>  3ca3a31 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/emitter.py 
> 050af16 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/faked_kerberos.py
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/spnego_kerberos_auth.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/kerberos.json
>  51f541f 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
>  22024bb 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/templates/metric_monitor.ini.j2
>  3823912 
> 
> Diff: https://reviews.apache.org/r/51724/diff/
> 
> 
> Testing
> ---
> 
> 1. passed existing unittest cases.
> 2. manually tested with HDP-2.5.0.0/trunk 
>1) Non http spnego enabled clusters
>2) http spnego enabled cluster - tested the trunk's monitor code against a 
> http spnego enabled cluster 
>   http spnego enabled cluster is currently broken in trunk: the 
> distributed metrics collector's http server currently will not started with 
> http spnego enabled. so I had to manually test the trunk's monitor code with 
> http spnego

Re: Review Request 51724: Add Kerberos HTTP SPNEGO authentication support to Ambari Metrics Monitor

2016-09-22 Thread Qin Liu 


> On Sept. 8, 2016, 6:13 p.m., Sid Wagle wrote:
> > ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/spnego_kerberos_auth.py,
> >  line 25
> > 
> >
> > I do not see a "import kerberos" anywhere in Ambari deployed cluster 
> > under: /usr/lib/python2.6/site-packages/
> > 
> > Where does this dependency get fulfiled ?
> 
> Qin Liu wrote:
> It is in /usr/lib/python2.6/site-packages/ipalib/plugins on all of my 
> clusters. I will include ipa-python in ambari-metrics-assembly/pom.xml. 
> Thanks Sid!
> 
> Sid Wagle wrote:
> This seems to be ipa only dependency, does addinig it as component build 
> dep make sense?
> What about MIT kereberos?
> 
> Qin Liu wrote:
> Hi Sid,
> Yes, ipa is the only dependency for metrics monitor code. I think it 
> makes sense to include ipa in ambari-metrics component build deployment.
> MIT kerberos is outside of this JIRA.
> 
> Sid Wagle wrote:
> In that case we should also log error on the import if module not found. 
> Additionally, open a Jira for getting this to work with MIT Kerberos so 
> someone can take it up. As far as this patch is concerened we are good 
> contingent on the small logging change and making sure we have a Jira to 
> track MIT Kerberos won't work the same way.

Hi Sid,
I updated the code to log error on the import if kerberos module is not found. 
On a deployed cluster, the exception should never occur because kerberos module 
should always be included in ipa lib. The gss API in kerberos module is used to 
authenticate using kerberos. The kerberos to be used is MIT kerberos because an 
existing MIT KDC should be chosen at cluster kerberos enabling.


- Qin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51724/#review148224
---


On Sept. 22, 2016, 10:14 a.m., Qin Liu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51724/
> ---
> 
> (Updated Sept. 22, 2016, 10:14 a.m.)
> 
> 
> Review request for Ambari, Di Li, Dmytro Sen, and Sid Wagle.
> 
> 
> Bugs: AMBARI-17898
> https://issues.apache.org/jira/browse/AMBARI-17898
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> ()This is a subtask of AMBARI-14384 "Ambari Metrics doesn't use SPNEGO to 
> authenticate".
> 
> In a Kerberos enabled cluster with SPNEGO enabled on Hadoop APIs, Ambari 
> Metrics Collector (in AMS distributed mode) web-console will be Kerberos HTTP 
> SPNEGO enabled too. But Ambari Metrics Monitor, a client of Ambari Metrics 
> Collector, currently does not support Kerberos HTTP SPNEGO authentication.
> 
> /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out:
> 2015-12-15 13:26:30,663 [INFO] emitter.py:101 - server: 
> http://metrics-collector:6188/ws/v1/timeline/metrics
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:84 - Error sending metrics to 
> server. HTTP Error 401: Authentication required
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:90 - Retrying after 5 ...
> 
> 
> Diffs
> -
> 
>   ambari-metrics/ambari-metrics-assembly/pom.xml d73a0af 
>   ambari-metrics/ambari-metrics-host-monitoring/conf/unix/metric_monitor.ini 
> e98c65c 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/config_reader.py
>  3ca3a31 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/emitter.py 
> 050af16 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/faked_kerberos.py
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/spnego_kerberos_auth.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/kerberos.json
>  51f541f 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
>  22024bb 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/templates/metric_monitor.ini.j2
>  3823912 
> 
> Diff: https://reviews.apache.org/r/51724/diff/
> 
> 
> Testing
> ---
> 
> 1. passed existing unittest cases.
> 2. manually tested with HDP-2.5.0.0/trunk 
>1) Non http spnego enabled clusters
>2) http spnego enabled cluster - tested the trunk's monitor code against a 
> http spnego enabled cluster 
>   http spnego enabled cluster is currently broken in trunk: the 
> distributed metrics collector's http server currently will not started with 
> http spnego enabled. so I had to manually test the trunk's monitor code with 
> http spnego enabled collector on a branch2.4 cluster. Need to mention that 
> the testing should be valid because the code change area is monitor code and 
> the trunk and branch2.4's

Re: Review Request 51724: Add Kerberos HTTP SPNEGO authentication support to Ambari Metrics Monitor

2016-09-22 Thread Qin Liu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51724/
---

(Updated Sept. 22, 2016, 10:14 a.m.)


Review request for Ambari, Di Li, Dmytro Sen, and Sid Wagle.


Changes
---

replace spnego principle with ams principle


Bugs: AMBARI-17898
https://issues.apache.org/jira/browse/AMBARI-17898


Repository: ambari


Description
---

()This is a subtask of AMBARI-14384 "Ambari Metrics doesn't use SPNEGO to 
authenticate".

In a Kerberos enabled cluster with SPNEGO enabled on Hadoop APIs, Ambari 
Metrics Collector (in AMS distributed mode) web-console will be Kerberos HTTP 
SPNEGO enabled too. But Ambari Metrics Monitor, a client of Ambari Metrics 
Collector, currently does not support Kerberos HTTP SPNEGO authentication.

/var/log/ambari-metrics-monitor/ambari-metrics-monitor.out:
2015-12-15 13:26:30,663 [INFO] emitter.py:101 - server: 
http://metrics-collector:6188/ws/v1/timeline/metrics
2015-12-15 13:26:30,671 [WARNING] emitter.py:84 - Error sending metrics to 
server. HTTP Error 401: Authentication required
2015-12-15 13:26:30,671 [WARNING] emitter.py:90 - Retrying after 5 ...


Diffs (updated)
-

  ambari-metrics/ambari-metrics-assembly/pom.xml d73a0af 
  ambari-metrics/ambari-metrics-host-monitoring/conf/unix/metric_monitor.ini 
e98c65c 
  
ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/config_reader.py
 3ca3a31 
  ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/emitter.py 
050af16 
  
ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/faked_kerberos.py
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/spnego_kerberos_auth.py
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/kerberos.json
 51f541f 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
 22024bb 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/templates/metric_monitor.ini.j2
 3823912 

Diff: https://reviews.apache.org/r/51724/diff/


Testing
---

1. passed existing unittest cases.
2. manually tested with HDP-2.5.0.0/trunk 
   1) Non http spnego enabled clusters
   2) http spnego enabled cluster - tested the trunk's monitor code against a 
http spnego enabled cluster 
  http spnego enabled cluster is currently broken in trunk: the distributed 
metrics collector's http server currently will not started with http spnego 
enabled. so I had to manually test the trunk's monitor code with http spnego 
enabled collector on a branch2.4 cluster. Need to mention that the testing 
should be valid because the code change area is monitor code and the trunk and 
branch2.4's collectors are same.


Thanks,

Qin Liu

Re: Review Request 51724: Add Kerberos HTTP SPNEGO authentication support to Ambari Metrics Monitor

2016-09-19 Thread Sid Wagle 


> On Sept. 19, 2016, 9:18 a.m., Robert Levas wrote:
> > ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/config_reader.py,
> >  line 112
> > 
> >
> > The path to the SPNEGO keytab file and the SPNEGO principal name must 
> > not be hard coded.  There should be a relevant config property for this and 
> > it should be set via the Kerberos descriptor.

Ahh thanks Robert for pointing out the keytab, I missed that part. @Qin: This 
should be easy to add by editing 
common-services/AMBARI_METRICS/0.1.0/kerberos.json.


- Sid


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51724/#review149443
---


On Sept. 19, 2016, 3:53 p.m., Qin Liu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51724/
> ---
> 
> (Updated Sept. 19, 2016, 3:53 p.m.)
> 
> 
> Review request for Ambari, Di Li, Dmytro Sen, and Sid Wagle.
> 
> 
> Bugs: AMBARI-17898
> https://issues.apache.org/jira/browse/AMBARI-17898
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> ()This is a subtask of AMBARI-14384 "Ambari Metrics doesn't use SPNEGO to 
> authenticate".
> 
> In a Kerberos enabled cluster with SPNEGO enabled on Hadoop APIs, Ambari 
> Metrics Collector (in AMS distributed mode) web-console will be Kerberos HTTP 
> SPNEGO enabled too. But Ambari Metrics Monitor, a client of Ambari Metrics 
> Collector, currently does not support Kerberos HTTP SPNEGO authentication.
> 
> /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out:
> 2015-12-15 13:26:30,663 [INFO] emitter.py:101 - server: 
> http://metrics-collector:6188/ws/v1/timeline/metrics
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:84 - Error sending metrics to 
> server. HTTP Error 401: Authentication required
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:90 - Retrying after 5 ...
> 
> 
> Diffs
> -
> 
>   ambari-metrics/ambari-metrics-assembly/pom.xml d73a0af 
>   ambari-metrics/ambari-metrics-host-monitoring/conf/unix/metric_monitor.ini 
> 59ca5d7 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/config_reader.py
>  44692f6 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/emitter.py 
> 050af16 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/faked_kerberos.py
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/spnego_kerberos_auth.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
>  3908b27 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/templates/metric_monitor.ini.j2
>  65129f8 
> 
> Diff: https://reviews.apache.org/r/51724/diff/
> 
> 
> Testing
> ---
> 
> 1. passed existing unittest cases.
> 2. manually tested with HDP-2.5.0.0/trunk 
>1) Non http spnego enabled clusters
>2) http spnego enabled cluster - tested the trunk's monitor code against a 
> http spnego enabled cluster 
>   http spnego enabled cluster is currently broken in trunk: the 
> distributed metrics collector's http server currently will not started with 
> http spnego enabled. so I had to manually test the trunk's monitor code with 
> http spnego enabled collector on a branch2.4 cluster. Need to mention that 
> the testing should be valid because the code change area is monitor code and 
> the trunk and branch2.4's collectors are same.
> 
> 
> Thanks,
> 
> Qin Liu
> 
>

Re: Review Request 51724: Add Kerberos HTTP SPNEGO authentication support to Ambari Metrics Monitor

2016-09-19 Thread Qin Liu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51724/
---

(Updated Sept. 19, 2016, 3:53 p.m.)


Review request for Ambari, Di Li, Dmytro Sen, and Sid Wagle.


Changes
---

new changes:
1. update ambari-metrics-assembly/pom.xml to include ipa lib to rpm
2. rebase emitter.py with recent change and issue warning if kerberos module is 
not found.


Bugs: AMBARI-17898
https://issues.apache.org/jira/browse/AMBARI-17898


Repository: ambari


Description
---

()This is a subtask of AMBARI-14384 "Ambari Metrics doesn't use SPNEGO to 
authenticate".

In a Kerberos enabled cluster with SPNEGO enabled on Hadoop APIs, Ambari 
Metrics Collector (in AMS distributed mode) web-console will be Kerberos HTTP 
SPNEGO enabled too. But Ambari Metrics Monitor, a client of Ambari Metrics 
Collector, currently does not support Kerberos HTTP SPNEGO authentication.

/var/log/ambari-metrics-monitor/ambari-metrics-monitor.out:
2015-12-15 13:26:30,663 [INFO] emitter.py:101 - server: 
http://metrics-collector:6188/ws/v1/timeline/metrics
2015-12-15 13:26:30,671 [WARNING] emitter.py:84 - Error sending metrics to 
server. HTTP Error 401: Authentication required
2015-12-15 13:26:30,671 [WARNING] emitter.py:90 - Retrying after 5 ...


Diffs (updated)
-

  ambari-metrics/ambari-metrics-assembly/pom.xml d73a0af 
  ambari-metrics/ambari-metrics-host-monitoring/conf/unix/metric_monitor.ini 
59ca5d7 
  
ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/config_reader.py
 44692f6 
  ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/emitter.py 
050af16 
  
ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/faked_kerberos.py
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/spnego_kerberos_auth.py
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
 3908b27 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/templates/metric_monitor.ini.j2
 65129f8 

Diff: https://reviews.apache.org/r/51724/diff/


Testing (updated)
---

1. passed existing unittest cases.
2. manually tested with HDP-2.5.0.0/trunk 
   1) Non http spnego enabled clusters
   2) http spnego enabled cluster - tested the trunk's monitor code against a 
http spnego enabled cluster 
  http spnego enabled cluster is currently broken in trunk: the distributed 
metrics collector's http server currently will not started with http spnego 
enabled. so I had to manually test the trunk's monitor code with http spnego 
enabled collector on a branch2.4 cluster. Need to mention that the testing 
should be valid because the code change area is monitor code and the trunk and 
branch2.4's collectors are same.


Thanks,

Qin Liu

Re: Review Request 51724: Add Kerberos HTTP SPNEGO authentication support to Ambari Metrics Monitor

2016-09-17 Thread Qin Liu 


> On Sept. 8, 2016, 6:13 p.m., Sid Wagle wrote:
> > ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/spnego_kerberos_auth.py,
> >  line 25
> > 
> >
> > I do not see a "import kerberos" anywhere in Ambari deployed cluster 
> > under: /usr/lib/python2.6/site-packages/
> > 
> > Where does this dependency get fulfiled ?
> 
> Qin Liu wrote:
> It is in /usr/lib/python2.6/site-packages/ipalib/plugins on all of my 
> clusters. I will include ipa-python in ambari-metrics-assembly/pom.xml. 
> Thanks Sid!
> 
> Sid Wagle wrote:
> This seems to be ipa only dependency, does addinig it as component build 
> dep make sense?
> What about MIT kereberos?

Hi Sid,
Yes, ipa is the only dependency for metrics monitor code. I think it makes 
sense to include ipa in ambari-metrics component build deployment.
MIT kerberos is outside of this JIRA.


On Sept. 8, 2016, 6:13 p.m., Qin Liu wrote:
> > We don't have unit test whcih setup secure cluster so we should definitely 
> > test this on trunk build, which should be working now.
> > The changes overall look good just want to make sure we have done some 
> > manual testing before commit.
> 
> Qin Liu wrote:
> Hi Sid,
> I tried to test it on HDP2.5.0.0/trunk. but I was not able to bring the 
> metrics up. Do you have this experience? 
> But I will try to test it on HDP2.4/trunk.
> 
> Qin Liu wrote:
> I got latest version of trunk yesterday and setup to run AMS distributed 
> mode without kerberos enabled but I was unable to start monitor and 
> collector. I got:
> 
> 1) Traceback (most recent call last):
>   File 
> "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py",
>  line 152, in 
> AmsCollector().execute()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 281, in execute
> method(env)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 734, in restart
> self.status(env)
>   File 
> "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py",
>  line 61, in status
> check_service_status(name='collector')
>   File 
> "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, 
> in thunk
> return fn(*args, **kwargs)
>   File 
> "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/status.py",
>  line 36, in check_service_status
> env.set_params(status_params)
> NameError: global name 'env' is not defined >introduced by 
> AMBARI-17159
> 
> then I switched back to previous version of status.py and was able to 
> start monitors.
> 
> Then, I got:
> 2) Traceback (most recent call last):
>   File 
> "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py",
>  line 152, in 
> AmsCollector().execute()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 284, in execute
> self.post_start()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 321, in post_start
> pid_files = self.get_pid_files()
>   File 
> "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py",
>  line 73, in get_pid_files
> return status.get_collector_pid_files()
> AttributeError: 'module' object has no attribute 'get_collector_pid_files'
> 
> then, I hardcoded to "return status.get_collector_pid_files()" with 
> "return []" and was able to start collector with a warning. But monitor got " 
> Error sending metrics to server. [Errno -2] Name or service not known" then " 
> Metric collector host ['myhost.example.com'] was blacklisted." I didn't get a 
> chance to look into it (it might be caused by 
> status.get_collector_pid_files() "return []").
> 
> Any way, anyone saw above issues? Thank you!
> 
> Sid Wagle wrote:
> Hi Qin Liu, Can you please open a jira for this in Apache for the errors? 
> Will get this resovled ASAP.
> 
> Aravindan Vijayan wrote:
> Hi Qin Liu,
> 
> On trunk deploy, AMS components are in fact running. But the bug is that 
> their statuses are not being reported to Ambari correctly. I have fixed it in 
> https://reviews.apache.org/r/51711/. I will check it in today.

Hi Sid, 
I no longer see "NameError: global name 'env' is not defined" and the collector 
and monitor status issues with recent code.


- Qin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51724/#review148224
---


On Sept.

Re: Review Request 51724: Add Kerberos HTTP SPNEGO authentication support to Ambari Metrics Monitor

2016-09-12 Thread Aravindan Vijayan 


On Sept. 8, 2016, 6:13 p.m., Qin Liu wrote:
> > We don't have unit test whcih setup secure cluster so we should definitely 
> > test this on trunk build, which should be working now.
> > The changes overall look good just want to make sure we have done some 
> > manual testing before commit.
> 
> Qin Liu wrote:
> Hi Sid,
> I tried to test it on HDP2.5.0.0/trunk. but I was not able to bring the 
> metrics up. Do you have this experience? 
> But I will try to test it on HDP2.4/trunk.
> 
> Qin Liu wrote:
> I got latest version of trunk yesterday and setup to run AMS distributed 
> mode without kerberos enabled but I was unable to start monitor and 
> collector. I got:
> 
> 1) Traceback (most recent call last):
>   File 
> "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py",
>  line 152, in 
> AmsCollector().execute()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 281, in execute
> method(env)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 734, in restart
> self.status(env)
>   File 
> "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py",
>  line 61, in status
> check_service_status(name='collector')
>   File 
> "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, 
> in thunk
> return fn(*args, **kwargs)
>   File 
> "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/status.py",
>  line 36, in check_service_status
> env.set_params(status_params)
> NameError: global name 'env' is not defined >introduced by 
> AMBARI-17159
> 
> then I switched back to previous version of status.py and was able to 
> start monitors.
> 
> Then, I got:
> 2) Traceback (most recent call last):
>   File 
> "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py",
>  line 152, in 
> AmsCollector().execute()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 284, in execute
> self.post_start()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 321, in post_start
> pid_files = self.get_pid_files()
>   File 
> "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py",
>  line 73, in get_pid_files
> return status.get_collector_pid_files()
> AttributeError: 'module' object has no attribute 'get_collector_pid_files'
> 
> then, I hardcoded to "return status.get_collector_pid_files()" with 
> "return []" and was able to start collector with a warning. But monitor got " 
> Error sending metrics to server. [Errno -2] Name or service not known" then " 
> Metric collector host ['myhost.example.com'] was blacklisted." I didn't get a 
> chance to look into it (it might be caused by 
> status.get_collector_pid_files() "return []").
> 
> Any way, anyone saw above issues? Thank you!
> 
> Sid Wagle wrote:
> Hi Qin Liu, Can you please open a jira for this in Apache for the errors? 
> Will get this resovled ASAP.

Hi Qin Liu,

On trunk deploy, AMS components are in fact running. But the bug is that their 
statuses are not being reported to Ambari correctly. I have fixed it in 
https://reviews.apache.org/r/51711/. I will check it in today.


- Aravindan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51724/#review148224
---


On Sept. 8, 2016, 5:27 p.m., Qin Liu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51724/
> ---
> 
> (Updated Sept. 8, 2016, 5:27 p.m.)
> 
> 
> Review request for Ambari, Di Li, Dmytro Sen, and Sid Wagle.
> 
> 
> Bugs: AMBARI-17898
> https://issues.apache.org/jira/browse/AMBARI-17898
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> ()This is a subtask of AMBARI-14384 "Ambari Metrics doesn't use SPNEGO to 
> authenticate".
> 
> In a Kerberos enabled cluster with SPNEGO enabled on Hadoop APIs, Ambari 
> Metrics Collector (in AMS distributed mode) web-console will be Kerberos HTTP 
> SPNEGO enabled too. But Ambari Metrics Monitor, a client of Ambari Metrics 
> Collector, currently does not support Kerberos HTTP SPNEGO authentication.
> 
> /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out:
> 2015-12-15 13:26:30,663 [INFO] emitter.py:101 - server: 
> http://metrics-collector:6188/ws/v1/timeline/metrics
> 2015-12-15 13:26:30,671

Re: Review Request 51724: Add Kerberos HTTP SPNEGO authentication support to Ambari Metrics Monitor

2016-09-12 Thread Sid Wagle 


> On Sept. 8, 2016, 6:13 p.m., Sid Wagle wrote:
> > ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/spnego_kerberos_auth.py,
> >  line 25
> > 
> >
> > I do not see a "import kerberos" anywhere in Ambari deployed cluster 
> > under: /usr/lib/python2.6/site-packages/
> > 
> > Where does this dependency get fulfiled ?
> 
> Qin Liu wrote:
> It is in /usr/lib/python2.6/site-packages/ipalib/plugins on all of my 
> clusters. I will include ipa-python in ambari-metrics-assembly/pom.xml. 
> Thanks Sid!

This seems to be ipa only dependency, does addinig it as component build dep 
make sense?
What about MIT kereberos?


On Sept. 8, 2016, 6:13 p.m., Qin Liu wrote:
> > We don't have unit test whcih setup secure cluster so we should definitely 
> > test this on trunk build, which should be working now.
> > The changes overall look good just want to make sure we have done some 
> > manual testing before commit.
> 
> Qin Liu wrote:
> Hi Sid,
> I tried to test it on HDP2.5.0.0/trunk. but I was not able to bring the 
> metrics up. Do you have this experience? 
> But I will try to test it on HDP2.4/trunk.
> 
> Qin Liu wrote:
> I got latest version of trunk yesterday and setup to run AMS distributed 
> mode without kerberos enabled but I was unable to start monitor and 
> collector. I got:
> 
> 1) Traceback (most recent call last):
>   File 
> "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py",
>  line 152, in 
> AmsCollector().execute()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 281, in execute
> method(env)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 734, in restart
> self.status(env)
>   File 
> "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py",
>  line 61, in status
> check_service_status(name='collector')
>   File 
> "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, 
> in thunk
> return fn(*args, **kwargs)
>   File 
> "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/status.py",
>  line 36, in check_service_status
> env.set_params(status_params)
> NameError: global name 'env' is not defined >introduced by 
> AMBARI-17159
> 
> then I switched back to previous version of status.py and was able to 
> start monitors.
> 
> Then, I got:
> 2) Traceback (most recent call last):
>   File 
> "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py",
>  line 152, in 
> AmsCollector().execute()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 284, in execute
> self.post_start()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 321, in post_start
> pid_files = self.get_pid_files()
>   File 
> "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py",
>  line 73, in get_pid_files
> return status.get_collector_pid_files()
> AttributeError: 'module' object has no attribute 'get_collector_pid_files'
> 
> then, I hardcoded to "return status.get_collector_pid_files()" with 
> "return []" and was able to start collector with a warning. But monitor got " 
> Error sending metrics to server. [Errno -2] Name or service not known" then " 
> Metric collector host ['myhost.example.com'] was blacklisted." I didn't get a 
> chance to look into it (it might be caused by 
> status.get_collector_pid_files() "return []").
> 
> Any way, anyone saw above issues? Thank you!

Hi Qin Liu, Can you please open a jira for this in Apache for the errors? Will 
get this resovled ASAP.


- Sid


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51724/#review148224
---


On Sept. 8, 2016, 5:27 p.m., Qin Liu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51724/
> ---
> 
> (Updated Sept. 8, 2016, 5:27 p.m.)
> 
> 
> Review request for Ambari, Di Li, Dmytro Sen, and Sid Wagle.
> 
> 
> Bugs: AMBARI-17898
> https://issues.apache.org/jira/browse/AMBARI-17898
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> ()This is a subtask of AMBARI-14384 "Ambari Metrics doesn't use SPNEGO to 
> authenticate".
> 
> In a Kerberos enabled cluster with SPNEGO enabled on

Re: Review Request 51724: Add Kerberos HTTP SPNEGO authentication support to Ambari Metrics Monitor

2016-09-12 Thread Qin Liu 


On Sept. 8, 2016, 6:13 p.m., Qin Liu wrote:
> > We don't have unit test whcih setup secure cluster so we should definitely 
> > test this on trunk build, which should be working now.
> > The changes overall look good just want to make sure we have done some 
> > manual testing before commit.
> 
> Qin Liu wrote:
> Hi Sid,
> I tried to test it on HDP2.5.0.0/trunk. but I was not able to bring the 
> metrics up. Do you have this experience? 
> But I will try to test it on HDP2.4/trunk.

I got latest version of trunk yesterday and setup to run AMS distributed mode 
without kerberos enabled but I was unable to start monitor and collector. I got:

1) Traceback (most recent call last):
  File 
"/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py",
 line 152, in 
AmsCollector().execute()
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
 line 281, in execute
method(env)
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
 line 734, in restart
self.status(env)
  File 
"/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py",
 line 61, in status
check_service_status(name='collector')
  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", 
line 89, in thunk
return fn(*args, **kwargs)
  File 
"/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/status.py",
 line 36, in check_service_status
env.set_params(status_params)
NameError: global name 'env' is not defined >introduced by AMBARI-17159

then I switched back to previous version of status.py and was able to start 
monitors.

Then, I got:
2) Traceback (most recent call last):
  File 
"/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py",
 line 152, in 
AmsCollector().execute()
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
 line 284, in execute
self.post_start()
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
 line 321, in post_start
pid_files = self.get_pid_files()
  File 
"/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py",
 line 73, in get_pid_files
return status.get_collector_pid_files()
AttributeError: 'module' object has no attribute 'get_collector_pid_files'

then, I hardcoded to "return status.get_collector_pid_files()" with "return []" 
and was able to start collector with a warning. But monitor got " Error sending 
metrics to server. [Errno -2] Name or service not known" then " Metric 
collector host ['myhost.example.com'] was blacklisted." I didn't get a chance 
to look into it (it might be caused by status.get_collector_pid_files() "return 
[]").

Any way, anyone saw above issues? Thank you!


- Qin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51724/#review148224
---


On Sept. 8, 2016, 5:27 p.m., Qin Liu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51724/
> ---
> 
> (Updated Sept. 8, 2016, 5:27 p.m.)
> 
> 
> Review request for Ambari, Di Li, Dmytro Sen, and Sid Wagle.
> 
> 
> Bugs: AMBARI-17898
> https://issues.apache.org/jira/browse/AMBARI-17898
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> ()This is a subtask of AMBARI-14384 "Ambari Metrics doesn't use SPNEGO to 
> authenticate".
> 
> In a Kerberos enabled cluster with SPNEGO enabled on Hadoop APIs, Ambari 
> Metrics Collector (in AMS distributed mode) web-console will be Kerberos HTTP 
> SPNEGO enabled too. But Ambari Metrics Monitor, a client of Ambari Metrics 
> Collector, currently does not support Kerberos HTTP SPNEGO authentication.
> 
> /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out:
> 2015-12-15 13:26:30,663 [INFO] emitter.py:101 - server: 
> http://metrics-collector:6188/ws/v1/timeline/metrics
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:84 - Error sending metrics to 
> server. HTTP Error 401: Authentication required
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:90 - Retrying after 5 ...
> 
> 
> Diffs
> -
> 
>   ambari-metrics/ambari-metrics-host-monitoring/conf/unix/metric_monitor.ini 
> 3e5d861 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/config_reader.py
>  02f0ce3 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/emitter.py 
> 6997108 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/faked_kerberos.py
>  PRE-CREATION 
>   
>

Re: Review Request 51724: Add Kerberos HTTP SPNEGO authentication support to Ambari Metrics Monitor

2016-09-09 Thread Qin Liu 


> On Sept. 8, 2016, 6:13 p.m., Sid Wagle wrote:
> > ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/emitter.py,
> >  line 145
> > 
> >
> > Shouldn't this be checking for None instead of empty string ?
> 
> Qin Liu wrote:
> I will do it. Thanks Sid!

Sorry, I was looking at the wrong place. It should be checking for empty string 
instead of None because the key "Negotiate" is in auth_header.


- Qin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51724/#review148224
---


On Sept. 8, 2016, 5:27 p.m., Qin Liu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51724/
> ---
> 
> (Updated Sept. 8, 2016, 5:27 p.m.)
> 
> 
> Review request for Ambari, Di Li, Dmytro Sen, and Sid Wagle.
> 
> 
> Bugs: AMBARI-17898
> https://issues.apache.org/jira/browse/AMBARI-17898
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> ()This is a subtask of AMBARI-14384 "Ambari Metrics doesn't use SPNEGO to 
> authenticate".
> 
> In a Kerberos enabled cluster with SPNEGO enabled on Hadoop APIs, Ambari 
> Metrics Collector (in AMS distributed mode) web-console will be Kerberos HTTP 
> SPNEGO enabled too. But Ambari Metrics Monitor, a client of Ambari Metrics 
> Collector, currently does not support Kerberos HTTP SPNEGO authentication.
> 
> /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out:
> 2015-12-15 13:26:30,663 [INFO] emitter.py:101 - server: 
> http://metrics-collector:6188/ws/v1/timeline/metrics
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:84 - Error sending metrics to 
> server. HTTP Error 401: Authentication required
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:90 - Retrying after 5 ...
> 
> 
> Diffs
> -
> 
>   ambari-metrics/ambari-metrics-host-monitoring/conf/unix/metric_monitor.ini 
> 3e5d861 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/config_reader.py
>  02f0ce3 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/emitter.py 
> 6997108 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/faked_kerberos.py
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/spnego_kerberos_auth.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
>  fc95aa7 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/templates/metric_monitor.ini.j2
>  383a0de 
> 
> Diff: https://reviews.apache.org/r/51724/diff/
> 
> 
> Testing
> ---
> 
> 1. The fix has been tested with HDP-2.5.0.0/branch-2.4 UI (spnego enabled and 
> non-kerbero scenarios)
> - the patch can be applied to branch-2.4 without any changes. 
> - I was not able to test it on trunk because the current trunk version of 
> metrics collector was not able to start  
> 2. The fix has passed existing unittest cases in both trunk and branch-2.4
> 3. I didn't add new unittest cases because I don't know how to create a 
> kerberos enabled the cluster in unittest cases. - please advise me on this, 
> thx!
> 
> 
> Thanks,
> 
> Qin Liu
> 
>

Re: Review Request 51724: Add Kerberos HTTP SPNEGO authentication support to Ambari Metrics Monitor

2016-09-09 Thread Qin Liu 


> On Sept. 8, 2016, 6:13 p.m., Sid Wagle wrote:
> > ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/spnego_kerberos_auth.py,
> >  line 25
> > 
> >
> > I do not see a "import kerberos" anywhere in Ambari deployed cluster 
> > under: /usr/lib/python2.6/site-packages/
> > 
> > Where does this dependency get fulfiled ?

It is in /usr/lib/python2.6/site-packages/ipalib/plugins on all of my clusters. 
I will include ipa-python in ambari-metrics-assembly/pom.xml. Thanks Sid!


> On Sept. 8, 2016, 6:13 p.m., Sid Wagle wrote:
> > ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/emitter.py,
> >  line 145
> > 
> >
> > Shouldn't this be checking for None instead of empty string ?

I will do it. Thanks Sid!


On Sept. 8, 2016, 6:13 p.m., Qin Liu wrote:
> > We don't have unit test whcih setup secure cluster so we should definitely 
> > test this on trunk build, which should be working now.
> > The changes overall look good just want to make sure we have done some 
> > manual testing before commit.

Hi Sid,
I tried to test it on HDP2.5.0.0/trunk. but I was not able to bring the metrics 
up. Do you have this experience? 
But I will try to test it on HDP2.4/trunk.


- Qin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51724/#review148224
---


On Sept. 8, 2016, 5:27 p.m., Qin Liu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51724/
> ---
> 
> (Updated Sept. 8, 2016, 5:27 p.m.)
> 
> 
> Review request for Ambari, Di Li, Dmytro Sen, and Sid Wagle.
> 
> 
> Bugs: AMBARI-17898
> https://issues.apache.org/jira/browse/AMBARI-17898
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> ()This is a subtask of AMBARI-14384 "Ambari Metrics doesn't use SPNEGO to 
> authenticate".
> 
> In a Kerberos enabled cluster with SPNEGO enabled on Hadoop APIs, Ambari 
> Metrics Collector (in AMS distributed mode) web-console will be Kerberos HTTP 
> SPNEGO enabled too. But Ambari Metrics Monitor, a client of Ambari Metrics 
> Collector, currently does not support Kerberos HTTP SPNEGO authentication.
> 
> /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out:
> 2015-12-15 13:26:30,663 [INFO] emitter.py:101 - server: 
> http://metrics-collector:6188/ws/v1/timeline/metrics
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:84 - Error sending metrics to 
> server. HTTP Error 401: Authentication required
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:90 - Retrying after 5 ...
> 
> 
> Diffs
> -
> 
>   ambari-metrics/ambari-metrics-host-monitoring/conf/unix/metric_monitor.ini 
> 3e5d861 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/config_reader.py
>  02f0ce3 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/emitter.py 
> 6997108 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/faked_kerberos.py
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/spnego_kerberos_auth.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
>  fc95aa7 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/templates/metric_monitor.ini.j2
>  383a0de 
> 
> Diff: https://reviews.apache.org/r/51724/diff/
> 
> 
> Testing
> ---
> 
> 1. The fix has been tested with HDP-2.5.0.0/branch-2.4 UI (spnego enabled and 
> non-kerbero scenarios)
> - the patch can be applied to branch-2.4 without any changes. 
> - I was not able to test it on trunk because the current trunk version of 
> metrics collector was not able to start  
> 2. The fix has passed existing unittest cases in both trunk and branch-2.4
> 3. I didn't add new unittest cases because I don't know how to create a 
> kerberos enabled the cluster in unittest cases. - please advise me on this, 
> thx!
> 
> 
> Thanks,
> 
> Qin Liu
> 
>

Re: Review Request 51724: Add Kerberos HTTP SPNEGO authentication support to Ambari Metrics Monitor

2016-09-08 Thread Sid Wagle 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51724/#review148224
---




ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/emitter.py 
(line 145)


Shouldn't this be checking for None instead of empty string ?



ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/spnego_kerberos_auth.py
 (line 25)


I do not see a "import kerberos" anywhere in Ambari deployed cluster under: 
/usr/lib/python2.6/site-packages/

Where does this dependency get fulfiled ?


We don't have unit test whcih setup secure cluster so we should definitely test 
this on trunk build, which should be working now.
The changes overall look good just want to make sure we have done some manual 
testing before commit.

- Sid Wagle


On Sept. 8, 2016, 5:27 p.m., Qin Liu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51724/
> ---
> 
> (Updated Sept. 8, 2016, 5:27 p.m.)
> 
> 
> Review request for Ambari, Di Li, Dmytro Sen, and Sid Wagle.
> 
> 
> Bugs: AMBARI-17898
> https://issues.apache.org/jira/browse/AMBARI-17898
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> ()This is a subtask of AMBARI-14384 "Ambari Metrics doesn't use SPNEGO to 
> authenticate".
> 
> In a Kerberos enabled cluster with SPNEGO enabled on Hadoop APIs, Ambari 
> Metrics Collector (in AMS distributed mode) web-console will be Kerberos HTTP 
> SPNEGO enabled too. But Ambari Metrics Monitor, a client of Ambari Metrics 
> Collector, currently does not support Kerberos HTTP SPNEGO authentication.
> 
> /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out:
> 2015-12-15 13:26:30,663 [INFO] emitter.py:101 - server: 
> http://metrics-collector:6188/ws/v1/timeline/metrics
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:84 - Error sending metrics to 
> server. HTTP Error 401: Authentication required
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:90 - Retrying after 5 ...
> 
> 
> Diffs
> -
> 
>   ambari-metrics/ambari-metrics-host-monitoring/conf/unix/metric_monitor.ini 
> 3e5d861 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/config_reader.py
>  02f0ce3 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/emitter.py 
> 6997108 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/faked_kerberos.py
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/spnego_kerberos_auth.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
>  fc95aa7 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/templates/metric_monitor.ini.j2
>  383a0de 
> 
> Diff: https://reviews.apache.org/r/51724/diff/
> 
> 
> Testing
> ---
> 
> 1. The fix has been tested with HDP-2.5.0.0/branch-2.4 UI (spnego enabled and 
> non-kerbero scenarios)
> - the patch can be applied to branch-2.4 without any changes. 
> - I was not able to test it on trunk because the current trunk version of 
> metrics collector was not able to start  
> 2. The fix has passed existing unittest cases in both trunk and branch-2.4
> 3. I didn't add new unittest cases because I don't know how to create a 
> kerberos enabled the cluster in unittest cases. - please advise me on this, 
> thx!
> 
> 
> Thanks,
> 
> Qin Liu
> 
>