date:20170912

Re: Review Request 62250: AMBARI-21939 : Ambari HDFS Heatmaps are not showing data (Not Applicable) for bytes read, bytes written & DataNode Process Network I/O Utilization

2017-09-12 Thread Jayush Luniya


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62250/#review185250
---


Ship it!




Ship It!

- Jayush Luniya


On Sept. 12, 2017, 4:58 p.m., Aravindan Vijayan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62250/
> ---
> 
> (Updated Sept. 12, 2017, 4:58 p.m.)
> 
> 
> Review request for Ambari, Dmytro Sen, Jayush Luniya, and Sid Wagle.
> 
> 
> Bugs: AMBARI-21939
> https://issues.apache.org/jira/browse/AMBARI-21939
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> PROBLEM: HDFS Heatmaps are not showing data (Not Applicable) for bytes read, 
> bytes written & DataNode Process Network I/O Utilization
> 
> EXPECTED BEHAVIOUR: Heatmaps should show data
> 
> ACTUAL BEHAVIOUR: Data is being shown as Not Applicable (Grey Box)
> 
> ANALYSIS
> This is reproducible in Ambari 2.2.2 and Ambari 2.5.1.0. If you go to HDFS, 
> then click on Heatmaps, you can see the API call go to
> fields=host_components/metrics/dfs/datanode/bytes_written._rate 
> however this returns no values. If you remove ._rate then you can see the 
> values which you would expect.
> Attach one or more files to this issue
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
>  9e145c0 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/widgets.json 
> bcfb2cc 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/widgets.json 
> eeb9ff8 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/FAKEHDFS/widgets.json
>  7a793f8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog260Test.java
>  db6ebc1 
> 
> 
> Diff: https://reviews.apache.org/r/62250/diff/1/
> 
> 
> Testing
> ---
> 
> Manually tested.
> UpgradeCatalog260 test added.
> 
> 
> Thanks,
> 
> Aravindan Vijayan
> 
>

Re: Review Request 62250: AMBARI-21939 : Ambari HDFS Heatmaps are not showing data (Not Applicable) for bytes read, bytes written & DataNode Process Network I/O Utilization

2017-09-12 Thread Sid Wagle


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62250/#review185246
---


Ship it!




Ship It!

- Sid Wagle


On Sept. 12, 2017, 4:58 p.m., Aravindan Vijayan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62250/
> ---
> 
> (Updated Sept. 12, 2017, 4:58 p.m.)
> 
> 
> Review request for Ambari, Dmytro Sen, Jayush Luniya, and Sid Wagle.
> 
> 
> Bugs: AMBARI-21939
> https://issues.apache.org/jira/browse/AMBARI-21939
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> PROBLEM: HDFS Heatmaps are not showing data (Not Applicable) for bytes read, 
> bytes written & DataNode Process Network I/O Utilization
> 
> EXPECTED BEHAVIOUR: Heatmaps should show data
> 
> ACTUAL BEHAVIOUR: Data is being shown as Not Applicable (Grey Box)
> 
> ANALYSIS
> This is reproducible in Ambari 2.2.2 and Ambari 2.5.1.0. If you go to HDFS, 
> then click on Heatmaps, you can see the API call go to
> fields=host_components/metrics/dfs/datanode/bytes_written._rate 
> however this returns no values. If you remove ._rate then you can see the 
> values which you would expect.
> Attach one or more files to this issue
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
>  9e145c0 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/widgets.json 
> bcfb2cc 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/widgets.json 
> eeb9ff8 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/FAKEHDFS/widgets.json
>  7a793f8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog260Test.java
>  db6ebc1 
> 
> 
> Diff: https://reviews.apache.org/r/62250/diff/1/
> 
> 
> Testing
> ---
> 
> Manually tested.
> UpgradeCatalog260 test added.
> 
> 
> Thanks,
> 
> Aravindan Vijayan
> 
>

Re: Review Request 62125: AMBARI-21893 : NameNode Heap Usage (Daily) metric alert status flips to UNKNOWN intermittently when AMS HTTPS is enabled.

2017-09-12 Thread Sid Wagle


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62125/#review185245
---


Fix it, then Ship it!





ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
Lines 101 (patched)


More redable as host.alias vs the other way.


- Sid Wagle


On Sept. 12, 2017, 7:21 p.m., Aravindan Vijayan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62125/
> ---
> 
> (Updated Sept. 12, 2017, 7:21 p.m.)
> 
> 
> Review request for Ambari, Dmytro Sen, Jayush Luniya, and Sid Wagle.
> 
> 
> Bugs: AMBARI-21893
> https://issues.apache.org/jira/browse/AMBARI-21893
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> PROBLEM
> This issue is because of AMS HTTPS + HA scenario. The CA cert file 
> (/etc/ambari-metrics-monitor/conf/ca.pem) found on every host is generated by 
> fetching a specific metric collector host's certificate from the truststore. 
> This certificate file is being used by alert script, service check and even 
> monitors to talk to collector.
> 
> For example, in a cluster with hosts H1 to H5, let's say there are 2 
> collectors - H1 & H2.
> On a node H3, let's say the ca.pem file was constructed using the certificate 
> for collector H2.
> Service check or metric monitor on H3 will NOT be able to talk to H1 since it 
> does not have the certificate for that host.
> 
> FIX
> Import certificates from metric collectors onto the pem file, if 
> metric_truststore_alias is not specified. If the cluster has multiple 
> collectors and the same certificate is used for all the collectors, then the 
> metric_truststore_alias must be used to specify the alias to import the 
> certificate.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
>  9e145c0 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-ssl-client.xml
>  cac39de 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/ams.py
>  4672501 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
>  071882b 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog260Test.java
>  db6ebc1 
> 
> 
> Diff: https://reviews.apache.org/r/62125/diff/2/
> 
> 
> Testing
> ---
> 
> Manually tested.
> python unit tests pass.
> 
> 
> Thanks,
> 
> Aravindan Vijayan
> 
>

Re: Review Request 62258: Stack selection page does not load the HDP stacks [Intermittent]

2017-09-12 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62258/#review185235
---


Fix it, then Ship it!





ambari-server/src/main/java/org/apache/ambari/server/stack/StackContext.java
Line 151 (original), 159 (patched)


can we move amount of threads to constant, to make it more easy to tune in 
the future?



ambari-server/src/main/java/org/apache/ambari/server/stack/StackContext.java
Lines 219 (patched)


Looks like not used assign



ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java
Lines 1154 (patched)


This could be moved to the top of the function and be used after for the 
check?


- Dmytro Grinenko


On Sept. 12, 2017, 8:10 p.m., Nate Cole wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62258/
> ---
> 
> (Updated Sept. 12, 2017, 8:10 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Jonathan 
> Hurley.
> 
> 
> Bugs: AMBARI-21941
> https://issues.apache.org/jira/browse/AMBARI-21941
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Starting Ambari will load all the hdp_urlinfo.json and files and VDF in a 
> sequential manner.  This is all in a single-threaded Executor.  When that 
> happens, the data can take upwards of a full minute to be populated in some 
> cases.  This should be multi-threaded.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackContext.java 
> da7f021059 
>   ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 
> cbbe92e8cc 
>   ambari-server/src/main/java/org/apache/ambari/server/state/StackInfo.java 
> f1412f543e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/LatestRepoCallable.java
>  c43ce7c628 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/RepoUrlInfoCallable.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/RepoVdfCallable.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62258/diff/1/
> 
> 
> Testing
> ---
> 
> This is a refactor only.  Existing tests cover the functionality needed while 
> just adding a performance boost to loading VDF.
> 
> Manual.  Automated pending.
> 
> 
> Thanks,
> 
> Nate Cole
> 
>

Review Request 62258: Stack selection page does not load the HDP stacks [Intermittent]

2017-09-12 Thread Nate Cole


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62258/
---

Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Jonathan 
Hurley.


Bugs: AMBARI-21941
https://issues.apache.org/jira/browse/AMBARI-21941


Repository: ambari


Description
---

Starting Ambari will load all the hdp_urlinfo.json and files and VDF in a 
sequential manner.  This is all in a single-threaded Executor.  When that 
happens, the data can take upwards of a full minute to be populated in some 
cases.  This should be multi-threaded.


Diffs
-

  ambari-server/src/main/java/org/apache/ambari/server/stack/StackContext.java 
da7f021059 
  ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 
cbbe92e8cc 
  ambari-server/src/main/java/org/apache/ambari/server/state/StackInfo.java 
f1412f543e 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/LatestRepoCallable.java
 c43ce7c628 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/RepoUrlInfoCallable.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/RepoVdfCallable.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/62258/diff/1/


Testing
---

Manual.  Automated pending.


Thanks,

Nate Cole

Re: Review Request 62245: Auto fix enhancement to remove more than 1 selected configs

2017-09-12 Thread Jonathan Hurley



> On Sept. 12, 2017, 11:59 a.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/checks/DatabaseConsistencyCheckHelper.java
> > Lines 415-449 (patched)
> > 
> >
> > Does it make sense to just query for all configuration types which are 
> > selected more than once directly?
> > 
> > ```SELECT COUNT(type_name), type_name FROM clusterconfig WHERE selected 
> > = 1 GROUP BY type_name HAVING COUNT(type_name) > 1```
> 
> Dmitro Lisnichenko wrote:
> even if I use this query, I'll only get config type names and counts. And 
> I would still have to fetch these entities in order to modify them (set 
> selected to false)

True, but you could eliminate all of the looping and tracking to find 
duplicates with 1 SQL call.


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62245/#review185203
---


On Sept. 12, 2017, 10:28 a.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62245/
> ---
> 
> (Updated Sept. 12, 2017, 10:28 a.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley and Nate Cole.
> 
> 
> Bugs: AMBARI-21935
> https://issues.apache.org/jira/browse/AMBARI-21935
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Sometimes Ambari is coming up with issues like more than 1 selected configs.
> 
> Error:
> {noformat}
> ERROR - You have config(s), in cluster BMOHDPQA01, that is(are) selected more 
> than once in clusterconfigmapping table: 
> falcon-startup.properties,zoo.cfg,zeppelin-config,zookeeper-log4j,tagsync-application-properties,storm-worker-log4j,hive-log4j2,admin-properties,oozie-env,ssl-server,hadoop-policy,infra-solr-log4j,oozie-log4j,core-site,mahout-log4j,kafka-broker,pig-log4j,livy-env,zeppelin-env,hive-log4j,slider-client,tez-env,storm-env,ranger-storm-plugin-properties,oozie-site,falcon-env,beeline-log4j2,ranger-storm-policymgr-ssl,falcon-runtime.properties,ranger-storm-security,hbase-policy,ams-env,capacity-scheduler,webhcat-log4j,atlas-log4j,livy-log4j-properties,spark-metrics-properties,hbase-env,ranger-atlas-security,flume-env,sqoop-atlas-application.properties,admin-log4j,hiveserver2-interactive-site,ranger-tagsync-site,application-properties,ranger-kafka-policymgr-ssl,ranger-admin-site,ranger-env,ranger-storm-audit,ranger-hbase-policymgr-ssl,hdfs-log4j
> {noformat}
> 
> It is very difficult to fix these kind of issues with queries. including auto 
> fix for this will be very helpful.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/DatabaseConsistencyCheckHelper.java
>  575485bd54 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterDAO.java 
> a23b914f34 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java
>  9bf03b343f 
>   
> ambari-server/src/test/java/org/apache/ambari/server/checks/DatabaseConsistencyCheckHelperTest.java
>  6c6c00f947 
> 
> 
> Diff: https://reviews.apache.org/r/62245/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>

Re: Review Request 62125: AMBARI-21893 : NameNode Heap Usage (Daily) metric alert status flips to UNKNOWN intermittently when AMS HTTPS is enabled.

2017-09-12 Thread Aravindan Vijayan

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62125/
---

(Updated Sept. 12, 2017, 7:21 p.m.)

Review request for Ambari, Dmytro Sen, Jayush Luniya, and Sid Wagle.

Bugs: AMBARI-21893
https://issues.apache.org/jira/browse/AMBARI-21893

Repository: ambari

Description
---

PROBLEM
This issue is because of AMS HTTPS + HA scenario. The CA cert file
(/etc/ambari-metrics-monitor/conf/ca.pem) found on every host is generated by
fetching a specific metric collector host's certificate from the truststore.
This certificate file is being used by alert script, service check and even
monitors to talk to collector.

For example, in a cluster with hosts H1 to H5, let's say there are 2 collectors
- H1 & H2.
On a node H3, let's say the ca.pem file was constructed using the certificate
for collector H2.
Service check or metric monitor on H3 will NOT be able to talk to H1 since it
does not have the certificate for that host.

FIX
Import certificates from metric collectors onto the pem file, if
metric_truststore_alias is not specified. If the cluster has multiple
collectors and the same certificate is used for all the collectors, then the
metric_truststore_alias must be used to specify the alias to import the
certificate.

56 matches

Mail list logo