date:20160621


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49033/#review139018
---


Ship it!




Ship It!

- Sebastian Toader


On June 21, 2016, 9:14 p.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49033/
> ---
> 
> (Updated June 21, 2016, 9:14 p.m.)
> 
> 
> Review request for Ambari, Don Bosco Durai, Miklos Gergely, Robert Nettleton, 
> Sumit Mohanty, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-17344
> https://issues.apache.org/jira/browse/AMBARI-17344
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> /security.json needs to be cleared if kerberos is disabled. can be a problem 
> if a cluster is kerberized, then kerberos is disabled after that
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logsearch_solr.py
>  33ba70c 
>   ambari-server/src/test/python/stacks/2.4/LOGSEARCH/test_solr.py 0474aed 
> 
> Diff: https://reviews.apache.org/r/49033/diff/
> 
> 
> Testing
> ---
> 
> Total run:1073
> Total errors:0
> Total failures:0
> 
> 
> Thanks,
> 
> Oliver Szabo
> 
>

Re: Review Request 48395: AMBARI-17027: Metrics Collector API: Introduce basic series aggregation functions

2016-06-21 Thread Jungtaek Lim



> On 6 21, 2016, 4:40 오후, Prajwal Rao wrote:
> > ambari-metrics/ambari-metrics-grafana/ambari-metrics/partials/query.editor.html,
> >  line 112
> > 
> >
> > Visually, I think it makes more sense to add this after "precision" as 
> > it seems to break the first row.

Addressed from third patch.


> On 6 21, 2016, 4:40 오후, Prajwal Rao wrote:
> > ambari-metrics/ambari-metrics-grafana/ambari-metrics/queryCtrl.js, line 39
> > 
> >
> > You will need to accommodate for seriesAggregator not being set at all 
> > (undefined), for example when dashboards are imported without the 
> > seriesAggregator in them, it will throw up an error.
> > 
> > You can rememdy this by doing a function similar to $scope.trasnform.

Thanks for finding missed spot. Addressed from third patch.


- Jungtaek


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48395/#review138870
---


On 6 22, 2016, 2:09 오전, Jungtaek Lim wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48395/
> ---
> 
> (Updated 6 22, 2016, 2:09 오전)
> 
> 
> Review request for Ambari, Aravindan Vijayan, Dmytro Sen, Prajwal Rao, 
> Sriharsha Chintalapani, Sid Wagle, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-17027
> https://issues.apache.org/jira/browse/AMBARI-17027
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMS doesn't provide tag so metric is identified by appId, metric name, 
> hostname, instanceId. In this situation metric name is normally consist of 
> origin metric name and tag values, like graphite, but unlike Graphite, AMS 
> also doesn't provide series aggregation functions so aggregation should be 
> done from caller side.
> 
> It would be great if Ambari Metrics Collector provides series aggregation 
> functions, like sumSeries / 
> averageSeries / minSeries / maxSeries on Graphite.
> 
> Query outputs: 
> https://gist.github.com/HeartSaVioR/f4f28b5b8b7bf2e5477e59d7fd56090f
> 
> Attached Grafana screenshots to AMBARI-17027. Please refer 
> https://issues.apache.org/jira/browse/AMBARI-17027 for details.
> 
> 
> Diffs
> -
> 
>   ambari-metrics/ambari-metrics-grafana/ambari-metrics/datasource.js b825774 
>   
> ambari-metrics/ambari-metrics-grafana/ambari-metrics/partials/query.editor.html
>  b034c03 
>   ambari-metrics/ambari-metrics-grafana/ambari-metrics/queryCtrl.js 2eb3613 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/HBaseTimelineMetricStore.java
>  1b2d02f 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStore.java
>  e37bc4d 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStoreWatcher.java
>  7d49070 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/AbstractTimelineMetricsSeriesAggregateFunction.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/SeriesAggregateFunction.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAggregateFunction.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAggregateFunctionFactory.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAvgAggregateFunction.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesMaxAggregateFunction.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesMinAggregateFunction.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesSumAggregateFunction.java
>

Re: Review Request 48395: AMBARI-17027: Metrics Collector API: Introduce basic series aggregation functions

2016-06-21 Thread Jungtaek Lim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48395/
---

(Updated 6 22, 2016, 2:09 오전)


Review request for Ambari, Aravindan Vijayan, Dmytro Sen, Prajwal Rao, 
Sriharsha Chintalapani, Sid Wagle, and Yusaku Sako.


Changes
---

Address review


Bugs: AMBARI-17027
https://issues.apache.org/jira/browse/AMBARI-17027


Repository: ambari


Description
---

AMS doesn't provide tag so metric is identified by appId, metric name, 
hostname, instanceId. In this situation metric name is normally consist of 
origin metric name and tag values, like graphite, but unlike Graphite, AMS also 
doesn't provide series aggregation functions so aggregation should be done from 
caller side.

It would be great if Ambari Metrics Collector provides series aggregation 
functions, like sumSeries / 
averageSeries / minSeries / maxSeries on Graphite.

Query outputs: 
https://gist.github.com/HeartSaVioR/f4f28b5b8b7bf2e5477e59d7fd56090f

Attached Grafana screenshots to AMBARI-17027. Please refer 
https://issues.apache.org/jira/browse/AMBARI-17027 for details.


Diffs (updated)
-

  ambari-metrics/ambari-metrics-grafana/ambari-metrics/datasource.js b825774 
  
ambari-metrics/ambari-metrics-grafana/ambari-metrics/partials/query.editor.html 
b034c03 
  ambari-metrics/ambari-metrics-grafana/ambari-metrics/queryCtrl.js 2eb3613 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/HBaseTimelineMetricStore.java
 1b2d02f 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStore.java
 e37bc4d 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStoreWatcher.java
 7d49070 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/AbstractTimelineMetricsSeriesAggregateFunction.java
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/SeriesAggregateFunction.java
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAggregateFunction.java
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAggregateFunctionFactory.java
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAvgAggregateFunction.java
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesMaxAggregateFunction.java
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesMinAggregateFunction.java
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesSumAggregateFunction.java
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/TimelineWebServices.java
 ee3a097 
  
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TestTimelineMetricStore.java
 cfd1f58 
  
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStoreWatcherTest.java
 a94f4c5 
  
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAggregateFunctionTest.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/48395/diff/


Testing
---

> mvn clean install

> cd ambari-metrics

> mvn test

```
[INFO] 
[INFO] Reactor Summary:
[INFO]
[INFO] ambari-metrics . SUCCESS [  1.064 s]
[INFO] Ambari Metrics Common .. SUCCESS [ 13.266 s]
[INFO] Ambari Metrics Hadoop Sink . SUCCESS [  4.746 s]
[INFO] Ambari Metrics Flume Sink .. SUCCESS [  6.594 s]
[INFO] Ambari Metrics Kafka Sink

Re: Review Request 47941: [AMBARI-16920] Follow up issue for Spark2 stack definition

2016-06-21 Thread Jeff Zhang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47941/
---

(Updated June 22, 2016, 12:06 a.m.)


Review request for Ambari, Jayush Luniya and Sumit Mohanty.


Changes
---

update the unit test


Bugs: AMBARI-16920
https://issues.apache.org/jira/browse/AMBARI-16920


Repository: ambari


Description
---

This a followup ticket for add spark2 stack definition. There's serveral issues:
1.  Spark2 thrift server can not started due to miss of 
spark-thrift-fairscheduler.xml
2.  Miss of add spark2 cache file in copy_barball.py
3.  Miss the role_commnad_order of spark2
4.  conf_select is missign for spark2


Diffs (updated)
-

  
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
 4eb0015 
  
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
 286df8d 
  
ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py
 be99edd 
  
ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_service.py
 c2385df 
  ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json 
f6011b0 
  
ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py
 6c7fe18 

Diff: https://reviews.apache.org/r/47941/diff/


Testing
---

Manually verified.


Thanks,

Jeff Zhang

Review Request 49050: spark.driver.extraJavaOptions and spark.yarn.am.extraJavaOptions property is set to empty

2016-06-21 Thread Weiqing Yang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49050/
---

Review request for Ambari, Zhe (Joe) Wang, Sumit Mohanty, and Srimanth Gunturi.


Bugs: AMBARI-17349
https://issues.apache.org/jira/browse/AMBARI-17349


Repository: ambari


Description
---

Currently spark.driver.extraJavaOptions and spark.yarn.am.extraJavaOptions is 
set to empty.
In this patch, set spark.driver.extraJavaOptions and 
spark.yarn.am.extraJavaOptions to default value "none".


Diffs
-

  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
 84eb805 

Diff: https://reviews.apache.org/r/49050/diff/


Testing
---

Tests passed in local cluster.


Thanks,

Weiqing Yang

Re: Review Request 48734: App timeline Server start fails on enabling HA because namenode is in safemode

2016-06-21 Thread Victor Galgo



> On June 21, 2016, 8:48 p.m., Jonathan Hurley wrote:
> > Ship It!

Jonathan can please do the honours of helping to commit this patch?


- Victor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48734/#review138935
---


On June 17, 2016, 10:45 p.m., Victor Galgo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48734/
> ---
> 
> (Updated June 17, 2016, 10:45 p.m.)
> 
> 
> Review request for Ambari, Andriy Babiichuk, Alexandr Antonenko, Andrew 
> Onischuk, Di Li, Dmitro Lisnichenko, Jonathan Hurley, Jayush Luniya, Robert 
> Levas, Sandor Magyari, Sumit Mohanty, Sebastian Toader, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-17182
> https://issues.apache.org/jira/browse/AMBARI-17182
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> On the last step "Start all" on enabling HA below happens:
> 
> Traceback (most recent call last):
> File 
> "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py",
>  line 147, in 
>   ApplicationTimelineServer().execute()
> File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 219, in execute
>   method(env)
> File 
> "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py",
>  line 43, in start
>   self.configure(env) # FOR SECURITY
> File 
> "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py",
>  line 54, in configure
>   yarn(name='apptimelineserver')
> File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", 
> line 89, in thunk
>   return fn(*args, **kwargs)
> File 
> "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py",
>  line 276, in yarn
>   mode=0755
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", 
> line 154, in __init__
>   self.env.run()
> File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 160, in run
>   self.run_action(resource, action)
> File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 124, in run_action
>   provider_action()
> File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py",
>  line 463, in action_create_on_execute
>   self.action_delayed("create")
> File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py",
>  line 460, in action_delayed
>   self.get_hdfs_resource_executor().action_delayed(action_name, self)
> File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py",
>  line 259, in action_delayed
>   self._set_mode(self.target_status)
> File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py",
>  line 366, in _set_mode
>   self.util.run_command(self.main_resource.resource.target, 
> 'SETPERMISSION', method='PUT', permission=self.mode, assertable_result=False)
> File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py",
>  line 195, in run_command
>   raise Fail(err_msg)
>   resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w 
> '%{http_code}' -X PUT 
> 'http://testvgalgo.org:50070/webhdfs/v1/ats/done?op=SETPERMISSION=hdfs=755''
>  returned status_code=403. 
>   {
> "RemoteException": {
>   "exception": "RetriableException", 
>   "javaClassName": "org.apache.hadoop.ipc.RetriableException", 
>   "message": "org.apache.hadoop.hdfs.server.namenode.SafeModeException: 
> Cannot set permission for /ats/done. Name node is in safe mode.\nThe reported 
> blocks 675 needs additional 16 blocks to reach the threshold 0.9900 of total 
> blocks 697.\nThe number of live datanodes 20 has reached the minimum number 
> 0. Safe mode will be turned off automatically once the thresholds have been 
> reached."
> }
>   }
>   
>   
> This happens because NN is not yet out of safemode at the moment of ats 
> start, because DNs just started.
> 
> To fix this "stop namenodes" has to be triggered before "start all".
> 
> If this is done, on "Start all" it will be ensured that datanodes start prior 
> to NN, and that NN are out of safemode before ATS start.
> 
> 
> Diffs
> -
> 
>   
> ambari-web/app/controllers/main/admin/highAvailability/nameNode/step9_controller.js
>  24677e4 
>   ambari-web/app/messages.js 6465812 
> 
> Diff: https://reviews.apache.org/r/48734/diff/
> 
> 
> Testing
> ---
> 
> Calling set on destroyed

Re: Review Request 48681: Ambari views logs should be more verbose and should include info/error/debug logs at appropriate places

2016-06-21 Thread Rohit Choudhary


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48681/#review138940
---


Ship it!




Ship It!

- Rohit Choudhary


On June 14, 2016, 10:20 a.m., Nitiraj Rathore wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48681/
> ---
> 
> (Updated June 14, 2016, 10:20 a.m.)
> 
> 
> Review request for Ambari, DIPAYAN BHOWMICK, Gaurav Nagar, Pallav 
> Kulshreshtha, Rohit Choudhary, and Ashwin Rajeev.
> 
> 
> Bugs: AMBARI-17214
> https://issues.apache.org/jira/browse/AMBARI-17214
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Added logs at various places.
> 
> 
> Diffs
> -
> 
>   
> contrib/views/capacity-scheduler/src/main/java/org/apache/ambari/view/capacityscheduler/ConfigurationService.java
>  2198331 
>   
> contrib/views/files/src/main/java/org/apache/ambari/view/filebrowser/DownloadService.java
>  4b8a546 
>   
> contrib/views/files/src/main/java/org/apache/ambari/view/filebrowser/FilePreviewService.java
>  3585516 
>   
> contrib/views/files/src/main/java/org/apache/ambari/view/filebrowser/PropertyValidator.java
>  2ad779c 
>   
> contrib/views/hive/src/main/java/org/apache/ambari/view/hive/backgroundjobs/BackgroundJobController.java
>  2f5c76c 
>   
> contrib/views/hive/src/main/java/org/apache/ambari/view/hive/client/UserLocalConnection.java
>  a86c84f 
>   
> contrib/views/pig/src/main/java/org/apache/ambari/view/pig/PropertyValidator.java
>  cd54690 
>   
> contrib/views/pig/src/main/java/org/apache/ambari/view/pig/resources/files/FileService.java
>  40bc9a7 
>   
> contrib/views/pig/src/main/java/org/apache/ambari/view/pig/resources/jobs/JobService.java
>  9ecbd75 
>   
> contrib/views/pig/src/main/java/org/apache/ambari/view/pig/resources/jobs/models/PigJob.java
>  6f80d6b 
>   
> contrib/views/pig/src/main/java/org/apache/ambari/view/pig/resources/scripts/ScriptService.java
>  46e6247 
>   
> contrib/views/pig/src/main/java/org/apache/ambari/view/pig/templeton/client/JSONRequest.java
>  39a595b 
>   
> contrib/views/pig/src/main/java/org/apache/ambari/view/pig/templeton/client/TempletonApi.java
>  66568d7 
> 
> Diff: https://reviews.apache.org/r/48681/diff/
> 
> 
> Testing
> ---
> 
> Manual testing done.
> 
> 
> Thanks,
> 
> Nitiraj Rathore
> 
>

Re: Review Request 48734: App timeline Server start fails on enabling HA because namenode is in safemode


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48734/#review138935
---


Ship it!




Ship It!

- Jonathan Hurley


On June 17, 2016, 6:45 p.m., Victor Galgo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48734/
> ---
> 
> (Updated June 17, 2016, 6:45 p.m.)
> 
> 
> Review request for Ambari, Andriy Babiichuk, Alexandr Antonenko, Andrew 
> Onischuk, Di Li, Dmitro Lisnichenko, Jonathan Hurley, Jayush Luniya, Robert 
> Levas, Sandor Magyari, Sumit Mohanty, Sebastian Toader, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-17182
> https://issues.apache.org/jira/browse/AMBARI-17182
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> On the last step "Start all" on enabling HA below happens:
> 
> Traceback (most recent call last):
> File 
> "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py",
>  line 147, in 
>   ApplicationTimelineServer().execute()
> File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 219, in execute
>   method(env)
> File 
> "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py",
>  line 43, in start
>   self.configure(env) # FOR SECURITY
> File 
> "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py",
>  line 54, in configure
>   yarn(name='apptimelineserver')
> File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", 
> line 89, in thunk
>   return fn(*args, **kwargs)
> File 
> "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py",
>  line 276, in yarn
>   mode=0755
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", 
> line 154, in __init__
>   self.env.run()
> File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 160, in run
>   self.run_action(resource, action)
> File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 124, in run_action
>   provider_action()
> File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py",
>  line 463, in action_create_on_execute
>   self.action_delayed("create")
> File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py",
>  line 460, in action_delayed
>   self.get_hdfs_resource_executor().action_delayed(action_name, self)
> File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py",
>  line 259, in action_delayed
>   self._set_mode(self.target_status)
> File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py",
>  line 366, in _set_mode
>   self.util.run_command(self.main_resource.resource.target, 
> 'SETPERMISSION', method='PUT', permission=self.mode, assertable_result=False)
> File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py",
>  line 195, in run_command
>   raise Fail(err_msg)
>   resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w 
> '%{http_code}' -X PUT 
> 'http://testvgalgo.org:50070/webhdfs/v1/ats/done?op=SETPERMISSION=hdfs=755''
>  returned status_code=403. 
>   {
> "RemoteException": {
>   "exception": "RetriableException", 
>   "javaClassName": "org.apache.hadoop.ipc.RetriableException", 
>   "message": "org.apache.hadoop.hdfs.server.namenode.SafeModeException: 
> Cannot set permission for /ats/done. Name node is in safe mode.\nThe reported 
> blocks 675 needs additional 16 blocks to reach the threshold 0.9900 of total 
> blocks 697.\nThe number of live datanodes 20 has reached the minimum number 
> 0. Safe mode will be turned off automatically once the thresholds have been 
> reached."
> }
>   }
>   
>   
> This happens because NN is not yet out of safemode at the moment of ats 
> start, because DNs just started.
> 
> To fix this "stop namenodes" has to be triggered before "start all".
> 
> If this is done, on "Start all" it will be ensured that datanodes start prior 
> to NN, and that NN are out of safemode before ATS start.
> 
> 
> Diffs
> -
> 
>   
> ambari-web/app/controllers/main/admin/highAvailability/nameNode/step9_controller.js
>  24677e4 
>   ambari-web/app/messages.js 6465812 
> 
> Diff: https://reviews.apache.org/r/48734/diff/
> 
> 
> Testing
> ---
> 
> Calling set on destroyed view
> Calling set on destroyed view
> Calling set on destroyed view
> Calling set on destroyed view
> 
>

Re: Review Request 48805: AMBARI-17280. RU to write out client configs that are dependencies of Hive, ATS, and Oozie during upgrades that change configs


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48805/#review138932
---


Ship it!




Ship It!

- Jonathan Hurley


On June 21, 2016, 4:02 p.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48805/
> ---
> 
> (Updated June 21, 2016, 4:02 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, 
> Jonathan Hurley, Nate Cole, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-17280
> https://issues.apache.org/jira/browse/AMBARI-17280
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> During RU, HiveServer2 is restarted but the newer tez configs have not yet 
> been saved, which is incorrect because Hive has a dependency on Tez.
> This is important when configs change during a major stack upgrade, e.g., HDP 
> 2.4 -> 2.5. What happens today is,
> 
> * Install packages generates /etc/tez/2.5.0.0-1/0 and copies the configs from 
> /etc/tez/2.4.0.0-1/0/ to the new folder
> * If configs change during RU, then Hive is restarted and the classpath means 
> that it will pick up the older tez configs from the new /etc/tez/2.5.0.0-1/0 
> folder
> 
> 
> This problem exists for all of these components:
> 
> * HiveServer: depends on Tez and MapReduce clients
> * ATS: depends on Tez and Spark clients
> * Oozie: depends on Tez, Spark, and MapReduce clients
> 
> This problem only exists when configs change (so crossing major stack 
> version) and during RU (because it is allowed to change configs during the 
> middle of restarting services).
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
>  4eb0015 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
>  b994fce 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> 49dcb4e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
>  fb3ae69 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  63c72f7 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/spark_client.py
>  ef41453 
>   
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py
>  44239c7 
>   
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez.py
>  67466e3 
>   
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py
>  c79d63b 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/mapreduce2_client.py
>  db22004 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
>  90f885a 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py
>  d1ec15b 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> 4187d64 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> 3461ad4 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml 
> 426b452 
>   ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py e53eb4b 
> 
> Diff: https://reviews.apache.org/r/48805/diff/
> 
> 
> Testing
> ---
> 
> Verified during RU from HDP 2.4 to 2.5 with ATS, Hive, Tez, Oozie, and Spark
> 
> Python unit tests passed,
> --
> Total run:1062
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Re: Review Request 48507: AMBARI-17074: Expose Spark daemon memory in Spark2

2016-06-21 Thread Srimanth Gunturi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48507/#review138929
---


Ship it!




Ship It!

- Srimanth Gunturi


On June 21, 2016, 6:18 p.m., Weiqing Yang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48507/
> ---
> 
> (Updated June 21, 2016, 6:18 p.m.)
> 
> 
> Review request for Ambari, Zhe (Joe) Wang, Sumit Mohanty, and Srimanth 
> Gunturi.
> 
> 
> Bugs: AMBARI-17074
> https://issues.apache.org/jira/browse/AMBARI-17074
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Expose Spark daemon memory in Spark2, so that the user can modify its size on 
> ambari web UI easily. The change is the same with the patch of jira 
> AMBARI-16757, which is for Spark 1.6.1.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/configuration/spark2-env.xml
>  76c670a 
>   
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py
>  6925ab5 
> 
> Diff: https://reviews.apache.org/r/48507/diff/
> 
> 
> Testing
> ---
> 
> Tests manually passed.
> 
> 
> Thanks,
> 
> Weiqing Yang
> 
>

Re: Review Request 49014: AMBARI-17331. Determine Tez for Hive2 config 'tez.am.resource.memory.mb' based on cluster capacity.

2016-06-21 Thread Siddharth Seth


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49014/#review138924
---


Ship it!




Looks good. The Xmx change isn't required since it's not set for tez either 
(and tez will set this to 80% of the container size in this case)

- Siddharth Seth


On June 21, 2016, 7:59 p.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49014/
> ---
> 
> (Updated June 21, 2016, 7:59 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17331
> https://issues.apache.org/jira/browse/AMBARI-17331
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The current computations done for tez ends up with a large AM size.
> For LLAP, we'd like stack-advisor to change these computations.
> 
> - Total Cluster Memory <=4GB - Tez AM size = 256MB and then normalized based 
> on YARN minimum container size.
> - Total Cluster Memory >4GB && <= 72GB - Tez AM size = 512MB and then 
> normalized based on YARN minimum container size.
> - Total Cluster Memory >72GB - Tez AM size = 1536MB and then normalized based 
> on YARN minimum container size.
> 
> 
> Fix for safeguaring 'llap' Queue creation to max. size of 100%, in case the 
> minimum size required caluclated size is more than 100%.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/tez-interactive-site.xml
>  3c83c5c 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 76654c3 
>   ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 
> 1bc53ea 
> 
> Diff: https://reviews.apache.org/r/49014/diff/
> 
> 
> Testing
> ---
> 
> Python UT added.
>   - Python UT passes.
> 
> 
> Thanks,
> 
> Swapan Shridhar
> 
>

Re: Review Request 48805: AMBARI-17280. RU to write out client configs that are dependencies of Hive, ATS, and Oozie during upgrades that change configs


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48805/#review138923
---




ambari-common/src/main/python/resource_management/libraries/script/script.py 
(line 158)


Renamed this.



ambari-common/src/main/python/resource_management/libraries/script/script.py 
(line 168)


Kept this since it's mainly meant to help developers.



ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml (line 
148)


This is much simpler now.


- Alejandro Fernandez


On June 21, 2016, 8:02 p.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48805/
> ---
> 
> (Updated June 21, 2016, 8:02 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, 
> Jonathan Hurley, Nate Cole, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-17280
> https://issues.apache.org/jira/browse/AMBARI-17280
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> During RU, HiveServer2 is restarted but the newer tez configs have not yet 
> been saved, which is incorrect because Hive has a dependency on Tez.
> This is important when configs change during a major stack upgrade, e.g., HDP 
> 2.4 -> 2.5. What happens today is,
> 
> * Install packages generates /etc/tez/2.5.0.0-1/0 and copies the configs from 
> /etc/tez/2.4.0.0-1/0/ to the new folder
> * If configs change during RU, then Hive is restarted and the classpath means 
> that it will pick up the older tez configs from the new /etc/tez/2.5.0.0-1/0 
> folder
> 
> 
> This problem exists for all of these components:
> 
> * HiveServer: depends on Tez and MapReduce clients
> * ATS: depends on Tez and Spark clients
> * Oozie: depends on Tez, Spark, and MapReduce clients
> 
> This problem only exists when configs change (so crossing major stack 
> version) and during RU (because it is allowed to change configs during the 
> middle of restarting services).
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
>  4eb0015 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
>  b994fce 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> 49dcb4e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
>  fb3ae69 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  63c72f7 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/spark_client.py
>  ef41453 
>   
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py
>  44239c7 
>   
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez.py
>  67466e3 
>   
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py
>  c79d63b 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/mapreduce2_client.py
>  db22004 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
>  90f885a 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py
>  d1ec15b 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> 4187d64 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> 3461ad4 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml 
> 426b452 
>   ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py e53eb4b 
> 
> Diff: https://reviews.apache.org/r/48805/diff/
> 
> 
> Testing
> ---
> 
> Verified during RU from HDP 2.4 to 2.5 with ATS, Hive, Tez, Oozie, and Spark
> 
> Python unit tests passed,
> --
> Total run:1062
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Re: Review Request 48805: AMBARI-17280. RU to write out client configs that are dependencies of Hive, ATS, and Oozie during upgrades that change configs


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48805/
---

(Updated June 21, 2016, 8:02 p.m.)


Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, Jonathan 
Hurley, Nate Cole, and Tim Thorpe.


Bugs: AMBARI-17280
https://issues.apache.org/jira/browse/AMBARI-17280


Repository: ambari


Description
---

During RU, HiveServer2 is restarted but the newer tez configs have not yet been 
saved, which is incorrect because Hive has a dependency on Tez.
This is important when configs change during a major stack upgrade, e.g., HDP 
2.4 -> 2.5. What happens today is,

* Install packages generates /etc/tez/2.5.0.0-1/0 and copies the configs from 
/etc/tez/2.4.0.0-1/0/ to the new folder
* If configs change during RU, then Hive is restarted and the classpath means 
that it will pick up the older tez configs from the new /etc/tez/2.5.0.0-1/0 
folder


This problem exists for all of these components:

* HiveServer: depends on Tez and MapReduce clients
* ATS: depends on Tez and Spark clients
* Oozie: depends on Tez, Spark, and MapReduce clients

This problem only exists when configs change (so crossing major stack version) 
and during RU (because it is allowed to change configs during the middle of 
restarting services).


Diffs (updated)
-

  
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
 4eb0015 
  
ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
 b994fce 
  ambari-common/src/main/python/resource_management/libraries/script/script.py 
49dcb4e 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
 fb3ae69 
  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
 63c72f7 
  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/spark_client.py
 ef41453 
  
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py
 44239c7 
  
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez.py
 67466e3 
  
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py
 c79d63b 
  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/mapreduce2_client.py
 db22004 
  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
 90f885a 
  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py
 d1ec15b 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
4187d64 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
3461ad4 
  ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml 
426b452 
  ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py e53eb4b 

Diff: https://reviews.apache.org/r/48805/diff/


Testing
---

Verified during RU from HDP 2.4 to 2.5 with ATS, Hive, Tez, Oozie, and Spark

Python unit tests passed,
--
Total run:1062
Total errors:0
Total failures:0
OK


Thanks,

Alejandro Fernandez

Re: Review Request 49014: AMBARI-17331. Determine Tez for Hive2 config 'tez.am.resource.memory.mb' based on cluster capacity.

2016-06-21 Thread Swapan Shridhar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49014/
---

(Updated June 21, 2016, 7:59 p.m.)


Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.


Changes
---

Updated Test cases.


Bugs: AMBARI-17331
https://issues.apache.org/jira/browse/AMBARI-17331


Repository: ambari


Description
---

The current computations done for tez ends up with a large AM size.
For LLAP, we'd like stack-advisor to change these computations.

- Total Cluster Memory <=4GB - Tez AM size = 256MB and then normalized based on 
YARN minimum container size.
- Total Cluster Memory >4GB && <= 72GB - Tez AM size = 512MB and then 
normalized based on YARN minimum container size.
- Total Cluster Memory >72GB - Tez AM size = 1536MB and then normalized based 
on YARN minimum container size.


Fix for safeguaring 'llap' Queue creation to max. size of 100%, in case the 
minimum size required caluclated size is more than 100%.


Diffs (updated)
-

  
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/tez-interactive-site.xml
 3c83c5c 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
76654c3 
  ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 1bc53ea 

Diff: https://reviews.apache.org/r/49014/diff/


Testing
---

Python UT added.
  - Python UT passes.


Thanks,

Swapan Shridhar

Re: Review Request 48805: AMBARI-17280. RU to write out client configs that are dependencies of Hive, ATS, and Oozie during upgrades that change configs


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48805/#review138921
---




ambari-common/src/main/python/resource_management/libraries/script/script.py 
(line 158)


Will do



ambari-common/src/main/python/resource_management/libraries/script/script.py 
(lines 168 - 171)


If I create constants for the name of these variables then a change in the 
variable names would still require changing the constant values.

This is more of a sanity check so the developer reading this can quickly 
pinpoint what the required variables are.

Will drop this issue.



ambari-common/src/main/python/resource_management/libraries/script/script.py 
(line 175)


Will fix.



ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java
 (lines 104 - 111)


Will remove this annotation.



ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml (line 
153)


Will keep it simple with just 3 excute tasks, no need to check the 
"if_component_exists".


- Alejandro Fernandez


On June 21, 2016, 1:12 a.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48805/
> ---
> 
> (Updated June 21, 2016, 1:12 a.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, 
> Jonathan Hurley, Nate Cole, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-17280
> https://issues.apache.org/jira/browse/AMBARI-17280
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> During RU, HiveServer2 is restarted but the newer tez configs have not yet 
> been saved, which is incorrect because Hive has a dependency on Tez.
> This is important when configs change during a major stack upgrade, e.g., HDP 
> 2.4 -> 2.5. What happens today is,
> 
> * Install packages generates /etc/tez/2.5.0.0-1/0 and copies the configs from 
> /etc/tez/2.4.0.0-1/0/ to the new folder
> * If configs change during RU, then Hive is restarted and the classpath means 
> that it will pick up the older tez configs from the new /etc/tez/2.5.0.0-1/0 
> folder
> 
> 
> This problem exists for all of these components:
> 
> * HiveServer: depends on Tez and MapReduce clients
> * ATS: depends on Tez and Spark clients
> * Oozie: depends on Tez, Spark, and MapReduce clients
> 
> This problem only exists when configs change (so crossing major stack 
> version) and during RU (because it is allowed to change configs during the 
> middle of restarting services).
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
>  4eb0015 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
>  b994fce 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> 49dcb4e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
>  fb3ae69 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java
>  80bb26c 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  63c72f7 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/spark_client.py
>  ef41453 
>   
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py
>  44239c7 
>   
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez.py
>  67466e3 
>   
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py
>  c79d63b 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/mapreduce2_client.py
>  db22004 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
>  90f885a 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py
>  d1ec15b 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> 4187d64 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> 3461ad4 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml 
> 426b452 
>   ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py e53eb4b 
> 
> Diff: https://reviews.apache.org/r/48805/diff/
> 
> 
> Testing
> ---
> 
> Verified during RU from HDP 2.4 to 2.5 with ATS, Hive, Tez, Oozie, and Spark
> 
> Python

Re: Review Request 49036: Allow https prtotocol for Log Search

2016-06-21 Thread Robert Nettleton



> On June 21, 2016, 7:28 p.m., Robert Nettleton wrote:
> > This looks fine to me.
> > 
> > One question: Have you tested out the Ambari UI integration once HTTPS is 
> > enabled?  I would expect this to fail, since this configuration option did 
> > not exist when the integration code was written.
> > 
> > This patch does not need to be held up due to this, just wanted to make 
> > sure people are aware that the integration layer will have to uptake this 
> > new configuration option before the UI integration will work properly with 
> > HTTPS. I can implement that during my next set of LogSearch Integration 
> > changes. 
> > 
> > Thanks.
> 
> Oliver Szabo wrote:
> you mean the REST API part? because maybe you are right about that. (from 
> the other side, its tested with SSL enabled Ambari UI). 
> we have now some variations here based on these:
> - ssl enabled for Solr
> - ssl enabled for LogSearch
> - ssl enabled for Ambari

Exactly, the REST API part is the area that will not likely work properly if 
HTTPS is enabled in the LogSearch server.  I'll just have to update the 
integration code to read the new config properties to establish the HTTP 
connection, rather than HTTP.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49036/#review138916
---


On June 21, 2016, 6:03 p.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49036/
> ---
> 
> (Updated June 21, 2016, 6:03 p.m.)
> 
> 
> Review request for Ambari, Oliver Szabo, Robert Nettleton, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17348
> https://issues.apache.org/jira/browse/AMBARI-17348
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add a property to choose whether or not the Log Search UI should run using 
> https.
> 
> 
> Diffs
> -
> 
>   ambari-logsearch/ambari-logsearch-logfeeder/src/main/scripts/run.sh 376dfee 
>   
> ambari-logsearch/ambari-logsearch-portal/src/main/java/org/apache/ambari/logsearch/LogSearch.java
>  993c532 
>   ambari-logsearch/ambari-logsearch-portal/src/main/scripts/run.sh 6c687b9 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logfeeder-env.xml
>  26a303c 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logsearch-env.xml
>  e0d42d3 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logsearch-solr-env.xml
>  155ff04 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/params.py
>  d21a97f 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/service_check.py
>  2993190 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logsearch.properties.j2
>  3f5db30 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/logfeeder-env.sh.j2
>  aba06c6 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/logsearch-env.sh.j2
>  7b0aed3 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/logsearch-solr-env.sh.j2
>  2b17e63 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/quicklinks/quicklinks.json
>  fc71dfc 
>   ambari-server/src/test/python/stacks/2.4/configs/default.json 848be40 
>   ambari-web/app/data/HDP2/site_properties.js 0262211 
> 
> Diff: https://reviews.apache.org/r/49036/diff/
> 
> 
> Testing
> ---
> 
> Tested on local cluster, works fine with every combination of with/without 
> SSL Solr connect, and http/https protocol for Log Search UI.
> 
> ambari-server:
> OK
> --
> Total run:1072
> Total errors:0
> Total failures:0
> 
> 
> Thanks,
> 
> Miklos Gergely
> 
>

Re: Review Request 49036: Allow https prtotocol for Log Search



> On June 21, 2016, 7:28 p.m., Robert Nettleton wrote:
> > This looks fine to me.
> > 
> > One question: Have you tested out the Ambari UI integration once HTTPS is 
> > enabled?  I would expect this to fail, since this configuration option did 
> > not exist when the integration code was written.
> > 
> > This patch does not need to be held up due to this, just wanted to make 
> > sure people are aware that the integration layer will have to uptake this 
> > new configuration option before the UI integration will work properly with 
> > HTTPS. I can implement that during my next set of LogSearch Integration 
> > changes. 
> > 
> > Thanks.

you mean the REST API part? because maybe you are right about that. (from the 
other side, its tested with SSL enabled Ambari UI). 
we have now some variations here based on these:
- ssl enabled for Solr
- ssl enabled for LogSearch
- ssl enabled for Ambari


- Oliver


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49036/#review138916
---


On June 21, 2016, 6:03 p.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49036/
> ---
> 
> (Updated June 21, 2016, 6:03 p.m.)
> 
> 
> Review request for Ambari, Oliver Szabo, Robert Nettleton, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17348
> https://issues.apache.org/jira/browse/AMBARI-17348
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add a property to choose whether or not the Log Search UI should run using 
> https.
> 
> 
> Diffs
> -
> 
>   ambari-logsearch/ambari-logsearch-logfeeder/src/main/scripts/run.sh 376dfee 
>   
> ambari-logsearch/ambari-logsearch-portal/src/main/java/org/apache/ambari/logsearch/LogSearch.java
>  993c532 
>   ambari-logsearch/ambari-logsearch-portal/src/main/scripts/run.sh 6c687b9 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logfeeder-env.xml
>  26a303c 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logsearch-env.xml
>  e0d42d3 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logsearch-solr-env.xml
>  155ff04 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/params.py
>  d21a97f 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/service_check.py
>  2993190 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logsearch.properties.j2
>  3f5db30 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/logfeeder-env.sh.j2
>  aba06c6 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/logsearch-env.sh.j2
>  7b0aed3 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/logsearch-solr-env.sh.j2
>  2b17e63 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/quicklinks/quicklinks.json
>  fc71dfc 
>   ambari-server/src/test/python/stacks/2.4/configs/default.json 848be40 
>   ambari-web/app/data/HDP2/site_properties.js 0262211 
> 
> Diff: https://reviews.apache.org/r/49036/diff/
> 
> 
> Testing
> ---
> 
> Tested on local cluster, works fine with every combination of with/without 
> SSL Solr connect, and http/https protocol for Log Search UI.
> 
> ambari-server:
> OK
> --
> Total run:1072
> Total errors:0
> Total failures:0
> 
> 
> Thanks,
> 
> Miklos Gergely
> 
>

Re: Review Request 49036: Allow https prtotocol for Log Search

2016-06-21 Thread Robert Nettleton


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49036/#review138916
---


Ship it!




This looks fine to me.

One question: Have you tested out the Ambari UI integration once HTTPS is 
enabled?  I would expect this to fail, since this configuration option did not 
exist when the integration code was written.

This patch does not need to be held up due to this, just wanted to make sure 
people are aware that the integration layer will have to uptake this new 
configuration option before the UI integration will work properly with HTTPS. I 
can implement that during my next set of LogSearch Integration changes. 

Thanks.

- Robert Nettleton


On June 21, 2016, 6:03 p.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49036/
> ---
> 
> (Updated June 21, 2016, 6:03 p.m.)
> 
> 
> Review request for Ambari, Oliver Szabo, Robert Nettleton, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17348
> https://issues.apache.org/jira/browse/AMBARI-17348
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add a property to choose whether or not the Log Search UI should run using 
> https.
> 
> 
> Diffs
> -
> 
>   ambari-logsearch/ambari-logsearch-logfeeder/src/main/scripts/run.sh 376dfee 
>   
> ambari-logsearch/ambari-logsearch-portal/src/main/java/org/apache/ambari/logsearch/LogSearch.java
>  993c532 
>   ambari-logsearch/ambari-logsearch-portal/src/main/scripts/run.sh 6c687b9 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logfeeder-env.xml
>  26a303c 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logsearch-env.xml
>  e0d42d3 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logsearch-solr-env.xml
>  155ff04 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/params.py
>  d21a97f 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/service_check.py
>  2993190 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logsearch.properties.j2
>  3f5db30 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/logfeeder-env.sh.j2
>  aba06c6 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/logsearch-env.sh.j2
>  7b0aed3 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/logsearch-solr-env.sh.j2
>  2b17e63 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/quicklinks/quicklinks.json
>  fc71dfc 
>   ambari-server/src/test/python/stacks/2.4/configs/default.json 848be40 
>   ambari-web/app/data/HDP2/site_properties.js 0262211 
> 
> Diff: https://reviews.apache.org/r/49036/diff/
> 
> 
> Testing
> ---
> 
> Tested on local cluster, works fine with every combination of with/without 
> SSL Solr connect, and http/https protocol for Log Search UI.
> 
> ambari-server:
> OK
> --
> Total run:1072
> Total errors:0
> Total failures:0
> 
> 
> Thanks,
> 
> Miklos Gergely
> 
>

Re: Review Request 49014: AMBARI-17331. Determine Tez for Hive2 config 'tez.am.resource.memory.mb' based on cluster capacity.

2016-06-21 Thread Sumit Mohanty


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49014/#review138915
---


Ship it!




Ship It!

- Sumit Mohanty


On June 21, 2016, 7:10 p.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49014/
> ---
> 
> (Updated June 21, 2016, 7:10 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17331
> https://issues.apache.org/jira/browse/AMBARI-17331
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The current computations done for tez ends up with a large AM size.
> For LLAP, we'd like stack-advisor to change these computations.
> 
> - Total Cluster Memory <=4GB - Tez AM size = 256MB and then normalized based 
> on YARN minimum container size.
> - Total Cluster Memory >4GB && <= 72GB - Tez AM size = 512MB and then 
> normalized based on YARN minimum container size.
> - Total Cluster Memory >72GB - Tez AM size = 1536MB and then normalized based 
> on YARN minimum container size.
> 
> 
> Fix for safeguaring 'llap' Queue creation to max. size of 100%, in case the 
> minimum size required caluclated size is more than 100%.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/tez-interactive-site.xml
>  3c83c5c 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 76654c3 
>   ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 
> 1bc53ea 
> 
> Diff: https://reviews.apache.org/r/49014/diff/
> 
> 
> Testing
> ---
> 
> Python UT added.
>   - Python UT passes.
> 
> 
> Thanks,
> 
> Swapan Shridhar
> 
>

Re: Review Request 49033: Clear /security.json config on solr znode when kerberos is disabled.

2016-06-21 Thread Robert Nettleton


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49033/#review138912
---


Ship it!




Ship It!

- Robert Nettleton


On June 21, 2016, 7:14 p.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49033/
> ---
> 
> (Updated June 21, 2016, 7:14 p.m.)
> 
> 
> Review request for Ambari, Don Bosco Durai, Miklos Gergely, Robert Nettleton, 
> Sumit Mohanty, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-17344
> https://issues.apache.org/jira/browse/AMBARI-17344
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> /security.json needs to be cleared if kerberos is disabled. can be a problem 
> if a cluster is kerberized, then kerberos is disabled after that
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logsearch_solr.py
>  33ba70c 
>   ambari-server/src/test/python/stacks/2.4/LOGSEARCH/test_solr.py 0474aed 
> 
> Diff: https://reviews.apache.org/r/49033/diff/
> 
> 
> Testing
> ---
> 
> Total run:1073
> Total errors:0
> Total failures:0
> 
> 
> Thanks,
> 
> Oliver Szabo
> 
>

Re: Review Request 49033: Clear /security.json config on solr znode when kerberos is disabled.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49033/
---

(Updated June 21, 2016, 7:14 p.m.)


Review request for Ambari, Don Bosco Durai, Miklos Gergely, Robert Nettleton, 
Sumit Mohanty, and Sebastian Toader.


Changes
---

- fix indents
- only 1 execute for ssl too


Bugs: AMBARI-17344
https://issues.apache.org/jira/browse/AMBARI-17344


Repository: ambari


Description
---

/security.json needs to be cleared if kerberos is disabled. can be a problem if 
a cluster is kerberized, then kerberos is disabled after that


Diffs (updated)
-

  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logsearch_solr.py
 33ba70c 
  ambari-server/src/test/python/stacks/2.4/LOGSEARCH/test_solr.py 0474aed 

Diff: https://reviews.apache.org/r/49033/diff/


Testing
---

Total run:1073
Total errors:0
Total failures:0


Thanks,

Oliver Szabo

Re: Review Request 49014: AMBARI-17331. Determine Tez for Hive2 config 'tez.am.resource.memory.mb' based on cluster capacity.

2016-06-21 Thread Swapan Shridhar

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49014/
---

(Updated June 21, 2016, 7:10 p.m.)

Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.

Changes
---

- Changed code to deposit/set normalized value for 'tez.am.resource.memory.mb'.
- Aded fix for safeguaring 'llap' Queue creation to max. size of 100%, in case
the minimum size required caluclated size is more than 100%.

Bugs: AMBARI-17331
https://issues.apache.org/jira/browse/AMBARI-17331

Repository: ambari

Description (updated)
---

The current computations done for tez ends up with a large AM size.
For LLAP, we'd like stack-advisor to change these computations.

- Total Cluster Memory <=4GB - Tez AM size = 256MB and then normalized based on
YARN minimum container size.
- Total Cluster Memory >4GB && <= 72GB - Tez AM size = 512MB and then
normalized based on YARN minimum container size.
- Total Cluster Memory >72GB - Tez AM size = 1536MB and then normalized based
on YARN minimum container size.

Fix for safeguaring 'llap' Queue creation to max. size of 100%, in case the
minimum size required caluclated size is more than 100%.

Diffs (updated)
-

ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/tez-interactive-site.xml
3c83c5c
ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py
76654c3
ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 1bc53ea

Diff: https://reviews.apache.org/r/49014/diff/

Testing
---

Python UT added.
- Python UT passes.

Thanks,

Swapan Shridhar

Re: Review Request 48734: App timeline Server start fails on enabling HA because namenode is in safemode



> On June 17, 2016, 5:44 p.m., Alejandro Fernandez wrote:
> > ambari-web/app/controllers/main/admin/highAvailability/nameNode/step9_controller.js,
> >  line 25
> > 
> >
> > How does this fix the issue? If NN just started, it still needs to get 
> > block reports, so ATS can still fail.
> 
> Victor Galgo wrote:
> Alejandro thanks for having a look!
> 
> This fixes the issue because when we do "Start All" later on, NNs start 
> is triggered before ATS start (role_command_order). And during NN start it 
> waits until safemode is off. To proceed with ATS and others.

I still prefer a custom command to wait for NN to leave safemode. A restart 
command has a timeout, so if it doesn't finish in time, a retry will stop NN 
again, which we don't want.
Further, a start command also has a timeout, and another start command may do a 
no-op since it's already started.
I think the logic is far cleaner and more maintainable with a custom command 
just for leaving safemode.


- Alejandro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48734/#review138263
---


On June 17, 2016, 10:45 p.m., Victor Galgo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48734/
> ---
> 
> (Updated June 17, 2016, 10:45 p.m.)
> 
> 
> Review request for Ambari, Andriy Babiichuk, Alexandr Antonenko, Andrew 
> Onischuk, Di Li, Dmitro Lisnichenko, Jonathan Hurley, Jayush Luniya, Robert 
> Levas, Sandor Magyari, Sumit Mohanty, Sebastian Toader, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-17182
> https://issues.apache.org/jira/browse/AMBARI-17182
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> On the last step "Start all" on enabling HA below happens:
> 
> Traceback (most recent call last):
> File 
> "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py",
>  line 147, in 
>   ApplicationTimelineServer().execute()
> File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 219, in execute
>   method(env)
> File 
> "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py",
>  line 43, in start
>   self.configure(env) # FOR SECURITY
> File 
> "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py",
>  line 54, in configure
>   yarn(name='apptimelineserver')
> File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", 
> line 89, in thunk
>   return fn(*args, **kwargs)
> File 
> "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py",
>  line 276, in yarn
>   mode=0755
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", 
> line 154, in __init__
>   self.env.run()
> File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 160, in run
>   self.run_action(resource, action)
> File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 124, in run_action
>   provider_action()
> File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py",
>  line 463, in action_create_on_execute
>   self.action_delayed("create")
> File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py",
>  line 460, in action_delayed
>   self.get_hdfs_resource_executor().action_delayed(action_name, self)
> File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py",
>  line 259, in action_delayed
>   self._set_mode(self.target_status)
> File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py",
>  line 366, in _set_mode
>   self.util.run_command(self.main_resource.resource.target, 
> 'SETPERMISSION', method='PUT', permission=self.mode, assertable_result=False)
> File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py",
>  line 195, in run_command
>   raise Fail(err_msg)
>   resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w 
> '%{http_code}' -X PUT 
> 'http://testvgalgo.org:50070/webhdfs/v1/ats/done?op=SETPERMISSION=hdfs=755''
>  returned status_code=403. 
>   {
> "RemoteException": {
>   "exception": "RetriableException", 
>   "javaClassName": "org.apache.hadoop.ipc.RetriableException", 
>   "message": "org.apache.hadoop.hdfs.server.namenode.SafeModeException: 
> Cannot set permission for

Re: Review Request 48507: AMBARI-17074: Expose Spark daemon memory in Spark2

2016-06-21 Thread Weiqing Yang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48507/
---

(Updated June 21, 2016, 6:18 p.m.)


Review request for Ambari, Zhe (Joe) Wang, Sumit Mohanty, and Srimanth Gunturi.


Bugs: AMBARI-17074
https://issues.apache.org/jira/browse/AMBARI-17074


Repository: ambari


Description
---

Expose Spark daemon memory in Spark2, so that the user can modify its size on 
ambari web UI easily. The change is the same with the patch of jira 
AMBARI-16757, which is for Spark 1.6.1.


Diffs (updated)
-

  
ambari-server/src/main/resources/common-services/SPARK2/2.0.0/configuration/spark2-env.xml
 76c670a 
  
ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py
 6925ab5 

Diff: https://reviews.apache.org/r/48507/diff/


Testing
---

Tests manually passed.


Thanks,

Weiqing Yang

Review Request 49036: Allow https prtotocol for Log Search


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49036/
---

Review request for Ambari, Oliver Szabo, Robert Nettleton, and Sumit Mohanty.


Bugs: AMBARI-17348
https://issues.apache.org/jira/browse/AMBARI-17348


Repository: ambari


Description
---

Add a property to choose whether or not the Log Search UI should run using 
https.


Diffs
-

  ambari-logsearch/ambari-logsearch-logfeeder/src/main/scripts/run.sh 376dfee 
  
ambari-logsearch/ambari-logsearch-portal/src/main/java/org/apache/ambari/logsearch/LogSearch.java
 993c532 
  ambari-logsearch/ambari-logsearch-portal/src/main/scripts/run.sh 6c687b9 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logfeeder-env.xml
 26a303c 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logsearch-env.xml
 e0d42d3 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logsearch-solr-env.xml
 155ff04 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/params.py
 d21a97f 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/service_check.py
 2993190 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logsearch.properties.j2
 3f5db30 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/logfeeder-env.sh.j2
 aba06c6 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/logsearch-env.sh.j2
 7b0aed3 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/logsearch-solr-env.sh.j2
 2b17e63 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/quicklinks/quicklinks.json
 fc71dfc 
  ambari-server/src/test/python/stacks/2.4/configs/default.json 848be40 
  ambari-web/app/data/HDP2/site_properties.js 0262211 

Diff: https://reviews.apache.org/r/49036/diff/


Testing
---

Tested on local cluster, works fine with every combination of with/without SSL 
Solr connect, and http/https protocol for Log Search UI.

ambari-server:
OK
--
Total run:1072
Total errors:0
Total failures:0


Thanks,

Miklos Gergely

Re: Review Request 48549: AMBARI-17165 Handle Java patches execution during Ranger upgrade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48549/#review138890
---


Ship it!




Ship It!

- Alejandro Fernandez


On June 21, 2016, 12:11 p.m., Mugdha Varadkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48549/
> ---
> 
> (Updated June 21, 2016, 12:11 p.m.)
> 
> 
> Review request for Ambari, Gautam Borad, Jonathan Hurley, Nate Cole, Srimanth 
> Gunturi, and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-17165
> https://issues.apache.org/jira/browse/AMBARI-17165
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Revisiting execution of java patches during upgrade for Ranger Service
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py
>  9e0fb7c 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py
>  cedb3f9 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py
>  4db9f10 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.3.xml
>  4fd5801 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.4.xml
>  272a3cc 
>   ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml 
> b0cff68 
>   ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.4.xml 
> 0b72254 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml
>  111b432 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  9365646 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  42c4979 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.3.xml 
> 712241b 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> 4187d64 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> 3461ad4 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml
>  e83b54b 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml
>  b1c2468d 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.4.xml 
> 4065e87 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml 
> 426b452 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml
>  460e6b3 
>   ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.5.xml 
> 6ce4c81 
> 
> Diff: https://reviews.apache.org/r/48549/diff/
> 
> 
> Testing
> ---
> 
> Tested Ranger upgrade from stack 2.4 to 2.5
> 
> 
> Thanks,
> 
> Mugdha Varadkar
> 
>

Re: Review Request 48998: AMBARI-17326 - Mark the role "(from group)" if there is no specific user level role set.

2016-06-21 Thread Zhe (Joe) Wang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48998/#review138885
---


Ship it!




Ship It!

- Zhe (Joe) Wang


On June 21, 2016, 2:38 a.m., Richard Zang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48998/
> ---
> 
> (Updated June 21, 2016, 2:38 a.m.)
> 
> 
> Review request for Ambari and Zhe (Joe) Wang.
> 
> 
> Bugs: AMBARI-17326
> https://issues.apache.org/jira/browse/AMBARI-17326
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Mark the role "(from group)" if there is no specific user level role set.
> 
> 
> Diffs
> -
> 
>   ambari-admin/src/main/resources/ui/admin-web/app/scripts/i18n.config.js 
> b4138b3 
>   
> ambari-admin/src/main/resources/ui/admin-web/app/views/clusters/userAccessList.html
>  c0f6144 
> 
> Diff: https://reviews.apache.org/r/48998/diff/
> 
> 
> Testing
> ---
> 
> Manually tested on live cluster.
> All unit tests passed.
> 
> 
> Thanks,
> 
> Richard Zang
> 
>

Re: Review Request 49033: Clear /security.json config on solr znode when kerberos is disabled.



> On June 21, 2016, 4:56 p.m., Sebastian Toader wrote:
> > ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logsearch_solr.py,
> >  lines 96-100
> > 
> >
> > Instead of having two Execute commands have just only one and set the 
> > value for security_content properly.

unfortunately it expects python variable inside {}, so in the end it looks more 
readable (if im not put into format)..and if i add escaping the python string 
will recognize that.


- Oliver


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49033/#review138878
---


On June 21, 2016, 4:52 p.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49033/
> ---
> 
> (Updated June 21, 2016, 4:52 p.m.)
> 
> 
> Review request for Ambari, Don Bosco Durai, Miklos Gergely, Robert Nettleton, 
> Sumit Mohanty, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-17344
> https://issues.apache.org/jira/browse/AMBARI-17344
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> /security.json needs to be cleared if kerberos is disabled. can be a problem 
> if a cluster is kerberized, then kerberos is disabled after that
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logsearch_solr.py
>  33ba70c 
>   ambari-server/src/test/python/stacks/2.4/LOGSEARCH/test_solr.py 0474aed 
> 
> Diff: https://reviews.apache.org/r/49033/diff/
> 
> 
> Testing
> ---
> 
> Total run:1073
> Total errors:0
> Total failures:0
> 
> 
> Thanks,
> 
> Oliver Szabo
> 
>

Review Request 49033: Clear /security.json config on solr znode when kerberos is disabled.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49033/
---

Review request for Ambari, Don Bosco Durai, Miklos Gergely, Robert Nettleton, 
Sumit Mohanty, and Sebastian Toader.


Bugs: AMBARI-17344
https://issues.apache.org/jira/browse/AMBARI-17344


Repository: ambari


Description
---

/security.json needs to be cleared if kerberos is disabled. can be a problem if 
a cluster is kerberized, then kerberos is disabled after that


Diffs
-

  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logsearch_solr.py
 33ba70c 
  ambari-server/src/test/python/stacks/2.4/LOGSEARCH/test_solr.py 0474aed 

Diff: https://reviews.apache.org/r/49033/diff/


Testing
---

Total run:1073
Total errors:0
Total failures:0


Thanks,

Oliver Szabo

Re: Review Request 49032: Hive metastore failed to start


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49032/#review138876
---


Ship it!




Ship It!

- Alejandro Fernandez


On June 21, 2016, 4:28 p.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49032/
> ---
> 
> (Updated June 21, 2016, 4:28 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, Dmytro Sen, 
> and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17345
> https://issues.apache.org/jira/browse/AMBARI-17345
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> STR:
> 1) Add Hive Metastore on hosts that do not have it
> 2) Restart services with stale configs
> It started fine on retrying
> 
> {code}
> ERROR: Unable to connect to the DB. Please check DB connection 
> properties.\njava.lang.ClassNotFoundException: 
> oracle.jdbc.driver.OracleDriver\n2016-06-20 13:18:14,206 - Retrying after 10 
> seconds. Reason: Execution of '/usr/jdk64/jdk1.8.0_77/bin/java -cp 
> /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/hdp/current/hive-metastore/lib/ojdbc6.jar
>  org.apache.ambari.server.DBConnectionVerification 
> 'jdbc:oracle:thin:@//172.22.124.192:1521/XE' hiveuser [PROTECTED] 
> oracle.jdbc.driver.OracleDriver' returned 1. ERROR: Unable to connect to the 
> DB. Please check DB connection properties.\njava.lang.ClassNotFoundException:
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
>  30cc942 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
>  6a53dca 
> 
> Diff: https://reviews.apache.org/r/49032/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Re: Review Request 49030: Ambari agent log contains failures for AMS status commands


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49030/#review138875
---


Ship it!




Ship It!

- Alejandro Fernandez


On June 21, 2016, 4 p.m., Dmytro Sen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49030/
> ---
> 
> (Updated June 21, 2016, 4 p.m.)
> 
> 
> Review request for Ambari, Aravindan Vijayan, Dmitro Lisnichenko, and Vitalyi 
> Brodetskyi.
> 
> 
> Bugs: AMBARI-17342
> https://issues.apache.org/jira/browse/AMBARI-17342
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Ambari agent logs contain failures for status commands for AMS collector is 
> status command is executed before AMS collector is actually installed. 
> Reproduced on blueprint deployments
> INFO 2016-06-15 08:48:17,190 PythonReflectiveExecutor.py:65 - Reflective 
> command failed with exception:
> Traceback (most recent call last):
>   File 
> "/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py", 
> line 57, in run_file
> imp.load_source('__main__', script)
>   File 
> "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py",
>  line 143, in 
> AmsCollector().execute()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 257, in execute
> method(env)
>   File 
> "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py",
>  line 98, in security_status
> is_hbase_distributed = 
> security_params['hbase-site']['hbase.cluster.distributed']
> KeyError: 'hbase.cluster.distributed'
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py
>  825a48e 
>   
> ambari-server/src/test/python/stacks/2.0.6/AMBARI_METRICS/test_metrics_collector.py
>  195d388 
> 
> Diff: https://reviews.apache.org/r/49030/diff/
> 
> 
> Testing
> ---
> 
> --
> Ran 261 tests in 6.512s
> 
> OK
> --
> Total run:1072
> Total errors:0
> Total failures:0
> OK
> 
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] Ambari Main ... SUCCESS [3.713s]
> [INFO] Apache Ambari Project POM . SUCCESS [0.031s]
> [INFO] Ambari Views .. SUCCESS [3.690s]
> [INFO] ambari-metrics  SUCCESS [0.375s]
> [INFO] Ambari Metrics Common . SUCCESS [0.721s]
> [INFO] Ambari Server . SUCCESS [59.800s]
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 1:08.949s
> [INFO] Finished at: Tue Jun 21 18:52:10 EEST 2016
> [INFO] Final Memory: 70M/948M
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Re: Review Request 49030: Ambari agent log contains failures for AMS status commands

2016-06-21 Thread Aravindan Vijayan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49030/#review138873
---


Ship it!




Ship It!

- Aravindan Vijayan


On June 21, 2016, 4 p.m., Dmytro Sen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49030/
> ---
> 
> (Updated June 21, 2016, 4 p.m.)
> 
> 
> Review request for Ambari, Aravindan Vijayan, Dmitro Lisnichenko, and Vitalyi 
> Brodetskyi.
> 
> 
> Bugs: AMBARI-17342
> https://issues.apache.org/jira/browse/AMBARI-17342
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Ambari agent logs contain failures for status commands for AMS collector is 
> status command is executed before AMS collector is actually installed. 
> Reproduced on blueprint deployments
> INFO 2016-06-15 08:48:17,190 PythonReflectiveExecutor.py:65 - Reflective 
> command failed with exception:
> Traceback (most recent call last):
>   File 
> "/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py", 
> line 57, in run_file
> imp.load_source('__main__', script)
>   File 
> "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py",
>  line 143, in 
> AmsCollector().execute()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 257, in execute
> method(env)
>   File 
> "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py",
>  line 98, in security_status
> is_hbase_distributed = 
> security_params['hbase-site']['hbase.cluster.distributed']
> KeyError: 'hbase.cluster.distributed'
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py
>  825a48e 
>   
> ambari-server/src/test/python/stacks/2.0.6/AMBARI_METRICS/test_metrics_collector.py
>  195d388 
> 
> Diff: https://reviews.apache.org/r/49030/diff/
> 
> 
> Testing
> ---
> 
> --
> Ran 261 tests in 6.512s
> 
> OK
> --
> Total run:1072
> Total errors:0
> Total failures:0
> OK
> 
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] Ambari Main ... SUCCESS [3.713s]
> [INFO] Apache Ambari Project POM . SUCCESS [0.031s]
> [INFO] Ambari Views .. SUCCESS [3.690s]
> [INFO] ambari-metrics  SUCCESS [0.375s]
> [INFO] Ambari Metrics Common . SUCCESS [0.721s]
> [INFO] Ambari Server . SUCCESS [59.800s]
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 1:08.949s
> [INFO] Finished at: Tue Jun 21 18:52:10 EEST 2016
> [INFO] Final Memory: 70M/948M
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Re: Review Request 48395: AMBARI-17027: Metrics Collector API: Introduce basic series aggregation functions

2016-06-21 Thread Prajwal Rao


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48395/#review138870
---




ambari-metrics/ambari-metrics-grafana/ambari-metrics/partials/query.editor.html 
(line 112)


Visually, I think it makes more sense to add this after "precision" as it 
seems to break the first row.



ambari-metrics/ambari-metrics-grafana/ambari-metrics/queryCtrl.js (line 39)


You will need to accommodate for seriesAggregator not being set at all 
(undefined), for example when dashboards are imported without the 
seriesAggregator in them, it will throw up an error.

You can rememdy this by doing a function similar to $scope.trasnform.


- Prajwal Rao


On June 21, 2016, 3:06 a.m., Jungtaek Lim wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48395/
> ---
> 
> (Updated June 21, 2016, 3:06 a.m.)
> 
> 
> Review request for Ambari, Aravindan Vijayan, Dmytro Sen, Prajwal Rao, 
> Sriharsha Chintalapani, Sid Wagle, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-17027
> https://issues.apache.org/jira/browse/AMBARI-17027
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMS doesn't provide tag so metric is identified by appId, metric name, 
> hostname, instanceId. In this situation metric name is normally consist of 
> origin metric name and tag values, like graphite, but unlike Graphite, AMS 
> also doesn't provide series aggregation functions so aggregation should be 
> done from caller side.
> 
> It would be great if Ambari Metrics Collector provides series aggregation 
> functions, like sumSeries / 
> averageSeries / minSeries / maxSeries on Graphite.
> 
> Query outputs: 
> https://gist.github.com/HeartSaVioR/f4f28b5b8b7bf2e5477e59d7fd56090f
> 
> Attached Grafana screenshots to AMBARI-17027. Please refer 
> https://issues.apache.org/jira/browse/AMBARI-17027 for details.
> 
> 
> Diffs
> -
> 
>   ambari-metrics/ambari-metrics-grafana/ambari-metrics/datasource.js b825774 
>   
> ambari-metrics/ambari-metrics-grafana/ambari-metrics/partials/query.editor.html
>  b034c03 
>   ambari-metrics/ambari-metrics-grafana/ambari-metrics/queryCtrl.js 2eb3613 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/HBaseTimelineMetricStore.java
>  1b2d02f 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStore.java
>  e37bc4d 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStoreWatcher.java
>  7d49070 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/AbstractTimelineMetricsSeriesAggregateFunction.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/SeriesAggregateFunction.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAggregateFunction.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAggregateFunctionFactory.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesAvgAggregateFunction.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesMaxAggregateFunction.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesMinAggregateFunction.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/function/TimelineMetricsSeriesSumAggregateFunction.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/TimelineWebServices.java
>  ee3a097 
>   
>

Re: Review Request 49032: Hive metastore failed to start


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49032/#review138866
---


Ship it!




Ship It!

- Dmytro Sen


On Июнь 21, 2016, 4:28 п.п., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49032/
> ---
> 
> (Updated Июнь 21, 2016, 4:28 п.п.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, Dmytro Sen, 
> and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17345
> https://issues.apache.org/jira/browse/AMBARI-17345
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> STR:
> 1) Add Hive Metastore on hosts that do not have it
> 2) Restart services with stale configs
> It started fine on retrying
> 
> {code}
> ERROR: Unable to connect to the DB. Please check DB connection 
> properties.\njava.lang.ClassNotFoundException: 
> oracle.jdbc.driver.OracleDriver\n2016-06-20 13:18:14,206 - Retrying after 10 
> seconds. Reason: Execution of '/usr/jdk64/jdk1.8.0_77/bin/java -cp 
> /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/hdp/current/hive-metastore/lib/ojdbc6.jar
>  org.apache.ambari.server.DBConnectionVerification 
> 'jdbc:oracle:thin:@//172.22.124.192:1521/XE' hiveuser [PROTECTED] 
> oracle.jdbc.driver.OracleDriver' returned 1. ERROR: Unable to connect to the 
> DB. Please check DB connection properties.\njava.lang.ClassNotFoundException:
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
>  30cc942 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
>  6a53dca 
> 
> Diff: https://reviews.apache.org/r/49032/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Re: Review Request 48970: Follow up NiFi log changes in the LogFeeder config

2016-06-21 Thread Jayush Luniya


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48970/#review138863
---


Ship it!




Ship It!

- Jayush Luniya


On June 20, 2016, 11:19 p.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48970/
> ---
> 
> (Updated June 20, 2016, 11:19 p.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Oliver Szabo.
> 
> 
> Bugs: AMBARI-17323
> https://issues.apache.org/jira/browse/AMBARI-17323
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The NiFi service log dir properties were recently changes. The LogFeeder 
> config needs to follow up this change to access the NiFi logs
> 
> 
> Diffs
> -
> 
>   
> ambari-logsearch/ambari-logsearch-portal/src/main/resources/HadoopServiceConfig.json
>  71a4c1c 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/params.py
>  5afc04c 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/input.config-nifi.json.j2
>  1bbab20 
> 
> Diff: https://reviews.apache.org/r/48970/diff/
> 
> 
> Testing
> ---
> 
> Tested on GCE cluster.
> 
> 
> Thanks,
> 
> Miklos Gergely
> 
>

Re: Review Request 49029: When requesting a Kerberos Descriptor via the REST API, 'when' clauses should optionally be processed


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49029/
---

(Updated June 21, 2016, 12:14 p.m.)


Review request for Ambari, Di Li, Jonathan Hurley, Nate Cole, Oliver Szabo, and 
Sandor Magyari.


Bugs: AMBARI-17226
https://issues.apache.org/jira/browse/AMBARI-17226


Repository: ambari


Description
---

When requesting a Kerberos Descriptor via the REST API, 'when' clauses should 
optionally be processed.  If elected to be processed, identities that contain 
`when` clauses will be included or excluded from the resulting descriptor based 
on the result of the evaluation. 

In the event of an _add service_ scenario, the services being added should be 
able to be specified so that they can be included in the data used for 
`when`-clause evaluation.  

#Solution
Add _GET directives_ to specify whether `when` clauses are to be evaluated (or 
not) while building the Kerberos Descriptor using the following API call:
```
GET 
/api/v1/clusters/CLUSTER_NAME/kerberos_descriptors/COMPOSITE?evaluate_when=true
```
If new services are being added, the `additional_services` directive should be 
added to the request so the evaluation can be preformed on the _future_ set of 
services, which may evaluate differently then the _current_ set of services.
```
GET 
/api/v1/clusters/CLUSTER_NAME/kerberos_descriptors/COMPOSITE?evaluate_when=true@additional_services=HIVE,TEZ,PIG
```

#Notes
- A lot of code was moved from ClusterKerberosDescriptorResourceProvider to 
KerberosHelper to reduce the likelyhood of duplication
- The logic to build a Kerberos descriptor (STACK, USER, COMPOSITE) has been 
consolidated in the KerberosHelper
- Some duplicate code in test cases have been conslidated


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/api/resources/ResourceInstanceFactoryImpl.java
 a2aeffa 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelper.java
 5ffc8a3 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
 cd79e46 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterKerberosDescriptorResourceProvider.java
 1f5d1d8 
  
ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/AbstractKerberosDescriptor.java
 79b9a55 
  
ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/AbstractKerberosDescriptorContainer.java
 64d9292 
  
ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/KerberosComponentDescriptor.java
 19d3f5e 
  
ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/KerberosDescriptor.java
 98f8883 
  
ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/KerberosDescriptorType.java
 0677de6 
  
ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/KerberosIdentityDescriptor.java
 2631d35 
  
ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/KerberosServiceDescriptor.java
 9eeb802 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
 7281e85 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClusterKerberosDescriptorResourceProviderTest.java
 898cf46 
  
ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/KerberosComponentDescriptorTest.java
 9f0f7a1 
  
ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/KerberosDescriptorTest.java
 004cd66 
  
ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/KerberosServiceDescriptorTest.java
 e5392c0 

Diff: https://reviews.apache.org/r/49029/diff/


Testing (updated)
---

Manually testing...  updated unit tests...

# Local test results: 

[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 1:27:14.265s
[INFO] Finished at: Tue Jun 21 12:03:08 EDT 2016
[INFO] Final Memory: 61M/1851M
[INFO] 

# Jenkins test results: PENDING


Thanks,

Robert Levas

Re: Review Request 47656: AMBARI-12885 - Dynamic stack extensions - install and upgrade support for custom services

2016-06-21 Thread Jayush Luniya


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47656/#review138862
---


Ship it!




Ship It!

- Jayush Luniya


On June 20, 2016, 4:18 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47656/
> ---
> 
> (Updated June 20, 2016, 4:18 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, Alejandro Fernandez, bhuvnesh 
> chaudhary, Jayush Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth 
> Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-12885
> https://issues.apache.org/jira/browse/AMBARI-12885
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The purpose of this proposal is to facilitate adding custom services to an 
> existing stack. Ideally this would support adding and upgrading custom 
> services separately from the core services defined in the stack. In 
> particular we are looking at custom services that need to support several 
> different stacks (different distributions of Ambari). The release cycle of 
> the custom services may be different from that of the core stack; that is, a 
> custom service may be upgraded at a different rate than the core distribution 
> itself and may be upgraded multiple times within the lifespan of a single 
> release of the core distribution.
> 
> One possible approach to handling this would be dynamically extending a stack 
> (after install time). It would be best to extend the stack in packages where 
> a stack extension package can have one or more custom services.
> 
> 
> Diffs
> -
> 
>   ambari-server/conf/unix/ambari.properties a88a025 
>   ambari-server/src/main/assemblies/server.xml 1560d8d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/resources/ExtensionLinkResourceDefinition.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/resources/ExtensionResourceDefinition.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/resources/ExtensionVersionResourceDefinition.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/resources/ResourceInstanceFactoryImpl.java
>  cf7c391 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  f0928cf 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/ExtensionLinksService.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/ExtensionsService.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/StacksService.java
>  557ce98 
>   
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
>  2b7fca0 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java
>  b488af3 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  b5c6fc2 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionLinkRequest.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionLinkResponse.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionRequest.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionResponse.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionVersionRequest.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/ExtensionVersionResponse.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AbstractControllerResourceProvider.java
>  3721113 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/Extension.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ExtensionLinkResourceProvider.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ExtensionResourceProvider.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ExtensionVersionResourceProvider.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/spi/Resource.java
>  99e4ccd 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ExtensionDAO.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ExtensionLinkDAO.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ExtensionEntity.java
>  PRE-CREATION 
>   
>

Re: Review Request 48413: Fix misnamed Zookeeper connect strings in Log Search


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48413/
---

(Updated June 21, 2016, 4:03 p.m.)


Review request for Ambari, Daniel Gergely, Oliver Szabo, Robert Nettleton, 
Sumit Mohanty, and Sebastian Toader.


Changes
---

Update as trunk has changed


Bugs: AMBARI-17117
https://issues.apache.org/jira/browse/AMBARI-17117


Repository: ambari


Description
---

Variables/properties holding zookeeper connect strings are misnamed as zk_host, 
or zk_hosts, which may be misleading. Variable / property names fixed.


Diffs (updated)
-

  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/logconfig/FetchConfigFromSolr.java
 4240b86 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputSolr.java
 c945ed7 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/SolrUtil.java
 31fbded 
  ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/config.json.j2 
b6301ca 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/logfeeder.properties
 b4655cc 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/output.config.json.j2
 a485600 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/OutputSolrTest.java
 3014ed8 
  
ambari-logsearch/ambari-logsearch-portal/src/main/java/org/apache/ambari/logsearch/dao/AuditSolrDao.java
 03ff0ff 
  
ambari-logsearch/ambari-logsearch-portal/src/main/java/org/apache/ambari/logsearch/dao/ServiceLogsSolrDao.java
 14125bc 
  
ambari-logsearch/ambari-logsearch-portal/src/main/java/org/apache/ambari/logsearch/dao/SolrDaoBase.java
 4564752 
  
ambari-logsearch/ambari-logsearch-portal/src/main/java/org/apache/ambari/logsearch/dao/UserConfigSolrDao.java
 edf1dcc 
  
ambari-logsearch/ambari-logsearch-portal/src/main/java/org/apache/ambari/logsearch/solr/metrics/SolrMetricsLoader.java
 21c010f 
  
ambari-logsearch/ambari-logsearch-portal/src/main/resources/logsearch.properties.j2
 0a94186 
  
ambari-logsearch/ambari-logsearch-solr-client/src/main/java/org/apache/ambari/logsearch/solr/AmbariSolrCloudCLI.java
 ad5d789 
  
ambari-logsearch/ambari-logsearch-solr-client/src/main/java/org/apache/ambari/logsearch/solr/AmbariSolrCloudClient.java
 32a1821 
  
ambari-logsearch/ambari-logsearch-solr-client/src/main/java/org/apache/ambari/logsearch/solr/AmbariSolrCloudClientBuilder.java
 50204c4 
  
ambari-logsearch/ambari-logsearch-solr-client/src/main/java/org/apache/ambari/logsearch/solr/commands/GetSolrHostsCommand.java
 f814678 
  
ambari-logsearch/ambari-logsearch-solr-client/src/test/java/org/apache/ambari/logsearch/solr/AmbariSolrCloudClientTest.java
 c382c14 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logfeeder.properties.j2
 31c252a 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logsearch.properties.j2
 0bbba0f 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/output.config.json.j2
 b31f39b 
  ambari-server/src/test/python/stacks/2.4/configs/default.json ff548e0 

Diff: https://reviews.apache.org/r/48413/diff/


Testing
---

Tested on local cluster

ambari-logsearch-logfeeder:
Tests run: 35, Failures: 0, Errors: 0, Skipped: 0


Thanks,

Miklos Gergely

Review Request 49030: Ambari agent log contains failures for AMS status commands


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49030/
---

Review request for Ambari, Aravindan Vijayan, Dmitro Lisnichenko, and Vitalyi 
Brodetskyi.


Bugs: AMBARI-17342
https://issues.apache.org/jira/browse/AMBARI-17342


Repository: ambari


Description
---

Ambari agent logs contain failures for status commands for AMS collector is 
status command is executed before AMS collector is actually installed. 
Reproduced on blueprint deployments
INFO 2016-06-15 08:48:17,190 PythonReflectiveExecutor.py:65 - Reflective 
command failed with exception:
Traceback (most recent call last):
  File 
"/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py", 
line 57, in run_file
imp.load_source('__main__', script)
  File 
"/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py",
 line 143, in 
AmsCollector().execute()
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
 line 257, in execute
method(env)
  File 
"/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py",
 line 98, in security_status
is_hbase_distributed = 
security_params['hbase-site']['hbase.cluster.distributed']
KeyError: 'hbase.cluster.distributed'


Diffs
-

  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py
 825a48e 
  
ambari-server/src/test/python/stacks/2.0.6/AMBARI_METRICS/test_metrics_collector.py
 195d388 

Diff: https://reviews.apache.org/r/49030/diff/


Testing
---

--
Ran 261 tests in 6.512s

OK
--
Total run:1072
Total errors:0
Total failures:0
OK

[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Ambari Main ... SUCCESS [3.713s]
[INFO] Apache Ambari Project POM . SUCCESS [0.031s]
[INFO] Ambari Views .. SUCCESS [3.690s]
[INFO] ambari-metrics  SUCCESS [0.375s]
[INFO] Ambari Metrics Common . SUCCESS [0.721s]
[INFO] Ambari Server . SUCCESS [59.800s]
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 1:08.949s
[INFO] Finished at: Tue Jun 21 18:52:10 EEST 2016
[INFO] Final Memory: 70M/948M


Thanks,

Dmytro Sen

Review Request 49029: When requesting a Kerberos Descriptor via the REST API, 'when' clauses should optionally be processed


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49029/
---

Review request for Ambari, Di Li, Jonathan Hurley, Nate Cole, Oliver Szabo, and 
Sandor Magyari.


Bugs: AMBARI-17226
https://issues.apache.org/jira/browse/AMBARI-17226


Repository: ambari


Description
---

When requesting a Kerberos Descriptor via the REST API, 'when' clauses should 
optionally be processed.  If elected to be processed, identities that contain 
`when` clauses will be included or excluded from the resulting descriptor based 
on the result of the evaluation. 

In the event of an _add service_ scenario, the services being added should be 
able to be specified so that they can be included in the data used for 
`when`-clause evaluation.  

#Solution
Add _GET directives_ to specify whether `when` clauses are to be evaluated (or 
not) while building the Kerberos Descriptor using the following API call:
```
GET 
/api/v1/clusters/CLUSTER_NAME/kerberos_descriptors/COMPOSITE?evaluate_when=true
```
If new services are being added, the `additional_services` directive should be 
added to the request so the evaluation can be preformed on the _future_ set of 
services, which may evaluate differently then the _current_ set of services.
```
GET 
/api/v1/clusters/CLUSTER_NAME/kerberos_descriptors/COMPOSITE?evaluate_when=true@additional_services=HIVE,TEZ,PIG
```

#Notes
- A lot of code was moved from ClusterKerberosDescriptorResourceProvider to 
KerberosHelper to reduce the likelyhood of duplication
- The logic to build a Kerberos descriptor (STACK, USER, COMPOSITE) has been 
consolidated in the KerberosHelper
- Some duplicate code in test cases have been conslidated


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/api/resources/ResourceInstanceFactoryImpl.java
 a2aeffa 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelper.java
 5ffc8a3 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
 cd79e46 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterKerberosDescriptorResourceProvider.java
 1f5d1d8 
  
ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/AbstractKerberosDescriptor.java
 79b9a55 
  
ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/AbstractKerberosDescriptorContainer.java
 64d9292 
  
ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/KerberosComponentDescriptor.java
 19d3f5e 
  
ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/KerberosDescriptor.java
 98f8883 
  
ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/KerberosDescriptorType.java
 0677de6 
  
ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/KerberosIdentityDescriptor.java
 2631d35 
  
ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/KerberosServiceDescriptor.java
 9eeb802 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
 7281e85 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClusterKerberosDescriptorResourceProviderTest.java
 898cf46 
  
ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/KerberosComponentDescriptorTest.java
 9f0f7a1 
  
ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/KerberosDescriptorTest.java
 004cd66 
  
ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/KerberosServiceDescriptorTest.java
 e5392c0 

Diff: https://reviews.apache.org/r/49029/diff/


Testing
---

Manually testing...  updated unit tests...

# Local test results: PENDING

# Jenkins test results: PENDING


Thanks,

Robert Levas

Re: Review Request 48734: App timeline Server start fails on enabling HA because namenode is in safemode

2016-06-21 Thread Victor Galgo



> On June 17, 2016, 6:44 p.m., Di Li wrote:
> > Hello Victor,
> > 
> > so I ran some tests and observed the following. I have a 3 node cluster, 
> > c1.apache.org, c2.apache.org, and c3.apache.org
> > 
> > 1. Right after finishing the manual steps listed on step "Initialize 
> > Metadata". I noticed c1.apache.org has NameNode process running but it's 
> > the standby. c2.apache.org (the new NN added) has NN stopped.
> > 
> > 2. The state of the two NNs in #1 seems to have cause the NN's 
> > check_is_active_namenode function call to return False, thus setting 
> > ensure_safemode_off to False as well. >> Skipping the safemode check 
> > altegather.
> > 
> > 3. If I just ran safemode check command line hadoop cmd, here are the 
> > results, notice the safemode is reported as ON on the Standby node and the 
> > ther one is a connection refused err
> > 
> > [hdfs@c1 ~]$ hdfs --config /usr/iop/current/hadoop-client/conf haadmin -ns 
> > binn -getServiceState nn1
> > standby
> > [hdfs@c1 ~]$ hdfs --config /usr/iop/current/hadoop-client/conf haadmin -ns 
> > binn -getServiceState nn2
> > 16/06/17 11:26:42 INFO ipc.Client: Retrying connect to server: 
> > c1.apache.org:8020. Already tried 0 time(s); retry policy is 
> > RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1000 
> > MILLISECONDS)
> > Operation failed: Call From c1.apache.org to c2.apache.org:8020 failed on 
> > connection exception: java.net.ConnectException: Connection refused; For 
> > more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
> > [hdfs@c1 ~]$ hdfs dfsadmin -safemode get
> > Safe mode is ON in c1.apache.org:8020
> > safemode: Call From c1.apache.org to c2.apache.org:8020 failed on 
> > connection exception: java.net.ConnectException: Connection refused; For 
> > more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
> > 
> > So in my opinion, the fix should be at the NameNode Python script level to 
> > always check safemode against the two NNs, and make sure the safemode is 
> > off on the active namenode. As a safeguard against offline active NN, the 
> > check should eventually timeout to unblock the rest of the start sequence.
> 
> Victor Galgo wrote:
> "So in my opinion, the fix should be at the NameNode Python script level 
> to always check safemode against the two NNs". 
> We cannot do that. Because at that points all Datanodes are stopped. 
> Which means NN will never go out of safemode.
> 
> Alejandro Fernandez wrote:
> Please include Jonathan Hurley in the code review since he recently 
> modified the function that waits to leave safemode.
> This is not the first time that we've had the need for a step to "leave 
> safe mode". So either we put it into the python code (and do a lot of testing 
> on it since it also impacts EU and RU), or make a custom command for HDFS 
> that is only available if HA is present, and it waits for NameNode to leave 
> safemode.
> 
> Jonathan Hurley wrote:
> Yes, I recently added something for the case during an EU where we know 
> that the NameNode probably won't leave Safemode. Essentially, don't try to 
> create any directories if the NN didn't wait for safemode to exit. That was 
> only for NN, though.
> 
> But this problem is a more generic case - it affects other services. 
> Since NN wasn't restarted it might be in Safemode. In this case, I think we 
> need to handle the retryable exception and back off and wait. 
> 
> However, you could also argue that since we know we're doing a restart 
> operation, we should be shutting down the NNs completely. If there's no issue 
> with shutting them stop during the HA process, then this patch seems fine for 
> now, but we should open another one for catching the RetryableException.

Thanks Jonathan! Absolutely agree with your points. Could you please Ship it?


- Victor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48734/#review138280
---


On June 17, 2016, 10:45 p.m., Victor Galgo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48734/
> ---
> 
> (Updated June 17, 2016, 10:45 p.m.)
> 
> 
> Review request for Ambari, Andriy Babiichuk, Alexandr Antonenko, Andrew 
> Onischuk, Di Li, Dmitro Lisnichenko, Jonathan Hurley, Jayush Luniya, Robert 
> Levas, Sandor Magyari, Sumit Mohanty, Sebastian Toader, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-17182
> https://issues.apache.org/jira/browse/AMBARI-17182
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> On the last step "Start all" on enabling HA below happens:
> 
> Traceback (most recent call last):
> File 
>

Re: Review Request 48722: Reduce the idle time before first command from next stage is executed on a host


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48722/#review138851
---



We have some logs which are triggered with every heartbeat. This would flood 
logs badly if done every second. Could we fix this. (possibly in a separate 
jira)

- Andrew Onischuk


On June 21, 2016, 3:19 p.m., Sebastian Toader wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48722/
> ---
> 
> (Updated June 21, 2016, 3:19 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Laszlo Puskas, Robert Levas, 
> Sandor Magyari, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17248
> https://issues.apache.org/jira/browse/AMBARI-17248
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Commands to be executed by ambari-agents are being sent down by the server in 
> the response message to agent heartbeat messages. 
> The server processes the received heartbeat, it checks if there are next 
> commands scheduled to be executed by ambari-agent and adds those to the 
> heartbeat response for the ambari-agent.
> The server organises the commands that can be executed in parallel into 
> stages. Ambari server ensures that only the commands of a single stage is 
> scheduled to be executed by the agent and starts scheduling the commands of 
> the next stage only after all commands of current stage has finished 
> successfully.
> The processing of command status received with the heartbeat message happens 
> asynchronously to heartbeat response in HeartBeatProcessor and 
> ActionScheduler creation thus when the heartbeat response is created the 
> commands for the next stage are not scheduled yet. This means that the next 
> commands will be sent to agent only with the next heartbeat.
> Agents currently sends a heartbeat to the server on command a completion or 
> at a timeout = self.netutil.HEARTBEAT_IDDLE_INTERVAL_SEC – 
> self.netutil.MINIMUM_INTERVAL_BETWEEN_HEARTBEATS interval which is ~10 
> seconds if there are no commands to be executed.
> This means that when the server receives a heartbeat triggered by the 
> completion of the last command from the current stage the server will send 
> the commands for the next stage only 10 seconds later when the next heartbeat 
> is received. This leads to agents spending considerable amount of time idle 
> when there are multiple stages to be executed.
> Agents should heartbeat at a higher rate while there are still pending stages 
> to be executed.
> 
> 
> Diffs
> -
> 
>   ambari-agent/conf/unix/ambari-agent.ini 8f2ab1b 
>   ambari-agent/conf/unix/upgrade_agent_configs.py 583b5aa 
>   ambari-agent/conf/windows/ambari-agent.ini df88be6 
>   ambari-agent/src/main/python/ambari_agent/AmbariConfig.py 89a881a 
>   ambari-agent/src/main/python/ambari_agent/Controller.py e981a76 
>   ambari-agent/src/main/python/ambari_agent/Heartbeat.py 91098e0 
>   ambari-agent/src/main/python/ambari_agent/NetUtil.py 80bf3ae 
>   ambari-agent/src/test/python/ambari_agent/TestHeartbeat.py f113083 
>   ambari-agent/src/test/python/ambari_agent/TestNetUtil.py d72e319 
>   ambari-agent/src/test/python/ambari_agent/examples/ControllerTester.py 
> 8103872 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
>  35a37e3 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatResponse.java
>  1ab7ae9 
>   ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java 
> ac0ddd2 
>   ambari-server/src/main/java/org/apache/ambari/server/state/Clusters.java 
> bd9de13 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
>  3d2388e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClustersImpl.java
>  c26e1e9 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/cluster/ClusterImplTest.java
>  627ade9 
> 
> Diff: https://reviews.apache.org/r/48722/diff/
> 
> 
> Testing
> ---
> 
> Manual testing.
> 
> Unit tests in succeeded.
> 
> 
> Thanks,
> 
> Sebastian Toader
> 
>

Re: Review Request 49026: Ambari alert "NameNode Last Checkpoint" failing when NameNode is HA


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49026/#review138850
---


Ship it!




Ship It!

- Vitalyi Brodetskyi


On Червень 21, 2016, 3:19 після полудня, Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49026/
> ---
> 
> (Updated Червень 21, 2016, 3:19 після полудня)
> 
> 
> Review request for Ambari and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-17339
> https://issues.apache.org/jira/browse/AMBARI-17339
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The "NameNode Last Checkpoint" alert is implemented in
> _alert_checkpoint_time.py_. This script uses the value of **dfs.namenode.http-
> address** property for querying the last checkpoint timestamp.
> 
> This works fine when the NN is not in HA mode. However when NN is in HA mode
> this property is either not defined or might not be pointing to the right NN
> instance.
> 
> The script needs to be improved such as to figure out if the NN is in HA mode.
> If it's in HA mode do not use **dfs.namenode.http-address** property but
> rather determine the correct URL to connect to similarly as how it's done in
> the _alert_ha_namenode_health.py_
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/namenode_ha_utils.py
>  ca36e62 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/alerts/alert_checkpoint_time.py
>  cb0d49e 
>   
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_alert_checkpoint_time.py 
> 26b3b9c 
> 
> Diff: https://reviews.apache.org/r/49026/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>

Re: Review Request 48722: Reduce the idle time before first command from next stage is executed on a host



> On June 20, 2016, 4:46 p.m., Andrew Onischuk wrote:
> > Also we could miss some other thing like this (state/status), which bring 
> > slowdown. Would be nice to deploy fullstack with and without your patch. 
> > And see if it brings speedup, and not actually slowdown.

The component status reports is not an issue as the commands are issued by the 
server every minute so it's not dependent on the heartbeat rate.


- Sebastian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48722/#review138605
---


On June 21, 2016, 5:19 p.m., Sebastian Toader wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48722/
> ---
> 
> (Updated June 21, 2016, 5:19 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Laszlo Puskas, Robert Levas, 
> Sandor Magyari, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17248
> https://issues.apache.org/jira/browse/AMBARI-17248
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Commands to be executed by ambari-agents are being sent down by the server in 
> the response message to agent heartbeat messages. 
> The server processes the received heartbeat, it checks if there are next 
> commands scheduled to be executed by ambari-agent and adds those to the 
> heartbeat response for the ambari-agent.
> The server organises the commands that can be executed in parallel into 
> stages. Ambari server ensures that only the commands of a single stage is 
> scheduled to be executed by the agent and starts scheduling the commands of 
> the next stage only after all commands of current stage has finished 
> successfully.
> The processing of command status received with the heartbeat message happens 
> asynchronously to heartbeat response in HeartBeatProcessor and 
> ActionScheduler creation thus when the heartbeat response is created the 
> commands for the next stage are not scheduled yet. This means that the next 
> commands will be sent to agent only with the next heartbeat.
> Agents currently sends a heartbeat to the server on command a completion or 
> at a timeout = self.netutil.HEARTBEAT_IDDLE_INTERVAL_SEC – 
> self.netutil.MINIMUM_INTERVAL_BETWEEN_HEARTBEATS interval which is ~10 
> seconds if there are no commands to be executed.
> This means that when the server receives a heartbeat triggered by the 
> completion of the last command from the current stage the server will send 
> the commands for the next stage only 10 seconds later when the next heartbeat 
> is received. This leads to agents spending considerable amount of time idle 
> when there are multiple stages to be executed.
> Agents should heartbeat at a higher rate while there are still pending stages 
> to be executed.
> 
> 
> Diffs
> -
> 
>   ambari-agent/conf/unix/ambari-agent.ini 8f2ab1b 
>   ambari-agent/conf/unix/upgrade_agent_configs.py 583b5aa 
>   ambari-agent/conf/windows/ambari-agent.ini df88be6 
>   ambari-agent/src/main/python/ambari_agent/AmbariConfig.py 89a881a 
>   ambari-agent/src/main/python/ambari_agent/Controller.py e981a76 
>   ambari-agent/src/main/python/ambari_agent/Heartbeat.py 91098e0 
>   ambari-agent/src/main/python/ambari_agent/NetUtil.py 80bf3ae 
>   ambari-agent/src/test/python/ambari_agent/TestHeartbeat.py f113083 
>   ambari-agent/src/test/python/ambari_agent/TestNetUtil.py d72e319 
>   ambari-agent/src/test/python/ambari_agent/examples/ControllerTester.py 
> 8103872 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
>  35a37e3 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatResponse.java
>  1ab7ae9 
>   ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java 
> ac0ddd2 
>   ambari-server/src/main/java/org/apache/ambari/server/state/Clusters.java 
> bd9de13 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
>  3d2388e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClustersImpl.java
>  c26e1e9 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/cluster/ClusterImplTest.java
>  627ade9 
> 
> Diff: https://reviews.apache.org/r/48722/diff/
> 
> 
> Testing
> ---
> 
> Manual testing.
> 
> Unit tests in succeeded.
> 
> 
> Thanks,
> 
> Sebastian Toader
> 
>

Re: Review Request 48805: AMBARI-17280. RU to write out client configs that are dependencies of Hive, ATS, and Oozie during upgrades that change configs


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48805/#review138847
---



I understand what you're doing here, but it's a lot of work for something that 
we are going to throw away anyway after 2.4 - is there no way to simply define 
multiple groupings so that we can still isolate the config changes by service?


ambari-common/src/main/python/resource_management/libraries/script/script.py 
(line 158)


This function returns a value - it doesn't _do_ anything ... can we rename 
it to reflect that?



ambari-common/src/main/python/resource_management/libraries/script/script.py 
(lines 168 - 171)


Way too easy to break this by simply renames in params.py ... can we 
extract these and make them constants somewhere?



ambari-common/src/main/python/resource_management/libraries/script/script.py 
(line 175)


no need for `if params.version` ... the `check_stack_feature` already does 
that.



ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java
 (lines 104 - 111)


What about the other types of groupings, like Colocated?



ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml (line 
153)


The SERVICE/COMPONENT string is too easy to mess up; we should have 
distinct service and component attributes


- Jonathan Hurley


On June 20, 2016, 9:12 p.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48805/
> ---
> 
> (Updated June 20, 2016, 9:12 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, 
> Jonathan Hurley, Nate Cole, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-17280
> https://issues.apache.org/jira/browse/AMBARI-17280
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> During RU, HiveServer2 is restarted but the newer tez configs have not yet 
> been saved, which is incorrect because Hive has a dependency on Tez.
> This is important when configs change during a major stack upgrade, e.g., HDP 
> 2.4 -> 2.5. What happens today is,
> 
> * Install packages generates /etc/tez/2.5.0.0-1/0 and copies the configs from 
> /etc/tez/2.4.0.0-1/0/ to the new folder
> * If configs change during RU, then Hive is restarted and the classpath means 
> that it will pick up the older tez configs from the new /etc/tez/2.5.0.0-1/0 
> folder
> 
> 
> This problem exists for all of these components:
> 
> * HiveServer: depends on Tez and MapReduce clients
> * ATS: depends on Tez and Spark clients
> * Oozie: depends on Tez, Spark, and MapReduce clients
> 
> This problem only exists when configs change (so crossing major stack 
> version) and during RU (because it is allowed to change configs during the 
> middle of restarting services).
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
>  4eb0015 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
>  b994fce 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> 49dcb4e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
>  fb3ae69 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java
>  80bb26c 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  63c72f7 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/spark_client.py
>  ef41453 
>   
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py
>  44239c7 
>   
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez.py
>  67466e3 
>   
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py
>  c79d63b 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/mapreduce2_client.py
>  db22004 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
>  90f885a 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py
>  d1ec15b 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> 4187d64 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> 3461ad4 
>

Re: Review Request 48722: Reduce the idle time before first command from next stage is executed on a host


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48722/
---

(Updated June 21, 2016, 5:19 p.m.)


Review request for Ambari, Andrew Onischuk, Laszlo Puskas, Robert Levas, Sandor 
Magyari, and Sumit Mohanty.


Changes
---

Made the hosts status reports to be generated independently from heartbeat 
rates periodically every 60 seconds (which is configurable).


Bugs: AMBARI-17248
https://issues.apache.org/jira/browse/AMBARI-17248


Repository: ambari


Description
---

Commands to be executed by ambari-agents are being sent down by the server in 
the response message to agent heartbeat messages. 
The server processes the received heartbeat, it checks if there are next 
commands scheduled to be executed by ambari-agent and adds those to the 
heartbeat response for the ambari-agent.
The server organises the commands that can be executed in parallel into stages. 
Ambari server ensures that only the commands of a single stage is scheduled to 
be executed by the agent and starts scheduling the commands of the next stage 
only after all commands of current stage has finished successfully.
The processing of command status received with the heartbeat message happens 
asynchronously to heartbeat response in HeartBeatProcessor and ActionScheduler 
creation thus when the heartbeat response is created the commands for the next 
stage are not scheduled yet. This means that the next commands will be sent to 
agent only with the next heartbeat.
Agents currently sends a heartbeat to the server on command a completion or at 
a timeout = self.netutil.HEARTBEAT_IDDLE_INTERVAL_SEC – 
self.netutil.MINIMUM_INTERVAL_BETWEEN_HEARTBEATS interval which is ~10 seconds 
if there are no commands to be executed.
This means that when the server receives a heartbeat triggered by the 
completion of the last command from the current stage the server will send the 
commands for the next stage only 10 seconds later when the next heartbeat is 
received. This leads to agents spending considerable amount of time idle when 
there are multiple stages to be executed.
Agents should heartbeat at a higher rate while there are still pending stages 
to be executed.


Diffs (updated)
-

  ambari-agent/conf/unix/ambari-agent.ini 8f2ab1b 
  ambari-agent/conf/unix/upgrade_agent_configs.py 583b5aa 
  ambari-agent/conf/windows/ambari-agent.ini df88be6 
  ambari-agent/src/main/python/ambari_agent/AmbariConfig.py 89a881a 
  ambari-agent/src/main/python/ambari_agent/Controller.py e981a76 
  ambari-agent/src/main/python/ambari_agent/Heartbeat.py 91098e0 
  ambari-agent/src/main/python/ambari_agent/NetUtil.py 80bf3ae 
  ambari-agent/src/test/python/ambari_agent/TestHeartbeat.py f113083 
  ambari-agent/src/test/python/ambari_agent/TestNetUtil.py d72e319 
  ambari-agent/src/test/python/ambari_agent/examples/ControllerTester.py 
8103872 
  
ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
 35a37e3 
  
ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatResponse.java
 1ab7ae9 
  ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java 
ac0ddd2 
  ambari-server/src/main/java/org/apache/ambari/server/state/Clusters.java 
bd9de13 
  
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
 3d2388e 
  
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClustersImpl.java
 c26e1e9 
  
ambari-server/src/test/java/org/apache/ambari/server/state/cluster/ClusterImplTest.java
 627ade9 

Diff: https://reviews.apache.org/r/48722/diff/


Testing
---

Manual testing.

Unit tests in succeeded.


Thanks,

Sebastian Toader

Review Request 49026: Ambari alert "NameNode Last Checkpoint" failing when NameNode is HA


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49026/
---

Review request for Ambari and Vitalyi Brodetskyi.


Bugs: AMBARI-17339
https://issues.apache.org/jira/browse/AMBARI-17339


Repository: ambari


Description
---

The "NameNode Last Checkpoint" alert is implemented in
_alert_checkpoint_time.py_. This script uses the value of **dfs.namenode.http-
address** property for querying the last checkpoint timestamp.

This works fine when the NN is not in HA mode. However when NN is in HA mode
this property is either not defined or might not be pointing to the right NN
instance.

The script needs to be improved such as to figure out if the NN is in HA mode.
If it's in HA mode do not use **dfs.namenode.http-address** property but
rather determine the correct URL to connect to similarly as how it's done in
the _alert_ha_namenode_health.py_


Diffs
-

  
ambari-common/src/main/python/resource_management/libraries/functions/namenode_ha_utils.py
 ca36e62 
  
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/alerts/alert_checkpoint_time.py
 cb0d49e 
  ambari-server/src/test/python/stacks/2.0.6/HDFS/test_alert_checkpoint_time.py 
26b3b9c 

Diff: https://reviews.apache.org/r/49026/diff/


Testing
---

mvn clean test


Thanks,

Andrew Onischuk

Re: Review Request 48844: Operations during upgrade are permitted by all roles


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48844/#review138848
---


Ship it!




Ship It!

- Robert Levas


On June 21, 2016, 10:56 a.m., Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48844/
> ---
> 
> (Updated June 21, 2016, 10:56 a.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Nate Cole, and Robert Levas.
> 
> 
> Bugs: AMBARI-17292
> https://issues.apache.org/jira/browse/AMBARI-17292
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> ambari-server --hash  
> 9a2943ba77371f1c20b4f3da900abb7c2e89d22b  
> Build# ambari-server-2.4.0.0-591.x86_64
> 
> **Steps**
> 
>   1. Create user with different roles like Cluster user, Service 
> Administrator etc.
>   2. Login as Ambari admin user and start Express Upgrade (register version, 
> install packages and start EU)
>   3. Pause the Upgrade at any step that requires manual intervention (like 
> stop YARN queue or backup DB or even at Finalize step)
>   4. Logout and login as cluster user
> 
> **Result**:  
> The logged in user has complete access to Upgrade Wizard and can resume
> upgrade  
> Also do actions like Downgrade, 'Ignore and Proceed', 'Retry'
> 
> The same is true for other roles like service administrator too, both during
> upgrade and downgrade
> 
> **Expected Result:** Only Ambari Admin and Cluster Admin should be permitted 
> to perform actions during cluster upgrade
> 
> Screenshots attached for reference while logged in as cluster user role
> (cluser)
> 
> Another observation: While upgrade is in progress, login in a different
> session as cluster user - the cluster user can view the upgrade wizard in
> exact same way as admin
> 
> 
> Diffs
> -
> 
>   ambari-server/pom.xml f0bd67c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeItemResourceProvider.java
>  0719430 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
>  fb3ae69 
>   
> ambari-server/src/main/java/org/apache/ambari/server/security/authorization/AmbariAuthorizationFilter.java
>  922a215 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderHDP22Test.java
>  c052a6c 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java
>  5bcfd86 
> 
> Diff: https://reviews.apache.org/r/48844/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>

Re: Review Request 47730: Improve TimelineMetricsCache eviction/flush logic using a cache library



> On Июнь 3, 2016, 3:20 п.п., Jonathan Hurley wrote:
> > Ship It!
> 
> Jonathan Hurley wrote:
> Ping. This review is about ~ 3 weeks old. Was it committed? Can we close 
> it out?

It has been moved out of 2.4.0.


- Dmytro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47730/#review136067
---


On Июнь 3, 2016, 3:15 п.п., Dmytro Sen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47730/
> ---
> 
> (Updated Июнь 3, 2016, 3:15 п.п.)
> 
> 
> Review request for Ambari, Aravindan Vijayan, Jonathan Hurley, Myroslav 
> Papirkovskyy, and Sid Wagle.
> 
> 
> Bugs: AMBARI-16821
> https://issues.apache.org/jira/browse/AMBARI-16821
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The TimelineMetricsCache implementation in the metrics sink side is currently 
> a ConcurrentSkipListMap. It is better to use pre built cache libraries like 
> Guava that offer more support.
> 
> 
> Diffs
> -
> 
>   ambari-metrics/ambari-metrics-common/pom.xml 41ba62e 
>   
> ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCache.java
>  0bed7d0 
>   
> ambari-metrics/ambari-metrics-common/src/test/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCacheTest.java
>  18d973c 
>   
> ambari-metrics/ambari-metrics-hadoop-sink/src/test/java/org/apache/hadoop/metrics2/sink/timeline/HadoopTimelineMetricsSinkTest.java
>  ea7f72d 
>   ambari-project/pom.xml 2fbb1e1 
>   ambari-server/pom.xml 20d3fab 
> 
> Diff: https://reviews.apache.org/r/47730/diff/
> 
> 
> Testing
> ---
> 
> Unit tests passed
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Re: Review Request 47730: Improve TimelineMetricsCache eviction/flush logic using a cache library



> On June 3, 2016, 11:20 a.m., Jonathan Hurley wrote:
> > Ship It!

Ping. This review is about ~ 3 weeks old. Was it committed? Can we close it out?


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47730/#review136067
---


On June 3, 2016, 11:15 a.m., Dmytro Sen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47730/
> ---
> 
> (Updated June 3, 2016, 11:15 a.m.)
> 
> 
> Review request for Ambari, Aravindan Vijayan, Jonathan Hurley, Myroslav 
> Papirkovskyy, and Sid Wagle.
> 
> 
> Bugs: AMBARI-16821
> https://issues.apache.org/jira/browse/AMBARI-16821
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The TimelineMetricsCache implementation in the metrics sink side is currently 
> a ConcurrentSkipListMap. It is better to use pre built cache libraries like 
> Guava that offer more support.
> 
> 
> Diffs
> -
> 
>   ambari-metrics/ambari-metrics-common/pom.xml 41ba62e 
>   
> ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCache.java
>  0bed7d0 
>   
> ambari-metrics/ambari-metrics-common/src/test/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCacheTest.java
>  18d973c 
>   
> ambari-metrics/ambari-metrics-hadoop-sink/src/test/java/org/apache/hadoop/metrics2/sink/timeline/HadoopTimelineMetricsSinkTest.java
>  ea7f72d 
>   ambari-project/pom.xml 2fbb1e1 
>   ambari-server/pom.xml 20d3fab 
> 
> Diff: https://reviews.apache.org/r/47730/diff/
> 
> 
> Testing
> ---
> 
> Unit tests passed
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Re: Review Request 48972: AMBARI-17253 Ambari Alert causes too many wanings in ZooKeeper logs.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48972/#review138843
---



None of the hadoop services should dump exceptions to logs on simple port 
checks. If, as a user, I can simply telnet to a port over and over and trash 
the logs, it's something that needs to be fixed on the service side. 

The solution here doesn't allow for future alerts to use this type of command. 
Instead, you should be extending port alerts to specify the socket command to 
send.


ambari-agent/src/main/python/ambari_agent/alerts/port_alert.py (lines 133 - 135)


We don't hardcode specific workarounds into the alerts framework. It's not 
maintainable.


- Jonathan Hurley


On June 20, 2016, 6:29 p.m., Masahiro Tanaka wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48972/
> ---
> 
> (Updated June 20, 2016, 6:29 p.m.)
> 
> 
> Review request for Ambari, Florian Barca, Jonathan Hurley, and Nate Cole.
> 
> 
> Bugs: AMBARI-17253
> https://issues.apache.org/jira/browse/AMBARI-17253
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> There are too many WARNING in ZooKeeper log.
> ```
> 2016-06-15 21:02:15,405 - WARN  
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of 
> stream exception
> EndOfStreamException: Unable to read additional data from client sessionid 
> 0x0, likely client has closed socket
> at 
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
> at 
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
> at java.lang.Thread.run(Thread.java:745)
> ```
> 
> It may be because of Ambari Alert. Ambari Alert pings to the zookeeper port 
> to do monitoring.
> We should use 'ruok' to monitor zookeepers.
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/main/python/ambari_agent/alerts/port_alert.py 1918327 
>   ambari-agent/src/test/python/ambari_agent/TestPortAlert.py dffa56c 
> 
> Diff: https://reviews.apache.org/r/48972/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> ```
> +1 overall. Here are the results of testing the latest attachment 
> http://issues.apache.org/jira/secure/attachment/12811835/AMBARI-17253.2.patch
> against trunk revision .
> +1 @author. The patch does not contain any @author tags.
> +1 tests included. The patch appears to include 1 new or modified test files.
> +1 javac. The applied patch does not increase the total number of javac 
> compiler warnings.
> +1 release audit. The applied patch does not increase the total number of 
> release audit warnings.
> +1 core tests. The patch passed unit tests in .
> Test results: 
> https://builds.apache.org/job/Ambari-trunk-test-patch/7427//testReport/
> Console output: 
> https://builds.apache.org/job/Ambari-trunk-test-patch/7427//console
> This message is automatically generated.
> ```
> 
> 
> Thanks,
> 
> Masahiro Tanaka
> 
>

Re: Review Request 48844: Operations during upgrade are permitted by all roles


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48844/
---

(Updated June 21, 2016, 2:56 p.m.)


Review request for Ambari, Jonathan Hurley, Nate Cole, and Robert Levas.


Bugs: AMBARI-17292
https://issues.apache.org/jira/browse/AMBARI-17292


Repository: ambari


Description
---

ambari-server --hash  
9a2943ba77371f1c20b4f3da900abb7c2e89d22b  
Build# ambari-server-2.4.0.0-591.x86_64

**Steps**

  1. Create user with different roles like Cluster user, Service Administrator 
etc.
  2. Login as Ambari admin user and start Express Upgrade (register version, 
install packages and start EU)
  3. Pause the Upgrade at any step that requires manual intervention (like stop 
YARN queue or backup DB or even at Finalize step)
  4. Logout and login as cluster user

**Result**:  
The logged in user has complete access to Upgrade Wizard and can resume
upgrade  
Also do actions like Downgrade, 'Ignore and Proceed', 'Retry'

The same is true for other roles like service administrator too, both during
upgrade and downgrade

**Expected Result:** Only Ambari Admin and Cluster Admin should be permitted to 
perform actions during cluster upgrade

Screenshots attached for reference while logged in as cluster user role
(cluser)

Another observation: While upgrade is in progress, login in a different
session as cluster user - the cluster user can view the upgrade wizard in
exact same way as admin


Diffs (updated)
-

  ambari-server/pom.xml f0bd67c 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeItemResourceProvider.java
 0719430 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
 fb3ae69 
  
ambari-server/src/main/java/org/apache/ambari/server/security/authorization/AmbariAuthorizationFilter.java
 922a215 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderHDP22Test.java
 c052a6c 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java
 5bcfd86 

Diff: https://reviews.apache.org/r/48844/diff/


Testing
---

mvn clean test


Thanks,

Andrew Onischuk

Re: Review Request 49012: AMBARI-17330 Ambari changes to support kerberized Ranger tagsync


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49012/#review138836
---




ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py
 (line 287)


The lower-case form of `current_host` should be forced - 
`current_host.lower()`



ambari-server/src/main/resources/common-services/RANGER/0.6.0/kerberos.json 
(lines 114 - 122)


This should go into a Jinja2 template file, which is comon practice for 
JAAS files.  See 
https://github.com/apache/ambari/blob/10ad39cfd87a2f11bdfbc3470fdf112bf1d83c1c/ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5/package/templates/zookeeper_jaas.conf.j2


- Robert Levas


On June 21, 2016, 3:12 a.m., Mugdha Varadkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49012/
> ---
> 
> (Updated June 21, 2016, 3:12 a.m.)
> 
> 
> Review request for Ambari, Gautam Borad, Robert Levas, Srimanth Gunturi, and 
> Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-17330
> https://issues.apache.org/jira/browse/AMBARI-17330
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Remove below properties from ranger-tagsync-site.xml:
> - ranger.tagsync.atlas.to.ranger.service.mapping
> - ranger.tagsync.atlas.custom.resource.mappers
> 
> Add below property in ranger-tagsync-site.xml:
> - ranger.tagsync.source.atlasrest.username (default - empty)
> 
> Add below properties in tagsync-application-properties.xml only if cluster is 
> kerberized:
> - 
> atlas.jaas.KafkaClient.loginModuleName=com.sun.security.auth.module.Krb5LoginModule
> - atlas.jaas.KafkaClient.loginModuleControlFlag=required
> - atlas.jaas.KafkaClient.option.useKeyTab=true
> - atlas.jaas.KafkaClient.option.storeKey=true
> - atlas.jaas.KafkaClient.option.serviceName=kafka
> - atlas.jaas.KafkaClient.option.keyTab=tagsync_keytab_path
> - atlas.jaas.KafkaClient.option.principal=tagsync_jaas_principal
> - atlas.kafka.sasl.kerberos.service.name=kafka
> - atlas.kafka.security.protocol=SASL_PLAINTEXT
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py
>  9e0fb7c 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.6.0/configuration/ranger-tagsync-site.xml
>  c5a575d 
>   ambari-server/src/main/resources/common-services/RANGER/0.6.0/kerberos.json 
> cd34cd9 
> 
> Diff: https://reviews.apache.org/r/49012/diff/
> 
> 
> Testing
> ---
> 
> Tested Ranger installation on centos 6 in secure env
> 
> 
> Thanks,
> 
> Mugdha Varadkar
> 
>

Re: Review Request 48413: Fix misnamed Zookeeper connect strings in Log Search


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48413/
---

(Updated June 21, 2016, 2:19 p.m.)


Review request for Ambari, Daniel Gergely, Oliver Szabo, Robert Nettleton, 
Sumit Mohanty, and Sebastian Toader.


Changes
---

Include the changes since so the patch is up-to-date.


Bugs: AMBARI-17117
https://issues.apache.org/jira/browse/AMBARI-17117


Repository: ambari


Description
---

Variables/properties holding zookeeper connect strings are misnamed as zk_host, 
or zk_hosts, which may be misleading. Variable / property names fixed.


Diffs (updated)
-

  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/logconfig/FetchConfigFromSolr.java
 f2d074a 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputSolr.java
 a7202e7 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/SolrUtil.java
 29feef7 
  ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/config.json.j2 
b6301ca 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/logfeeder.properties
 6cba826 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/output.config.json.j2
 a485600 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/OutputSolrTest.java
 afbccca 
  
ambari-logsearch/ambari-logsearch-portal/src/main/java/org/apache/ambari/logsearch/dao/AuditSolrDao.java
 03ff0ff 
  
ambari-logsearch/ambari-logsearch-portal/src/main/java/org/apache/ambari/logsearch/dao/ServiceLogsSolrDao.java
 14125bc 
  
ambari-logsearch/ambari-logsearch-portal/src/main/java/org/apache/ambari/logsearch/dao/SolrDaoBase.java
 4564752 
  
ambari-logsearch/ambari-logsearch-portal/src/main/java/org/apache/ambari/logsearch/dao/UserConfigSolrDao.java
 edf1dcc 
  
ambari-logsearch/ambari-logsearch-portal/src/main/java/org/apache/ambari/logsearch/solr/metrics/SolrMetricsLoader.java
 21c010f 
  
ambari-logsearch/ambari-logsearch-portal/src/main/resources/logsearch.properties.j2
 0a94186 
  
ambari-logsearch/ambari-logsearch-solr-client/src/main/java/org/apache/ambari/logsearch/solr/AmbariSolrCloudCLI.java
 ad5d789 
  
ambari-logsearch/ambari-logsearch-solr-client/src/main/java/org/apache/ambari/logsearch/solr/AmbariSolrCloudClient.java
 32a1821 
  
ambari-logsearch/ambari-logsearch-solr-client/src/main/java/org/apache/ambari/logsearch/solr/AmbariSolrCloudClientBuilder.java
 50204c4 
  
ambari-logsearch/ambari-logsearch-solr-client/src/main/java/org/apache/ambari/logsearch/solr/commands/GetSolrHostsCommand.java
 f814678 
  
ambari-logsearch/ambari-logsearch-solr-client/src/test/java/org/apache/ambari/logsearch/solr/AmbariSolrCloudClientTest.java
 c382c14 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logfeeder.properties.j2
 9887e76 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logsearch.properties.j2
 0bbba0f 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/output.config.json.j2
 b31f39b 
  ambari-server/src/test/python/stacks/2.4/configs/default.json ff548e0 

Diff: https://reviews.apache.org/r/48413/diff/


Testing
---

Tested on local cluster

ambari-logsearch-logfeeder:
Tests run: 35, Failures: 0, Errors: 0, Skipped: 0


Thanks,

Miklos Gergely

Re: Review Request 48793: Log Level filter not applied before Log Search Starts at first


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48793/
---

(Updated June 21, 2016, 2:16 p.m.)


Review request for Ambari, Don Bosco Durai, Dharmesh Makwana, Oliver Szabo, 
Robert Nettleton, and Sumit Mohanty.


Changes
---

Updated to match latest trunk


Bugs: Ambari-17277
https://issues.apache.org/jira/browse/Ambari-17277


Repository: ambari


Description
---

After Log Search is started it sends the filters to the Solr which describes 
what log levels should be persisted. In the meantime Logfeeders start to push 
their data, which is not filtered. The result is misleading, the user may find 
it add that some specific level logs are loaded, others are not.


Diffs (updated)
-

  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeeder.java
 166c0f3 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/logconfig/FetchConfigFromSolr.java
 f2d074a 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/logconfig/LogFeederConstants.java
 f61dc1b 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputSolr.java
 a7202e7 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/SolrUtil.java
 29feef7 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/logfeeder.properties
 6cba826 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/OutputSolrTest.java
 afbccca 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logfeeder.properties.j2
 9887e76 

Diff: https://reviews.apache.org/r/48793/diff/


Testing
---

logfeeder:
Tests run: 35, Failures: 0, Errors: 0, Skipped: 0

ambari-server
Total run:1062
Total errors:0
Total failures:0


Thanks,

Miklos Gergely

Re: Review Request 49024: Client installs failed on debian 7


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49024/#review138835
---


Ship it!




Ship It!

- Vitalyi Brodetskyi


On Червень 21, 2016, 2:12 після полудня, Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49024/
> ---
> 
> (Updated Червень 21, 2016, 2:12 після полудня)
> 
> 
> Review request for Ambari and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-17337
> https://issues.apache.org/jira/browse/AMBARI-17337
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Type of install : UI  
> Install failures of : HBASE_CLIENT, HDFS_CLIENT, YARN_CLIENT, MR2 client,
> MAHOUT, SLIDER and other clients  
> OS : Debian7  
> Cause : Looks like a common cause :  
> Applying File['/usr/hdp/current/hadoop-client/conf/core-site.xml'](https://hor
> tonworks.jira.com/wiki/display/BUG/%27%2Fusr%2Fhdp%2Fcurrent%2Fhadoop-
> client%2Fconf%2Fcore-site.xml%27) failed, parent directory /usr/hdp/current
> /hadoop-client/conf doesn't exist
> 
> Artifacts here :  
>  /ambari-hosts/artifacts/screenshots/com.hw.ambari.ui.tests.heavyweights.TestRe
> InstallHostComponents/testA_PrepareCluster/_18_14_49_36_Page_with_URL__https_1
> 72_22_107_798443_installer_step10__has_not_been_loaded_within_180/lastAvailabl
> eRequests.txt>
> 
> Live cluster here :  
> 
> 
> Checked the folder :
> 
> 
> 
> 
> root@nat-os-d7-sbhat-ambari-hosts-6-5:~# ls -l /usr/hdp/current/
> total 12
> drwxr-x--- 4 cstm-falcon root 4096 Jun 19 07:37 falcon-client
> drwxr-x--- 3 rootroot 4096 Jun 19 07:37 hbase-client
> drwxr-xr-x 4 rootroot 4096 Jun 19 07:37 storm-client
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/packages_analyzer.py
>  860c1af 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py
>  dfba302 
> 
> Diff: https://reviews.apache.org/r/49024/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>

Re: Review Request 49014: AMBARI-17331. Determine Tez for Hive2 config 'tez.am.resource.memory.mb' based on cluster capacity.

2016-06-21 Thread Sumit Mohanty


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49014/#review138833
---


Ship it!




Ship It!

- Sumit Mohanty


On June 21, 2016, 9:25 a.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49014/
> ---
> 
> (Updated June 21, 2016, 9:25 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17331
> https://issues.apache.org/jira/browse/AMBARI-17331
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The current computations done for tez ends up with a large AM size.
> For LLAP, we'd like stack-advisor to change these computations.
> 
> - Total Cluster Memory <=4GB - Tez AM size = 256MB and then normalized based 
> on YARN minimum container size.
> - Total Cluster Memory >4GB && <= 72GB - Tez AM size = 512MB and then 
> normalized based on YARN minimum container size.
> - Total Cluster Memory >72GB - Tez AM size = 1536MB and then normalized based 
> on YARN minimum container size.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/tez-interactive-site.xml
>  3c83c5c 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 76654c3 
>   ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 
> 1bc53ea 
> 
> Diff: https://reviews.apache.org/r/49014/diff/
> 
> 
> Testing
> ---
> 
> Python UT added.
>   - Python UT passes.
> 
> 
> Thanks,
> 
> Swapan Shridhar
> 
>

Review Request 49024: Client installs failed on debian 7


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49024/
---

Review request for Ambari and Vitalyi Brodetskyi.


Bugs: AMBARI-17337
https://issues.apache.org/jira/browse/AMBARI-17337


Repository: ambari


Description
---

Type of install : UI  
Install failures of : HBASE_CLIENT, HDFS_CLIENT, YARN_CLIENT, MR2 client,
MAHOUT, SLIDER and other clients  
OS : Debian7  
Cause : Looks like a common cause :  
Applying File['/usr/hdp/current/hadoop-client/conf/core-site.xml'](https://hor
tonworks.jira.com/wiki/display/BUG/%27%2Fusr%2Fhdp%2Fcurrent%2Fhadoop-
client%2Fconf%2Fcore-site.xml%27) failed, parent directory /usr/hdp/current
/hadoop-client/conf doesn't exist

Artifacts here :  


Live cluster here :  


Checked the folder :




root@nat-os-d7-sbhat-ambari-hosts-6-5:~# ls -l /usr/hdp/current/
total 12
drwxr-x--- 4 cstm-falcon root 4096 Jun 19 07:37 falcon-client
drwxr-x--- 3 rootroot 4096 Jun 19 07:37 hbase-client
drwxr-xr-x 4 rootroot 4096 Jun 19 07:37 storm-client


Diffs
-

  
ambari-common/src/main/python/resource_management/libraries/functions/packages_analyzer.py
 860c1af 
  
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py
 dfba302 

Diff: https://reviews.apache.org/r/49024/diff/


Testing
---

mvn clean test


Thanks,

Andrew Onischuk

Re: Review Request 48829: AMBARI-17245 Failed to start Hive metastore due to UnicodeDecodeError

2016-06-21 Thread Masahiro Tanaka



> On June 17, 2016, 9:44 a.m., Andrew Onischuk wrote:
> > Ship It!

Thank you! could you commit this ?


- Masahiro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48829/#review138187
---


On June 17, 2016, 12:18 a.m., Masahiro Tanaka wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48829/
> ---
> 
> (Updated June 17, 2016, 12:18 a.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk and jun aoki.
> 
> 
> Bugs: AMBARI-17245
> https://issues.apache.org/jira/browse/AMBARI-17245
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When we start hivemestore in the environment described above, we got an error
> ```
> Traceback (most recent call last):
>   File 
> "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py",
>  line 254, in 
> HiveMetastore().execute()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 257, in execute
> method(env)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 686, in restart
> self.start(env, upgrade_type=upgrade_type)
>   File 
> "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py",
>  line 61, in start
> hive_service('metastore', action='start', upgrade_type=upgrade_type)
>   File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", 
> line 89, in thunk
> return fn(*args, **kwargs)
>   File 
> "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py",
>  line 70, in hive_service
> pid = get_user_call_output.get_user_call_output(format("cat {pid_file}"), 
> user=params.hive_user, is_checked_call=False)[1]
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/get_user_call_output.py",
>  line 59, in get_user_call_output
> err_msg = Logger.filter_text(("Execution of '%s' returned %d. %s") % 
> (command_string, code, all_output))
>   File "/usr/lib/python2.6/site-packages/resource_management/core/logger.py", 
> line 101, in filter_text
> text = text.replace(unprotected_string, protected_string)
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 117: 
> ordinal not in range(128)
> ```
> I think the reason why this error occurs is because 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/get_user_call_output.py".
>  This script calls system command (e.g. cat), writes stdout & stderr in the 
> tempfile, and read it. The outputs (stdout, stderr) are in UTF-8 encoded 
> Japanese in the above environment.
> 
> When reading string from file, python regards the file as ASCII encoded by 
> default. But if the file is utf-8 encoded, this causes an error.
> 
> We should decode UTF-8 when reading the file.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/get_user_call_output.py
>  016161c 
> 
> Diff: https://reviews.apache.org/r/48829/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test && manual test
> 
> 
> Thanks,
> 
> Masahiro Tanaka
> 
>

Re: Review Request 48741: LogSearch Solr kerberos support


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48741/
---

(Updated June 21, 2016, 12:32 p.m.)


Review request for Ambari, Don Bosco Durai, Miklos Gergely, Robert Levas, 
Robert Nettleton, Sumit Mohanty, and Sebastian Toader.


Changes
---

- put log4 content to a more proper place


Bugs: AMBARI-16736
https://issues.apache.org/jira/browse/AMBARI-16736


Repository: ambari


Description
---

Basuc logsearch solr kerberos support.


Diffs (updated)
-

  
ambari-common/src/main/python/resource_management/libraries/functions/solr_cloud_util.py
 e0a30ed 
  ambari-logsearch/ambari-logsearch-logfeeder/pom.xml 0684a95 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputSolr.java
 6e3248b 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/logfeeder.properties
 fc72ce0 
  ambari-logsearch/ambari-logsearch-portal/pom.xml e83c75b 
  
ambari-logsearch/ambari-logsearch-portal/src/main/java/org/apache/ambari/logsearch/dao/SolrDaoBase.java
 cd4ef93 
  
ambari-logsearch/ambari-logsearch-portal/src/main/resources/logsearch.properties
 37c8317 
  ambari-logsearch/ambari-logsearch-solr-client/pom.xml b634928 
  
ambari-logsearch/ambari-logsearch-solr-client/src/main/java/org/apache/ambari/logsearch/solr/AmbariSolrCloudCLI.java
 a2da737 
  
ambari-logsearch/ambari-logsearch-solr-client/src/main/java/org/apache/ambari/logsearch/solr/AmbariSolrCloudClient.java
 2805b0b 
  
ambari-logsearch/ambari-logsearch-solr-client/src/main/java/org/apache/ambari/logsearch/solr/AmbariSolrCloudClientBuilder.java
 0813221 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
 9d1ce8f 
  ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/alerts.json 
85762e0 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logfeeder-env.xml
 d27067c 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logsearch-env.xml
 3682e5d 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logsearch-solr-env.xml
 2420be0 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/kerberos.json 
6dd4aa7 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logfeeder.py
 ce7f71c 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logsearch.py
 2b5fdf7 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logsearch_common.py
 d0ac389 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logsearch_solr.py
 b55f3d6 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/params.py
 5afc04c 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logfeeder.py
 5d1ca85 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logsearch.py
 e51720f 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logsearch_solr.py
 f5f4e3f 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/status_params.py
 1efe605 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logfeeder.properties.j2
 6a52708 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logfeeder_jaas.conf.j2
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logsearch.properties.j2
 9ada5bf 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logsearch_jaas.conf.j2
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logsearch_solr_jaas.conf.j2
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/zoo.cfg.j2
 1f3808b 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/logsearch-log4j.xml.j2
 15345cf 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/logsearch-solr-env.sh.j2
 9f76d18 
  ambari-web/app/data/HDP2/site_properties.js 09b2572 

Diff: https://reviews.apache.org/r/48741/diff/


Testing
---

ambari server python unit test output:

Total run:1061
Total errors:0
Total failures:0


Thanks,

Oliver Szabo

Re: Review Request 48549: AMBARI-17165 Handle Java patches execution during Ranger upgrade

2016-06-21 Thread Mugdha Varadkar



> On June 21, 2016, 1:04 a.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml,
> >  line 704
> > 
> >
> > Nonrolling upgrades have already stopped all services, no need to call 
> > "stop" again.

Updated in latest patch


> On June 21, 2016, 1:04 a.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.4.xml,
> >  line 942
> > 
> >
> > If have 2 consecutive groups that run on all hosts, then don't need the 
> > "sequential" flag. That is used when you have say 3 hosts, one step runs on 
> > A and another step runs on B, and you want step A to finish then step B, as 
> > opposed to them running in parallel because B isn't already running a task.

Updated in latest patch


- Mugdha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48549/#review138725
---


On June 21, 2016, 12:11 p.m., Mugdha Varadkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48549/
> ---
> 
> (Updated June 21, 2016, 12:11 p.m.)
> 
> 
> Review request for Ambari, Gautam Borad, Jonathan Hurley, Nate Cole, Srimanth 
> Gunturi, and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-17165
> https://issues.apache.org/jira/browse/AMBARI-17165
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Revisiting execution of java patches during upgrade for Ranger Service
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py
>  9e0fb7c 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py
>  cedb3f9 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py
>  4db9f10 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.3.xml
>  4fd5801 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.4.xml
>  272a3cc 
>   ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml 
> b0cff68 
>   ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.4.xml 
> 0b72254 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml
>  111b432 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  9365646 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  42c4979 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.3.xml 
> 712241b 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> 4187d64 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> 3461ad4 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml
>  e83b54b 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml
>  b1c2468d 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.4.xml 
> 4065e87 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml 
> 426b452 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml
>  460e6b3 
>   ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.5.xml 
> 6ce4c81 
> 
> Diff: https://reviews.apache.org/r/48549/diff/
> 
> 
> Testing
> ---
> 
> Tested Ranger upgrade from stack 2.4 to 2.5
> 
> 
> Thanks,
> 
> Mugdha Varadkar
> 
>

Re: Review Request 48549: AMBARI-17165 Handle Java patches execution during Ranger upgrade

2016-06-21 Thread Mugdha Varadkar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48549/
---

(Updated June 21, 2016, 12:11 p.m.)


Review request for Ambari, Gautam Borad, Jonathan Hurley, Nate Cole, Srimanth 
Gunturi, and Velmurugan Periasamy.


Changes
---

Handle Alejandro Fernandez comment to remove "sequential" flag, stop task from 
nonrolling-2.x.xml and re-order hdp-select, conf-select


Bugs: AMBARI-17165
https://issues.apache.org/jira/browse/AMBARI-17165


Repository: ambari


Description
---

Revisiting execution of java patches during upgrade for Ranger Service


Diffs (updated)
-

  
ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py
 9e0fb7c 
  
ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py
 cedb3f9 
  
ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py
 4db9f10 
  
ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.3.xml
 4fd5801 
  
ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/nonrolling-upgrade-2.4.xml
 272a3cc 
  ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml 
b0cff68 
  ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.4.xml 
0b72254 
  
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml
 111b432 
  
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
 9365646 
  
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
 42c4979 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.3.xml 
712241b 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
4187d64 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
3461ad4 
  
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml
 e83b54b 
  
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml
 b1c2468d 
  ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.4.xml 
4065e87 
  ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml 
426b452 
  
ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml
 460e6b3 
  ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.5.xml 
6ce4c81 

Diff: https://reviews.apache.org/r/48549/diff/


Testing
---

Tested Ranger upgrade from stack 2.4 to 2.5


Thanks,

Mugdha Varadkar

Re: Review Request 48741: LogSearch Solr kerberos support


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48741/
---

(Updated June 21, 2016, 12:01 p.m.)


Review request for Ambari, Don Bosco Durai, Miklos Gergely, Robert Levas, 
Robert Nettleton, Sumit Mohanty, and Sebastian Toader.


Changes
---

- rename array to dict
- more configurable log4j content/location


Bugs: AMBARI-16736
https://issues.apache.org/jira/browse/AMBARI-16736


Repository: ambari


Description
---

Basuc logsearch solr kerberos support.


Diffs (updated)
-

  
ambari-common/src/main/python/resource_management/libraries/functions/solr_cloud_util.py
 e0a30ed 
  ambari-logsearch/ambari-logsearch-logfeeder/pom.xml 0684a95 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputSolr.java
 6e3248b 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/logfeeder.properties
 fc72ce0 
  ambari-logsearch/ambari-logsearch-portal/pom.xml e83c75b 
  
ambari-logsearch/ambari-logsearch-portal/src/main/java/org/apache/ambari/logsearch/dao/SolrDaoBase.java
 cd4ef93 
  
ambari-logsearch/ambari-logsearch-portal/src/main/resources/logsearch.properties
 37c8317 
  ambari-logsearch/ambari-logsearch-solr-client/pom.xml b634928 
  
ambari-logsearch/ambari-logsearch-solr-client/src/main/java/org/apache/ambari/logsearch/solr/AmbariSolrCloudCLI.java
 a2da737 
  
ambari-logsearch/ambari-logsearch-solr-client/src/main/java/org/apache/ambari/logsearch/solr/AmbariSolrCloudClient.java
 2805b0b 
  
ambari-logsearch/ambari-logsearch-solr-client/src/main/java/org/apache/ambari/logsearch/solr/AmbariSolrCloudClientBuilder.java
 0813221 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
 9d1ce8f 
  ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/alerts.json 
85762e0 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logfeeder-env.xml
 d27067c 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logsearch-env.xml
 3682e5d 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logsearch-solr-env.xml
 2420be0 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/kerberos.json 
6dd4aa7 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logfeeder.py
 ce7f71c 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logsearch.py
 2b5fdf7 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logsearch_common.py
 d0ac389 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logsearch_solr.py
 b55f3d6 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/params.py
 5afc04c 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logfeeder.py
 5d1ca85 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logsearch.py
 e51720f 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logsearch_solr.py
 f5f4e3f 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/status_params.py
 1efe605 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logfeeder.properties.j2
 6a52708 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logfeeder_jaas.conf.j2
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logsearch.properties.j2
 9ada5bf 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logsearch_jaas.conf.j2
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logsearch_solr_jaas.conf.j2
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/zoo.cfg.j2
 1f3808b 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/logsearch-log4j.xml.j2
 15345cf 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/logsearch-solr-env.sh.j2
 9f76d18 
  ambari-web/app/data/HDP2/site_properties.js 09b2572 

Diff: https://reviews.apache.org/r/48741/diff/


Testing
---

ambari server python unit test output:

Total run:1061
Total errors:0
Total failures:0


Thanks,

Oliver Szabo

Re: Review Request 48741: LogSearch Solr kerberos support


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48741/#review138819
---


Ship it!




Ship It!

- Sebastian Toader


On June 21, 2016, 12:55 p.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48741/
> ---
> 
> (Updated June 21, 2016, 12:55 p.m.)
> 
> 
> Review request for Ambari, Don Bosco Durai, Miklos Gergely, Robert Levas, 
> Robert Nettleton, Sumit Mohanty, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-16736
> https://issues.apache.org/jira/browse/AMBARI-16736
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Basuc logsearch solr kerberos support.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/solr_cloud_util.py
>  e0a30ed 
>   ambari-logsearch/ambari-logsearch-logfeeder/pom.xml 0684a95 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputSolr.java
>  6e3248b 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/logfeeder.properties
>  fc72ce0 
>   ambari-logsearch/ambari-logsearch-portal/pom.xml e83c75b 
>   
> ambari-logsearch/ambari-logsearch-portal/src/main/java/org/apache/ambari/logsearch/dao/SolrDaoBase.java
>  cd4ef93 
>   
> ambari-logsearch/ambari-logsearch-portal/src/main/resources/logsearch.properties
>  37c8317 
>   ambari-logsearch/ambari-logsearch-solr-client/pom.xml b634928 
>   
> ambari-logsearch/ambari-logsearch-solr-client/src/main/java/org/apache/ambari/logsearch/solr/AmbariSolrCloudCLI.java
>  a2da737 
>   
> ambari-logsearch/ambari-logsearch-solr-client/src/main/java/org/apache/ambari/logsearch/solr/AmbariSolrCloudClient.java
>  2805b0b 
>   
> ambari-logsearch/ambari-logsearch-solr-client/src/main/java/org/apache/ambari/logsearch/solr/AmbariSolrCloudClientBuilder.java
>  0813221 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
>  9d1ce8f 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/alerts.json 
> 85762e0 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logfeeder-env.xml
>  d27067c 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logsearch-env.xml
>  3682e5d 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logsearch-solr-env.xml
>  2420be0 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/kerberos.json
>  6dd4aa7 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logfeeder.py
>  ce7f71c 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logsearch.py
>  2b5fdf7 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logsearch_common.py
>  d0ac389 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logsearch_solr.py
>  b55f3d6 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/params.py
>  5afc04c 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logfeeder.py
>  5d1ca85 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logsearch.py
>  e51720f 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logsearch_solr.py
>  f5f4e3f 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/status_params.py
>  1efe605 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logfeeder.properties.j2
>  6a52708 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logfeeder_jaas.conf.j2
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logsearch.properties.j2
>  9ada5bf 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logsearch_jaas.conf.j2
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logsearch_solr_jaas.conf.j2
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/zoo.cfg.j2
>  1f3808b 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/logsearch-log4j.xml.j2
>  15345cf 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/logsearch-solr-env.sh.j2
>  9f76d18 
>   ambari-web/app/data/HDP2/site_properties.js 09b2572 
> 
> Diff: https://reviews.apache.org/r/48741/diff/
> 
> 
> Testing
> ---
> 
> ambari server python unit

Re: Review Request 48741: LogSearch Solr kerberos support


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48741/#review138817
---


Ship it!




Ship It!

- Miklos Gergely


On June 21, 2016, 10:55 a.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48741/
> ---
> 
> (Updated June 21, 2016, 10:55 a.m.)
> 
> 
> Review request for Ambari, Don Bosco Durai, Miklos Gergely, Robert Levas, 
> Robert Nettleton, Sumit Mohanty, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-16736
> https://issues.apache.org/jira/browse/AMBARI-16736
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Basuc logsearch solr kerberos support.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/solr_cloud_util.py
>  e0a30ed 
>   ambari-logsearch/ambari-logsearch-logfeeder/pom.xml 0684a95 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputSolr.java
>  6e3248b 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/logfeeder.properties
>  fc72ce0 
>   ambari-logsearch/ambari-logsearch-portal/pom.xml e83c75b 
>   
> ambari-logsearch/ambari-logsearch-portal/src/main/java/org/apache/ambari/logsearch/dao/SolrDaoBase.java
>  cd4ef93 
>   
> ambari-logsearch/ambari-logsearch-portal/src/main/resources/logsearch.properties
>  37c8317 
>   ambari-logsearch/ambari-logsearch-solr-client/pom.xml b634928 
>   
> ambari-logsearch/ambari-logsearch-solr-client/src/main/java/org/apache/ambari/logsearch/solr/AmbariSolrCloudCLI.java
>  a2da737 
>   
> ambari-logsearch/ambari-logsearch-solr-client/src/main/java/org/apache/ambari/logsearch/solr/AmbariSolrCloudClient.java
>  2805b0b 
>   
> ambari-logsearch/ambari-logsearch-solr-client/src/main/java/org/apache/ambari/logsearch/solr/AmbariSolrCloudClientBuilder.java
>  0813221 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
>  9d1ce8f 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/alerts.json 
> 85762e0 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logfeeder-env.xml
>  d27067c 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logsearch-env.xml
>  3682e5d 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logsearch-solr-env.xml
>  2420be0 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/kerberos.json
>  6dd4aa7 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logfeeder.py
>  ce7f71c 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logsearch.py
>  2b5fdf7 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logsearch_common.py
>  d0ac389 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logsearch_solr.py
>  b55f3d6 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/params.py
>  5afc04c 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logfeeder.py
>  5d1ca85 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logsearch.py
>  e51720f 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logsearch_solr.py
>  f5f4e3f 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/status_params.py
>  1efe605 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logfeeder.properties.j2
>  6a52708 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logfeeder_jaas.conf.j2
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logsearch.properties.j2
>  9ada5bf 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logsearch_jaas.conf.j2
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logsearch_solr_jaas.conf.j2
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/zoo.cfg.j2
>  1f3808b 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/logsearch-log4j.xml.j2
>  15345cf 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/logsearch-solr-env.sh.j2
>  9f76d18 
>   ambari-web/app/data/HDP2/site_properties.js 09b2572 
> 
> Diff: https://reviews.apache.org/r/48741/diff/
> 
> 
> Testing
> ---
> 
> ambari server python unit

Re: Review Request 48793: Log Level filter not applied before Log Search Starts at first


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48793/#review138816
---


Ship it!




Ship It!

- Oliver Szabo


On June 20, 2016, 4:15 p.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48793/
> ---
> 
> (Updated June 20, 2016, 4:15 p.m.)
> 
> 
> Review request for Ambari, Don Bosco Durai, Dharmesh Makwana, Oliver Szabo, 
> Robert Nettleton, and Sumit Mohanty.
> 
> 
> Bugs: Ambari-17277
> https://issues.apache.org/jira/browse/Ambari-17277
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> After Log Search is started it sends the filters to the Solr which describes 
> what log levels should be persisted. In the meantime Logfeeders start to push 
> their data, which is not filtered. The result is misleading, the user may 
> find it add that some specific level logs are loaded, others are not.
> 
> 
> Diffs
> -
> 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeeder.java
>  f3dd4bf 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/logconfig/FetchConfigFromSolr.java
>  f2d074a 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/logconfig/LogFeederConstants.java
>  f61dc1b 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputSolr.java
>  b14c273 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/SolrUtil.java
>  29feef7 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/logfeeder.properties
>  076c09c 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/OutputSolrTest.java
>  afbccca 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logfeeder.properties.j2
>  6a52708 
> 
> Diff: https://reviews.apache.org/r/48793/diff/
> 
> 
> Testing
> ---
> 
> logfeeder:
> Tests run: 35, Failures: 0, Errors: 0, Skipped: 0
> 
> ambari-server
> Total run:1062
> Total errors:0
> Total failures:0
> 
> 
> Thanks,
> 
> Miklos Gergely
> 
>

Re: Review Request 49019: Registering 2.5 HDP repo URLs throwing 500 error in the backend


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49019/#review138814
---


Ship it!




Ship It!

- Vitalyi Brodetskyi


On Червень 21, 2016, 10:55 до полудня, Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49019/
> ---
> 
> (Updated Червень 21, 2016, 10:55 до полудня)
> 
> 
> Review request for Ambari, Nate Cole and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-17334
> https://issues.apache.org/jira/browse/AMBARI-17334
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Registering the version HDP-2.5 fails (Even when you check skip repo checks 
> button). The default repo entries were used
> 
> {code}
> Caught AmbariException when modifying a resource
> 
> org.apache.ambari.server.AmbariException: Version HDP-2.5.0.0 needs to belong 
> to stack HDP-2.5
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/RepositoryVersionEntity.java
>  db393b9 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/CompatibleRepositoryVersionResourceProviderTest.java
>  066e0d0 
> 
> Diff: https://reviews.apache.org/r/49019/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>

Review Request 49019: Registering 2.5 HDP repo URLs throwing 500 error in the backend

2016-06-21 Thread Dmitro Lisnichenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49019/
---

Review request for Ambari, Nate Cole and Vitalyi Brodetskyi.


Bugs: AMBARI-17334
https://issues.apache.org/jira/browse/AMBARI-17334


Repository: ambari


Description
---

Registering the version HDP-2.5 fails (Even when you check skip repo checks 
button). The default repo entries were used

{code}
Caught AmbariException when modifying a resource

org.apache.ambari.server.AmbariException: Version HDP-2.5.0.0 needs to belong 
to stack HDP-2.5
{code}


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/RepositoryVersionEntity.java
 db393b9 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/CompatibleRepositoryVersionResourceProviderTest.java
 066e0d0 

Diff: https://reviews.apache.org/r/49019/diff/


Testing
---

mvn clean test


Thanks,

Dmitro Lisnichenko

Re: Review Request 48741: LogSearch Solr kerberos support


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48741/
---

(Updated June 21, 2016, 10:55 a.m.)


Review request for Ambari, Don Bosco Durai, Miklos Gergely, Robert Levas, 
Robert Nettleton, Sumit Mohanty, and Sebastian Toader.


Changes
---

- custom log4j content for solr-client (if logsearch is not used)
- use jaas file to create collection for atlas


Bugs: AMBARI-16736
https://issues.apache.org/jira/browse/AMBARI-16736


Repository: ambari


Description
---

Basuc logsearch solr kerberos support.


Diffs (updated)
-

  
ambari-common/src/main/python/resource_management/libraries/functions/solr_cloud_util.py
 e0a30ed 
  ambari-logsearch/ambari-logsearch-logfeeder/pom.xml 0684a95 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputSolr.java
 6e3248b 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/resources/logfeeder.properties
 fc72ce0 
  ambari-logsearch/ambari-logsearch-portal/pom.xml e83c75b 
  
ambari-logsearch/ambari-logsearch-portal/src/main/java/org/apache/ambari/logsearch/dao/SolrDaoBase.java
 cd4ef93 
  
ambari-logsearch/ambari-logsearch-portal/src/main/resources/logsearch.properties
 37c8317 
  ambari-logsearch/ambari-logsearch-solr-client/pom.xml b634928 
  
ambari-logsearch/ambari-logsearch-solr-client/src/main/java/org/apache/ambari/logsearch/solr/AmbariSolrCloudCLI.java
 a2da737 
  
ambari-logsearch/ambari-logsearch-solr-client/src/main/java/org/apache/ambari/logsearch/solr/AmbariSolrCloudClient.java
 2805b0b 
  
ambari-logsearch/ambari-logsearch-solr-client/src/main/java/org/apache/ambari/logsearch/solr/AmbariSolrCloudClientBuilder.java
 0813221 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
 9d1ce8f 
  ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/alerts.json 
85762e0 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logfeeder-env.xml
 d27067c 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logsearch-env.xml
 3682e5d 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logsearch-solr-env.xml
 2420be0 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/kerberos.json 
6dd4aa7 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logfeeder.py
 ce7f71c 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logsearch.py
 2b5fdf7 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logsearch_common.py
 d0ac389 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logsearch_solr.py
 b55f3d6 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/params.py
 5afc04c 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logfeeder.py
 5d1ca85 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logsearch.py
 e51720f 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logsearch_solr.py
 f5f4e3f 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/status_params.py
 1efe605 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logfeeder.properties.j2
 6a52708 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logfeeder_jaas.conf.j2
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logsearch.properties.j2
 9ada5bf 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logsearch_jaas.conf.j2
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/logsearch_solr_jaas.conf.j2
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/zoo.cfg.j2
 1f3808b 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/logsearch-log4j.xml.j2
 15345cf 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/logsearch-solr-env.sh.j2
 9f76d18 
  ambari-web/app/data/HDP2/site_properties.js 09b2572 

Diff: https://reviews.apache.org/r/48741/diff/


Testing
---

ambari server python unit test output:

Total run:1061
Total errors:0
Total failures:0


Thanks,

Oliver Szabo

Re: Review Request 49018: Downloading sqoop client files throws error


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49018/#review138810
---


Ship it!




Ship It!

- Dmytro Sen


On Июнь 21, 2016, 9:55 д.п., Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49018/
> ---
> 
> (Updated Июнь 21, 2016, 9:55 д.п.)
> 
> 
> Review request for Ambari and Dmytro Sen.
> 
> 
> Bugs: AMBARI-17332
> https://issues.apache.org/jira/browse/AMBARI-17332
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Downloading sqoop client files from Service Actions menu throws the below
> error
> 
> 
> 
> 
> status: 500,
> message: "org.apache.ambari.server.controller.spi.SystemException: 
> Execution of "ambari-python-wrap 
> /var/lib/ambari-server/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/sqoop_client.py
>  generate_configs /var/lib/ambari-server/data/tmp/SQOOP-configuration.json 
> /var/lib/ambari-server/resources/common-services/SQOOP/1.4.4.2.0/package 
> /var/lib/ambari-server/data/tmp/structured-out.json INFO 
> /var/lib/ambari-server/data/tmp" returned 1. java.lang.Throwable: 2016-06-21 
> 07:14:04,201 - Directory['/var/lib/ambari-server/data/tmp'] 
> {'create_parents': True} 2016-06-21 07:14:04,203 - 
> XmlConfig['squoop-site.xml'] {'conf_dir': 
> '/var/lib/ambari-server/data/tmp/tmpjWQnzj', 'mode': 0644, 
> 'configuration_attributes': [EMPTY], 'configurations': [EMPTY]} 2016-06-21 
> 07:14:04,213 - Generating config: 
> /var/lib/ambari-server/data/tmp/tmpjWQnzj/squoop-site.xml 2016-06-21 
> 07:14:04,213 - 
> File['/var/lib/ambari-server/data/tmp/tmpjWQnzj/squoop-site.xml'] {'owner': 
> None, 'content': InlineTemplate(.
 ..), 'group': None, 'mode': 0644, 'encoding': 'UTF-8'} 2016-06-21 07:14:04,219 
- Directory['/var/lib/ambari-server/data/tmp/tmpjWQnzj'] {'action': ['delete']} 
2016-06-21 07:14:04,219 - Removing directory 
Directory['/var/lib/ambari-server/data/tmp/tmpjWQnzj'] and all its content 
Traceback (most recent call last): File 
"/var/lib/ambari-server/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/sqoop_client.py",
 line 66, in  SqoopClient().execute() File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
 line 257, in execute method(env) File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
 line 781, in generate_configs 
**self.generate_configs_get_xml_file_content(filename, dict) File 
"/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, 
in __init__ self.env.run() File 
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
line 160, in run self.run_action(resource
 , action) File 
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
line 124, in run_action provider_action() File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/providers/xml_config.py",
 line 66, in action_create encoding = self.resource.encoding File 
"/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, 
in __init__ self.env.run() File 
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
line 160, in run self.run_action(resource, action) File 
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
line 124, in run_action provider_action() File 
"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
 line 123, in action_create content = self._get_content() File 
"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
 line 160, in _get_content return content() File 
"/usr/lib/python2.6/site-packages/resource_management/core/source.
 py", line 51, in __call__ return self.get_content() File 
"/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 
142, in get_content rendered = self.template.render(self.context) File 
"/usr/lib/python2.6/site-packages/ambari_jinja2/environment.py", line 891, in 
render return self.environment.handle_exception(exc_info, True) File 
"", line 2, in top-level template code File 
"/usr/lib/python2.6/site-packages/ambari_jinja2/filters.py", line 176, in 
do_dictsort return sorted(value.items(), key=sort_func) File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py",
 line 73, in __getattr__ raise Fail("Configuration parameter '" + self.name + 
"' was not found in configurations dictionary!") 
resource_management.core.exceptions.Fail: Configuration parameter 'squoop-site' 
was not found in configurations dictionary! "
> }
> 
> 
> Could you please help take a look.
> 
> 
> Diffs
> -
> 
>   
>

Review Request 49014: AMBARI-17331. Determine Tez for Hive2 config 'tez.am.resource.memory.mb' based on cluster capacity.

2016-06-21 Thread Swapan Shridhar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49014/
---

Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.


Bugs: AMBARI-17331
https://issues.apache.org/jira/browse/AMBARI-17331


Repository: ambari


Description
---

The current computations done for tez ends up with a large AM size.
For LLAP, we'd like stack-advisor to change these computations.

- Total Cluster Memory <=4GB - Tez AM size = 256MB and then normalized based on 
YARN minimum container size.
- Total Cluster Memory >4GB && <= 72GB - Tez AM size = 512MB and then 
normalized based on YARN minimum container size.
- Total Cluster Memory >72GB - Tez AM size = 1536MB and then normalized based 
on YARN minimum container size.


Diffs
-

  
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/tez-interactive-site.xml
 3c83c5c 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
76654c3 
  ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 1bc53ea 

Diff: https://reviews.apache.org/r/49014/diff/


Testing
---

Python UT added.
  - Python UT passes.


Thanks,

Swapan Shridhar

Re: Review Request 48805: AMBARI-17280. RU to write out client configs that are dependencies of Hive, ATS, and Oozie during upgrades that change configs

2016-06-21 Thread Dmitro Lisnichenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48805/#review138806
---


Ship it!





ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py (line 35)


typo?


- Dmitro Lisnichenko


On June 21, 2016, 4:12 a.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48805/
> ---
> 
> (Updated June 21, 2016, 4:12 a.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, 
> Jonathan Hurley, Nate Cole, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-17280
> https://issues.apache.org/jira/browse/AMBARI-17280
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> During RU, HiveServer2 is restarted but the newer tez configs have not yet 
> been saved, which is incorrect because Hive has a dependency on Tez.
> This is important when configs change during a major stack upgrade, e.g., HDP 
> 2.4 -> 2.5. What happens today is,
> 
> * Install packages generates /etc/tez/2.5.0.0-1/0 and copies the configs from 
> /etc/tez/2.4.0.0-1/0/ to the new folder
> * If configs change during RU, then Hive is restarted and the classpath means 
> that it will pick up the older tez configs from the new /etc/tez/2.5.0.0-1/0 
> folder
> 
> 
> This problem exists for all of these components:
> 
> * HiveServer: depends on Tez and MapReduce clients
> * ATS: depends on Tez and Spark clients
> * Oozie: depends on Tez, Spark, and MapReduce clients
> 
> This problem only exists when configs change (so crossing major stack 
> version) and during RU (because it is allowed to change configs during the 
> middle of restarting services).
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
>  4eb0015 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
>  b994fce 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> 49dcb4e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
>  fb3ae69 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ClusterGrouping.java
>  80bb26c 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  63c72f7 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/spark_client.py
>  ef41453 
>   
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py
>  44239c7 
>   
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez.py
>  67466e3 
>   
> ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py
>  c79d63b 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/mapreduce2_client.py
>  db22004 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
>  90f885a 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py
>  d1ec15b 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> 4187d64 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> 3461ad4 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml 
> 426b452 
>   ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py e53eb4b 
> 
> Diff: https://reviews.apache.org/r/48805/diff/
> 
> 
> Testing
> ---
> 
> Verified during RU from HDP 2.4 to 2.5 with ATS, Hive, Tez, Oozie, and Spark
> 
> Python unit tests passed,
> --
> Total run:1062
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Re: Review Request 48863: Ambari-server upgrade results in "DB configs consistency check failed. "