Re: Review Request 50653: AMBARI-17308 Ambari Logfeeder outputs a lot of errors due to parse date

[Sebastian Toader]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50653/#review144381
---




ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerDefaultImpl.java
 (line 51)


The 'X' stands for the ISO timezone representation. 

I think grok supports ISO timezone format as well. Can you confirm that 
grok patterns do not support iso 8061 timezone format?

If it doesn't than go ahead and change for X to Z


- Sebastian Toader


On Aug. 1, 2016, 12:23 p.m., Masahiro Tanaka wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50653/
> ---
> 
> (Updated Aug. 1, 2016, 12:23 p.m.)
> 
> 
> Review request for Ambari, Daniel Gergely, Dmytro Sen, Oliver Szabo, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-17308
> https://issues.apache.org/jira/browse/AMBARI-17308
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> In logsearch_feeder service log, we got errors like below
> ```
> 2016-06-20 15:28:09,368 ERROR file=ambari-audit.log 
> org.apache.ambari.logfeeder.mapper.MapperDate LogFeederUtil.java:356 - Error 
> applying date transformation. isEpoch=false, 
> dateFormat=-MM-dd'T'HH:mm:ss.SSSZ, value=2016-06-20T15:28:08.000. 
> mapClass=map_date, input=input:source=file, 
> path=/var/log/ambari-server/ambari-audit.log, fieldName=logtime. Messages 
> suppressed before: 2
> java.text.ParseException: Unparseable date: "2016-06-20T15:28:08.000"
>   at java.text.DateFormat.parse(DateFormat.java:366)
>   at 
> org.apache.ambari.logfeeder.mapper.MapperDate.apply(MapperDate.java:83)
>   at org.apache.ambari.logfeeder.filter.Filter.apply(Filter.java:154)
>   at 
> org.apache.ambari.logfeeder.filter.FilterGrok.applyMessage(FilterGrok.java:291)
>   at 
> org.apache.ambari.logfeeder.filter.FilterGrok.flush(FilterGrok.java:320)
>   at org.apache.ambari.logfeeder.input.Input.flush(Input.java:125)
>   at 
> org.apache.ambari.logfeeder.input.InputFile.processFile(InputFile.java:430)
>   at org.apache.ambari.logfeeder.input.InputFile.start(InputFile.java:260)
>   at org.apache.ambari.logfeeder.input.Input.run(Input.java:100)
>   at java.lang.Thread.run(Thread.java:745)
> ```
> ambari-audit.log is like below
> ```
> 2016-07-21T01:52:49.875+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
> 2016-07-21T01:52:49.905+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu16/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu16), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu16/2.x/BUILDS/2.5.0.0-1025)
> 2016-07-21T01:52:50.015+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-UTILS-1.1.0.21),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-UTILS-1.1.0.21), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/ubuntu14)
> ```
> I think date format of the ambari-audit.log (2016-07-21T01:52:49.875+09) 
> should be like 2016-07-21T01:52:49.875+0900, since grok-pattern can't handle 
> 2016-07-21T01:52:49.875+09 format.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerDefaultImpl.java
>  ea6524f 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/input.config-ambari.json.j2
>  5f751bf 
> 
> Diff: https://reviews.apache.org/r/50653/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Masahiro Tanaka
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

[Saisai Shao]

> On Aug. 2, 2016, 12:34 a.m., Jayush Luniya wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java,
> >  line 2273
> > 
> >
> > We need to replace hdp_full_version with full_stack_version. The SPARK 
> > code has been cleaned up to remove hdp hardcodings in Ambari 2.4. If you 
> > see params.py the variable hdp_full_version is not even defined. So when 
> > you upgrade ambari-server from 2.1.0 or 2.2.0 with my hotfix the config 
> > will remain -Dhdp.version={{hdp_full_version}}.
> 
> Saisai Shao wrote:
> From my understanding, you mean that in the params.py we should also take 
> care of {{hdp_full_version}} if amabri is upgraded from lower version. Can 
> you please explain more about it.
> 
> Jayush Luniya wrote:
> If you look at 
> https://github.com/apache/ambari/blob/branch-2.1.2/ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py
>  and 
> https://github.com/apache/ambari/blob/branch-2.1.2/ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-defaults.xml#L128-L133
>  you will see that we use to set these properties as  
> -Dhdp.version={{hdp_full_version}}. So when you upgrade Ambari 2.1.X -> 
> 2.4.0, these config properties will continue to be 
> -Dhdp.version={{hdp_full_version}}. However params.py doesnt have this 
> variable defined anymore. It has been renamed to full_stack_version. So when 
> we do an ambari-server upgrade we should update these config properties to 
> find and replace hdp_full_version with full_stack_version.

I understand now. So you mean that we still need to keep these codes rather 
than delete it, is that right?

Or put the logic into params.py?


- Saisai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144418
---


On Aug. 1, 2016, 1:22 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated Aug. 1, 2016, 1:22 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

[Jayush Luniya]

> On Aug. 2, 2016, 12:59 a.m., Jayush Luniya wrote:
> > It doesnt look like you have verified ambari-server upgrade scenarios with 
> > the patch. This is very important to validate. You need to ensure backward 
> > compatibility is maintained
> > 
> > Please test the following scenarios
> > Scenario 1:
> > 1. Install Ambari 2.1.0
> > 2. Deploy HDP-2.3.2
> > 3. Upgrade to Ambari 2.4.0
> > 4. Restart SPARK
> > 5. Verify SPARK restarts
> > 6. Upgrade HDP to 2.3.4 and verify upgrade succeeds
> > 7. Also upgrade to HDP 2.4+ and verify upgrade succeeds.
> > 
> > Scenario 2:
> > 1. Install Ambari 2.1.0
> > 2. Deploy HDP-2.3.2
> > 3. Upgrade to Ambari 2.2.0
> > 4. Restart Spark
> > 5. Confirm that the Spark configs are not getting written correctly due to 
> > the backward compatibility issue. Restart spark will cause issues.
> > 6. Upgrade to Hotfix build 2.2.0.13-1
> > 7. Update java_opts using following command 
> > ./configs.sh set   
> > spark-javaopts-properties content "-Dhdp.version=hdp_full_version"
> > 8. Restart SPARK service
> > 9. Verify that the Spark configs are getting written correctly.
> > 10. Now upgrade Ambari to Ambari 2.4.0
> > 11. Restart Spark
> > 12. Verify SPARK restarts
> > 13. Upgrade to HDP 2.3.4 
> > 14. Also upgrade to HDP-2.4+
> 
> Saisai Shao wrote:
> Where to get the specific version of Ambari and how to upgrade to the 
> specific version?

http://docs.hortonworks.com/index.html  has information about how to install a 
specific Ambari version and how to upgrade Ambari

Installing Ambari 2.1.2.1
http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.1/bk_Installing_HDP_AMB/content/ch_Installing_Ambari.html

Upgrading to Ambari-2.2.0
http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_upgrading_Ambari/content/_upgrading_ambari.html


- Jayush


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144425
---


On Aug. 1, 2016, 1:22 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated Aug. 1, 2016, 1:22 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

[Jayush Luniya]

> On Aug. 2, 2016, 12:34 a.m., Jayush Luniya wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java,
> >  line 2273
> > 
> >
> > We need to replace hdp_full_version with full_stack_version. The SPARK 
> > code has been cleaned up to remove hdp hardcodings in Ambari 2.4. If you 
> > see params.py the variable hdp_full_version is not even defined. So when 
> > you upgrade ambari-server from 2.1.0 or 2.2.0 with my hotfix the config 
> > will remain -Dhdp.version={{hdp_full_version}}.
> 
> Saisai Shao wrote:
> From my understanding, you mean that in the params.py we should also take 
> care of {{hdp_full_version}} if amabri is upgraded from lower version. Can 
> you please explain more about it.

If you look at 
https://github.com/apache/ambari/blob/branch-2.1.2/ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py
 and 
https://github.com/apache/ambari/blob/branch-2.1.2/ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-defaults.xml#L128-L133
 you will see that we use to set these properties as  
-Dhdp.version={{hdp_full_version}}. So when you upgrade Ambari 2.1.X -> 2.4.0, 
these config properties will continue to be -Dhdp.version={{hdp_full_version}}. 
However params.py doesnt have this variable defined anymore. It has been 
renamed to full_stack_version. So when we do an ambari-server upgrade we should 
update these config properties to find and replace hdp_full_version with 
full_stack_version.


- Jayush


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144418
---


On Aug. 1, 2016, 1:22 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated Aug. 1, 2016, 1:22 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

[Saisai Shao]

> On Aug. 2, 2016, 12:59 a.m., Jayush Luniya wrote:
> > It doesnt look like you have verified ambari-server upgrade scenarios with 
> > the patch. This is very important to validate. You need to ensure backward 
> > compatibility is maintained
> > 
> > Please test the following scenarios
> > Scenario 1:
> > 1. Install Ambari 2.1.0
> > 2. Deploy HDP-2.3.2
> > 3. Upgrade to Ambari 2.4.0
> > 4. Restart SPARK
> > 5. Verify SPARK restarts
> > 6. Upgrade HDP to 2.3.4 and verify upgrade succeeds
> > 7. Also upgrade to HDP 2.4+ and verify upgrade succeeds.
> > 
> > Scenario 2:
> > 1. Install Ambari 2.1.0
> > 2. Deploy HDP-2.3.2
> > 3. Upgrade to Ambari 2.2.0
> > 4. Restart Spark
> > 5. Confirm that the Spark configs are not getting written correctly due to 
> > the backward compatibility issue. Restart spark will cause issues.
> > 6. Upgrade to Hotfix build 2.2.0.13-1
> > 7. Update java_opts using following command 
> > ./configs.sh set   
> > spark-javaopts-properties content "-Dhdp.version=hdp_full_version"
> > 8. Restart SPARK service
> > 9. Verify that the Spark configs are getting written correctly.
> > 10. Now upgrade Ambari to Ambari 2.4.0
> > 11. Restart Spark
> > 12. Verify SPARK restarts
> > 13. Upgrade to HDP 2.3.4 
> > 14. Also upgrade to HDP-2.4+

Where to get the specific version of Ambari and how to upgrade to the specific 
version?


- Saisai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144425
---


On Aug. 1, 2016, 1:22 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated Aug. 1, 2016, 1:22 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

[Jayush Luniya]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144425
---



It doesnt look like you have verified ambari-server upgrade scenarios with the 
patch. This is very important to validate. You need to ensure backward 
compatibility is maintained

Please test the following scenarios
Scenario 1:
1. Install Ambari 2.1.0
2. Deploy HDP-2.3.2
3. Upgrade to Ambari 2.4.0
4. Restart SPARK
5. Verify SPARK restarts
6. Upgrade HDP to 2.3.4 and verify upgrade succeeds
7. Also upgrade to HDP 2.4+ and verify upgrade succeeds.

Scenario 2:
1. Install Ambari 2.1.0
2. Deploy HDP-2.3.2
3. Upgrade to Ambari 2.2.0
4. Restart Spark
5. Confirm that the Spark configs are not getting written correctly due to the 
backward compatibility issue. Restart spark will cause issues.
6. Upgrade to Hotfix build 2.2.0.13-1
7. Update java_opts using following command 
./configs.sh set   
spark-javaopts-properties content "-Dhdp.version=hdp_full_version"
8. Restart SPARK service
9. Verify that the Spark configs are getting written correctly.
10. Now upgrade Ambari to Ambari 2.4.0
11. Restart Spark
12. Verify SPARK restarts
13. Upgrade to HDP 2.3.4 
14. Also upgrade to HDP-2.4+

- Jayush Luniya


On Aug. 1, 2016, 1:22 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated Aug. 1, 2016, 1:22 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

[Jayush Luniya]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144421
---




ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
 (line 284)


You can set this to 2.3.4.0 so that we dont create this for HDP-2.3.4+ 
instead of HDP-2.4+. Also if you you want to keep it as < HDP-2.4, you can set 
max_version as "2.4.0.0" instead of "2.3.99.99". The max_version field is 
exclusive 
i.e. [min_version, max_version)


- Jayush Luniya


On Aug. 1, 2016, 1:22 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated Aug. 1, 2016, 1:22 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

[Saisai Shao]

> On Aug. 2, 2016, 12:34 a.m., Jayush Luniya wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java,
> >  line 2273
> > 
> >
> > We need to replace hdp_full_version with full_stack_version. The SPARK 
> > code has been cleaned up to remove hdp hardcodings in Ambari 2.4. If you 
> > see params.py the variable hdp_full_version is not even defined. So when 
> > you upgrade ambari-server from 2.1.0 or 2.2.0 with my hotfix the config 
> > will remain -Dhdp.version={{hdp_full_version}}.

>From my understanding, you mean that in the params.py we should also take care 
>of {{hdp_full_version}} if amabri is upgraded from lower version. Can you 
>please explain more about it.


- Saisai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144418
---


On Aug. 1, 2016, 1:22 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated Aug. 1, 2016, 1:22 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

[Jayush Luniya]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144418
---




ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
 


We need to replace hdp_full_version with full_stack_version. The SPARK code 
has been cleaned up to remove hdp hardcodings in Ambari 2.4. If you see 
params.py the variable hdp_full_version is not even defined. So when you 
upgrade ambari-server from 2.1.0 or 2.2.0 with my hotfix the config will remain 
-Dhdp.version={{hdp_full_version}}.



ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
 


Same as above



ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
 


Same as above


- Jayush Luniya


On Aug. 1, 2016, 1:22 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated Aug. 1, 2016, 1:22 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50645: Ambari upgrade to 2.4.0 adds several config properties that are not necessary and some are incorrect

[Sumit Mohanty]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50645/
---

(Updated Aug. 2, 2016, 12:20 a.m.)


Review request for Ambari, Alejandro Fernandez, Dmitro Lisnichenko, Jaimin 
Jetly, and Mugdha Varadkar.


Changes
---

Final patch with test results.


Bugs: AMBARI-17971
https://issues.apache.org/jira/browse/AMBARI-17971


Repository: ambari


Description (updated)
---

Ambari upgrade to 2.4.0 adds several config properties that are not necessary 
and some are incorrect.


Diffs
-

  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/application-properties.xml
 108c7b4 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/atlas-log4j.xml
 a071442 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/atlas-solrconfig.xml
 35e9f58 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
 1a0e797 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
 8006158 
  
ambari-server/src/main/resources/common-services/ATLAS/0.7.0.2.5/configuration/atlas-solrconfig.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/configuration/falcon-atlas-application.properties.xml
 e653651 
  
ambari-server/src/main/resources/common-services/KAFKA/0.9.0/configuration/kafka-broker.xml
 0275358 
  
ambari-server/src/main/resources/common-services/KAFKA/0.9.0/configuration/ranger-kafka-audit.xml
 5257549 
  
ambari-server/src/main/resources/common-services/KAFKA/0.9.0/configuration/ranger-kafka-plugin-properties.xml
 74cdf63 
  
ambari-server/src/main/resources/common-services/KAFKA/0.9.0/configuration/ranger-kafka-policymgr-ssl.xml
 f0fc160 
  
ambari-server/src/main/resources/common-services/KAFKA/0.9.0/configuration/ranger-kafka-security.xml
 3967e3e 
  
ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/configuration/ranger-knox-plugin-properties.xml
 f0f6f41 
  
ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/configuration/ranger-kms-audit.xml
 50a997c 
  
ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/configuration/sqoop-site.xml
 f7823d2 
  
ambari-server/src/main/resources/common-services/STORM/0.10.0/configuration/ranger-storm-audit.xml
 b7cf4c5 
  
ambari-server/src/main/resources/common-services/STORM/0.10.0/configuration/ranger-storm-policymgr-ssl.xml
 9592914 
  
ambari-server/src/main/resources/common-services/STORM/0.10.0/configuration/ranger-storm-security.xml
 5f0e944 
  
ambari-server/src/main/resources/common-services/STORM/0.9.1/configuration/storm-site.xml
 ab7d9cb 
  
ambari-server/src/main/resources/common-services/STORM/0.9.3/configuration/ranger-storm-plugin-properties.xml
 9ba58b5 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/hbase-env.xml
 7b55225 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/hbase-site.xml
 3f860e0 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/ranger-hbase-plugin-properties.xml
 adcf116 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/ranger-hdfs-plugin-properties.xml
 1e65763 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/ranger-hive-plugin-properties.xml
 d82c8b4 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/KNOX/configuration/ranger-knox-plugin-properties.xml
 ad2b1e4 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/configuration/ranger-hbase-audit.xml
 f670d7e 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/configuration/ranger-hbase-policymgr-ssl.xml
 79370bc 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/configuration/ranger-hbase-security.xml
 7ef63d8 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/configuration/ranger-hdfs-audit.xml
 63154eb 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/configuration/ranger-hdfs-policymgr-ssl.xml
 e14a9e8 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/configuration/ranger-hdfs-security.xml
 a5f1e24 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/configuration/ranger-hive-audit.xml
 874d0d5 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/configuration/ranger-hive-policymgr-ssl.xml
 14e7b16 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/configuration/ranger-hive-security.xml
 61e008f 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/KAFKA/configuration/ranger-kafka-policymgr-ssl.xml
 2f4c121 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/KNOX/configuration/ranger-knox-audit.xml
 abdf2bd 
  

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

[Sumit Mohanty]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144416
---


Ship it!




Ship It!

- Sumit Mohanty


On Aug. 1, 2016, 1:22 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated Aug. 1, 2016, 1:22 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50679: AMBARI-17983. (1). Revert AMBARI-16031 (Create "/hadoop/llap/local" on each host and disk in Kerberized cluster for LLAP), and (2). Remove the config 'hive.llap.daemon.work.d

[Sumit Mohanty]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50679/#review144411
---


Ship it!




Ship It!

- Sumit Mohanty


On Aug. 1, 2016, 11:26 p.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50679/
> ---
> 
> (Updated Aug. 1, 2016, 11:26 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, and Sumit 
> Mohanty.
> 
> 
> Bugs: AMBARI-17983
> https://issues.apache.org/jira/browse/AMBARI-17983
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> (1). Revert AMBARI-16031 (Create "/hadoop/llap/local" on each host and disk 
> in Kerberized cluster for LLAP)
>
>- Earlier, we had logic for using YARN work dirs for LLAP by HIVE config 
> 'hive.llap.daemon.work.dirs' having same value as of YARN config 
> 'yarn.nodemanager.local-dirs'.
>- In kerberized setup, we were creating separate directory for LLAP, where 
> Hive user had persmissions.
>
>
>- That is not required now, as HIVE is itself taking care of workdirs in 
> kerberized and un-kerberized environment. 
>- Thus, the revert.
> 
> (2). Remove the config 'hive.llap.daemon.work.dirs' as HIVE will manage the 
> work directories itself.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/metainfo.xml 
> 35329bc 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/nodemanager.py
>  106b734 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
>  154e433 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py
>  cf5736e 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml
>  776c2b6 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/kerberos.json 
> 4fae665 
>   ambari-web/app/controllers/main/service/item.js e3df70b 
>   ambari-web/app/messages.js 62dc058 
>   ambari-web/app/models/host_component.js 91642c9 
>   ambari-web/app/views/main/service/item.js 02ac426 
>   ambari-web/test/views/main/service/item_test.js ecfa7de 
> 
> Diff: https://reviews.apache.org/r/50679/diff/
> 
> 
> Testing
> ---
> 
> Backend:
> 
> - Python UT passes.
> 
> 
> UI:
> 
> - $ npm test
>  29426 tests complete (27 seconds)
>  154 tests pending
> 
> 
> Thanks,
> 
> Swapan Shridhar
> 
>

Re: Review Request 50679: AMBARI-17983. (1). Revert AMBARI-16031 (Create "/hadoop/llap/local" on each host and disk in Kerberized cluster for LLAP), and (2). Remove the config 'hive.llap.daemon.work.d

[Sumit Mohanty]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50679/#review144410
---




ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
 (line 206)


Are these two lines needed?


- Sumit Mohanty


On Aug. 1, 2016, 11:26 p.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50679/
> ---
> 
> (Updated Aug. 1, 2016, 11:26 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, and Sumit 
> Mohanty.
> 
> 
> Bugs: AMBARI-17983
> https://issues.apache.org/jira/browse/AMBARI-17983
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> (1). Revert AMBARI-16031 (Create "/hadoop/llap/local" on each host and disk 
> in Kerberized cluster for LLAP)
>
>- Earlier, we had logic for using YARN work dirs for LLAP by HIVE config 
> 'hive.llap.daemon.work.dirs' having same value as of YARN config 
> 'yarn.nodemanager.local-dirs'.
>- In kerberized setup, we were creating separate directory for LLAP, where 
> Hive user had persmissions.
>
>
>- That is not required now, as HIVE is itself taking care of workdirs in 
> kerberized and un-kerberized environment. 
>- Thus, the revert.
> 
> (2). Remove the config 'hive.llap.daemon.work.dirs' as HIVE will manage the 
> work directories itself.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/metainfo.xml 
> 35329bc 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/nodemanager.py
>  106b734 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
>  154e433 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py
>  cf5736e 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml
>  776c2b6 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/kerberos.json 
> 4fae665 
>   ambari-web/app/controllers/main/service/item.js e3df70b 
>   ambari-web/app/messages.js 62dc058 
>   ambari-web/app/models/host_component.js 91642c9 
>   ambari-web/app/views/main/service/item.js 02ac426 
>   ambari-web/test/views/main/service/item_test.js ecfa7de 
> 
> Diff: https://reviews.apache.org/r/50679/diff/
> 
> 
> Testing
> ---
> 
> Backend:
> 
> - Python UT passes.
> 
> 
> UI:
> 
> - $ npm test
>  29426 tests complete (27 seconds)
>  154 tests pending
> 
> 
> Thanks,
> 
> Swapan Shridhar
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

[Saisai Shao]

> On Aug. 1, 2016, 9:27 p.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py,
> >  line 130
> > 
> >
> > The addition of -Dhdp.version should also be under condition 
> > check_stack_feature(StackFeature.SPARK_JAVA_OPTS_SUPPORT, 
> > effective_version).
> > 
> > I assume -Dhdp.version is to be added only for HDP-2.3 and below.
> 
> Saisai Shao wrote:
> I think it's fine to leave as it is, since in HDP-2.4 above version, we 
> will remove this configuration and not generate the java-opts file.

Also this java-opts file is not used for any other user-level configuration, so 
it is safe to delete the file.


- Saisai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144388
---


On Aug. 1, 2016, 1:22 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated Aug. 1, 2016, 1:22 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

[Saisai Shao]

> On Aug. 1, 2016, 9:27 p.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py,
> >  line 130
> > 
> >
> > The addition of -Dhdp.version should also be under condition 
> > check_stack_feature(StackFeature.SPARK_JAVA_OPTS_SUPPORT, 
> > effective_version).
> > 
> > I assume -Dhdp.version is to be added only for HDP-2.3 and below.

I think it's fine to leave as it is, since in HDP-2.4 above version, we will 
remove this configuration and not generate the java-opts file.


- Saisai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144388
---


On Aug. 1, 2016, 1:22 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated Aug. 1, 2016, 1:22 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50679: AMBARI-17983. (1). Revert AMBARI-16031 (Create "/hadoop/llap/local" on each host and disk in Kerberized cluster for LLAP), and (2). Remove the config 'hive.llap.daemon.work.d

[Swapan Shridhar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50679/
---

(Updated Aug. 1, 2016, 11:26 p.m.)


Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, and Sumit Mohanty.


Changes
---

Updated the review with 'hive.llap.daemon.work.dirs' config removal.


Bugs: AMBARI-17983
https://issues.apache.org/jira/browse/AMBARI-17983


Repository: ambari


Description
---

(1). Revert AMBARI-16031 (Create "/hadoop/llap/local" on each host and disk in 
Kerberized cluster for LLAP)
   
   - Earlier, we had logic for using YARN work dirs for LLAP by HIVE config 
'hive.llap.daemon.work.dirs' having same value as of YARN config 
'yarn.nodemanager.local-dirs'.
   - In kerberized setup, we were creating separate directory for LLAP, where 
Hive user had persmissions.
   
   
   - That is not required now, as HIVE is itself taking care of workdirs in 
kerberized and un-kerberized environment. 
   - Thus, the revert.

(2). Remove the config 'hive.llap.daemon.work.dirs' as HIVE will manage the 
work directories itself.


Diffs (updated)
-

  ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/metainfo.xml 
35329bc 
  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/nodemanager.py
 106b734 
  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
 154e433 
  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py
 cf5736e 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml
 776c2b6 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/kerberos.json 
4fae665 
  ambari-web/app/controllers/main/service/item.js e3df70b 
  ambari-web/app/messages.js 62dc058 
  ambari-web/app/models/host_component.js 91642c9 
  ambari-web/app/views/main/service/item.js 02ac426 
  ambari-web/test/views/main/service/item_test.js ecfa7de 

Diff: https://reviews.apache.org/r/50679/diff/


Testing
---

Backend:

- Python UT passes.


UI:

- $ npm test
 29426 tests complete (27 seconds)
 154 tests pending


Thanks,

Swapan Shridhar

Re: Review Request 50679: AMBARI-17983. (1). Revert AMBARI-16031 (Create "/hadoop/llap/local" on each host and disk in Kerberized cluster for LLAP), and (2). Remove the config 'hive.llap.daemon.work.d

[Jaimin Jetly]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50679/#review144404
---


Ship it!




Ship It!

- Jaimin Jetly


On Aug. 1, 2016, 11:26 p.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50679/
> ---
> 
> (Updated Aug. 1, 2016, 11:26 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, and Sumit 
> Mohanty.
> 
> 
> Bugs: AMBARI-17983
> https://issues.apache.org/jira/browse/AMBARI-17983
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> (1). Revert AMBARI-16031 (Create "/hadoop/llap/local" on each host and disk 
> in Kerberized cluster for LLAP)
>
>- Earlier, we had logic for using YARN work dirs for LLAP by HIVE config 
> 'hive.llap.daemon.work.dirs' having same value as of YARN config 
> 'yarn.nodemanager.local-dirs'.
>- In kerberized setup, we were creating separate directory for LLAP, where 
> Hive user had persmissions.
>
>
>- That is not required now, as HIVE is itself taking care of workdirs in 
> kerberized and un-kerberized environment. 
>- Thus, the revert.
> 
> (2). Remove the config 'hive.llap.daemon.work.dirs' as HIVE will manage the 
> work directories itself.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/metainfo.xml 
> 35329bc 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/nodemanager.py
>  106b734 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
>  154e433 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py
>  cf5736e 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml
>  776c2b6 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/kerberos.json 
> 4fae665 
>   ambari-web/app/controllers/main/service/item.js e3df70b 
>   ambari-web/app/messages.js 62dc058 
>   ambari-web/app/models/host_component.js 91642c9 
>   ambari-web/app/views/main/service/item.js 02ac426 
>   ambari-web/test/views/main/service/item_test.js ecfa7de 
> 
> Diff: https://reviews.apache.org/r/50679/diff/
> 
> 
> Testing
> ---
> 
> Backend:
> 
> - Python UT passes.
> 
> 
> UI:
> 
> - $ npm test
>  29426 tests complete (27 seconds)
>  154 tests pending
> 
> 
> Thanks,
> 
> Swapan Shridhar
> 
>

Review Request 50679: AMBARI-17983. (1). Revert AMBARI-16031 (Create "/hadoop/llap/local" on each host and disk in Kerberized cluster for LLAP), and (2). Remove the config 'hive.llap.daemon.work.dirs

[Swapan Shridhar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50679/
---

Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, and Sumit Mohanty.


Bugs: AMBARI-17983
https://issues.apache.org/jira/browse/AMBARI-17983


Repository: ambari


Description
---

(1). Revert AMBARI-16031 (Create "/hadoop/llap/local" on each host and disk in 
Kerberized cluster for LLAP)
   
   - Earlier, we had logic for using YARN work dirs for LLAP by HIVE config 
'hive.llap.daemon.work.dirs' having same value as of YARN config 
'yarn.nodemanager.local-dirs'.
   - In kerberized setup, we were creating separate directory for LLAP, where 
Hive user had persmissions.
   
   
   - That is not required now, as HIVE is itself taking care of workdirs in 
kerberized and un-kerberized environment. 
   - Thus, the revert.

(2). Remove the config 'hive.llap.daemon.work.dirs' as HIVE will manage the 
work directories itself.


Diffs
-

  ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/metainfo.xml 
35329bc 
  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/nodemanager.py
 106b734 
  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
 154e433 
  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py
 cf5736e 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/kerberos.json 
4fae665 
  ambari-web/app/controllers/main/service/item.js e3df70b 
  ambari-web/app/messages.js 62dc058 
  ambari-web/app/models/host_component.js 91642c9 
  ambari-web/app/views/main/service/item.js 02ac426 
  ambari-web/test/views/main/service/item_test.js ecfa7de 

Diff: https://reviews.apache.org/r/50679/diff/


Testing
---

Backend:

- Python UT passes.


UI:

- $ npm test
 29426 tests complete (27 seconds)
 154 tests pending


Thanks,

Swapan Shridhar

Re: Review Request 50666: AMBARI-10145 : Add support for tee to another Storage service?

[Jameel Mohamed]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50666/
---

(Updated Aug. 1, 2016, 10:11 p.m.)


Review request for Ambari, Aravindan Vijayan, Sumit Mohanty, and Sid Wagle.


Bugs: AMBARI-10145
https://issues.apache.org/jira/browse/AMBARI-10145


Repository: ambari


Description
---

Added new Interface to drain aggregated metrics to external storage. Added unit 
test and sample implementation (TimelineMetricsAggregatorMemorySink.java) to 
store metrics in memory.


Diffs (updated)
-

  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixHBaseAccessor.java
 dfd730d 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricConfiguration.java
 e82d65a 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricsAggregatorSink.java
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixHBaseAccessorTest.java
 0ea668a 
  
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricsAggregatorMemorySink.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/50666/diff/


Testing
---

Manually tested.


Added unit test.


mvn clean test


Thanks,

Jameel Mohamed

Re: Review Request 50666: AMBARI-10145 : Add support for tee to another Storage service?

[Jameel Mohamed]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50666/
---

(Updated Aug. 1, 2016, 10:05 p.m.)


Review request for Ambari, Aravindan Vijayan, Sumit Mohanty, and Sid Wagle.


Changes
---

Addressed all review comments


Bugs: AMBARI-10145
https://issues.apache.org/jira/browse/AMBARI-10145


Repository: ambari


Description
---

Added new Interface to drain aggregated metrics to external storage. Added unit 
test and sample implementation (TimelineMetricsAggregatorMemorySink.java) to 
store metrics in memory.


Diffs (updated)
-

  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixHBaseAccessor.java
 dfd730d 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricConfiguration.java
 e82d65a 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricsAggregatorSink.java
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixHBaseAccessorTest.java
 0ea668a 
  
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricsAggregatorMemorySink.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/50666/diff/


Testing
---

Manually tested.


Added unit test.


mvn clean test


Thanks,

Jameel Mohamed

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

[Sumit Mohanty]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144388
---




ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
 (line 129)


The addition of -Dhdp.version should also be under condition 
check_stack_feature(StackFeature.SPARK_JAVA_OPTS_SUPPORT, effective_version).

I assume -Dhdp.version is to be added only for HDP-2.3 and below.


- Sumit Mohanty


On Aug. 1, 2016, 1:22 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated Aug. 1, 2016, 1:22 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50595: AMBARI-17955: Enabled download of berkeley DB jar file to be provisioned during install

[Venkat Ranganathan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50595/
---

(Updated Aug. 1, 2016, 2:21 p.m.)


Review request for Ambari and Alejandro Fernandez.


Changes
---

Fixed another UT failure in ambariserver


Bugs: AMBARI-17955
https://issues.apache.org/jira/browse/AMBARI-17955


Repository: ambari


Description
---

Enabled the registration of optionally downloaded berkeley db jar file.   
Introduce Berekley DB as a jdbc-db (even though it is actually not.   Download 
je-5.0.73.jar if not already present and inform users on how to register

also, fixed Atlas integration changes with classname and packagename


Diffs (updated)
-

  ambari-server/src/main/python/ambari_server/dbConfiguration.py fc68a4c 
  ambari-server/src/main/python/ambari_server/dbConfiguration_linux.py d09bd77 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon.py
 dd1cda0 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py
 ebe3634 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
8d5cdc9 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
f8ba3e9 
  ambari-server/src/test/python/TestAmbariServer.py 279a172 
  ambari-server/src/test/python/stacks/2.1/FALCON/test_falcon_server.py 2dd0da9 

Diff: https://reviews.apache.org/r/50595/diff/


Testing
---

Tested downloading of the jar file by registering with ambari server
Tested Falcon/Atlas component names


Thanks,

Venkat Ranganathan

Re: Review Request 50666: AMBARI-10145 : Add support for tee to another Storage service?

[Jameel Mohamed]

> On Aug. 1, 2016, 6:52 p.m., Ajit Kumar wrote:
> > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TimelineMetricsAggregatorMemorySink.java,
> >  line 68
> > 
> >
> > Any reason why getters are not part of interface?

Getter are added just for testing purpose. Since this is sink interface which 
define only those sink methods.


> On Aug. 1, 2016, 6:52 p.m., Ajit Kumar wrote:
> > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricsAggregatorSink.java,
> >  line 27
> > 
> >
> > Some documentation?

Added comments


> On Aug. 1, 2016, 6:52 p.m., Ajit Kumar wrote:
> > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TimelineMetricsAggregatorMemorySink.java,
> >  line 37
> > 
> >
> > Any particular reason for having static block? Why not instantiated 
> > while declaring the variable?

No particular reason. Changed it


> On Aug. 1, 2016, 6:52 p.m., Ajit Kumar wrote:
> > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TimelineMetricsAggregatorMemorySink.java,
> >  line 69
> > 
> >
> > Do you want to return copy or immutable map?

Changed to immutable map


> On Aug. 1, 2016, 6:52 p.m., Ajit Kumar wrote:
> > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixHBaseAccessorTest.java,
> >  line 256
> > 
> >
> > Map clusterAggregateMap 
> > = new HashMap<>();

done


- Jameel


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50666/#review144370
---


On Aug. 1, 2016, 6:31 p.m., Jameel Mohamed wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50666/
> ---
> 
> (Updated Aug. 1, 2016, 6:31 p.m.)
> 
> 
> Review request for Ambari, Aravindan Vijayan, Sumit Mohanty, and Sid Wagle.
> 
> 
> Bugs: AMBARI-10145
> https://issues.apache.org/jira/browse/AMBARI-10145
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Added new Interface to drain aggregated metrics to external storage. Added 
> unit test and sample implementation 
> (TimelineMetricsAggregatorMemorySink.java) to store metrics in memory.
> 
> 
> Diffs
> -
> 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixHBaseAccessor.java
>  dfd730d 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricConfiguration.java
>  e82d65a 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricsAggregatorSink.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TimelineMetricsAggregatorMemorySink.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixHBaseAccessorTest.java
>  0ea668a 
> 
> Diff: https://reviews.apache.org/r/50666/diff/
> 
> 
> Testing
> ---
> 
> Manually tested.
> 
> 
> Added unit test.
> 
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Jameel Mohamed
> 
>

Re: Review Request 50645: Ambari upgrade to 2.4.0 adds several config properties that are not necessary and some are incorrect

[Sumit Mohanty]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50645/
---

(Updated Aug. 1, 2016, 8:17 p.m.)


Review request for Ambari, Alejandro Fernandez, Dmitro Lisnichenko, Jaimin 
Jetly, and Mugdha Varadkar.


Changes
---

Updated with additional changes from Mugdha Varadkar


Bugs: AMBARI-17971
https://issues.apache.org/jira/browse/AMBARI-17971


Repository: ambari


Description
---

Ambari upgrade to 2.4.0 adds several config properties that are not necessary 
and some are incorrect. (interim patch)


Diffs (updated)
-

  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/application-properties.xml
 108c7b4 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/atlas-log4j.xml
 a071442 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/atlas-solrconfig.xml
 35e9f58 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
 1a0e797 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
 8006158 
  
ambari-server/src/main/resources/common-services/ATLAS/0.7.0.2.5/configuration/atlas-solrconfig.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/configuration/falcon-atlas-application.properties.xml
 e653651 
  
ambari-server/src/main/resources/common-services/KAFKA/0.9.0/configuration/kafka-broker.xml
 0275358 
  
ambari-server/src/main/resources/common-services/KAFKA/0.9.0/configuration/ranger-kafka-audit.xml
 5257549 
  
ambari-server/src/main/resources/common-services/KAFKA/0.9.0/configuration/ranger-kafka-plugin-properties.xml
 74cdf63 
  
ambari-server/src/main/resources/common-services/KAFKA/0.9.0/configuration/ranger-kafka-policymgr-ssl.xml
 f0fc160 
  
ambari-server/src/main/resources/common-services/KAFKA/0.9.0/configuration/ranger-kafka-security.xml
 3967e3e 
  
ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/configuration/ranger-knox-plugin-properties.xml
 f0f6f41 
  
ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/configuration/ranger-kms-audit.xml
 50a997c 
  
ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/configuration/sqoop-site.xml
 f7823d2 
  
ambari-server/src/main/resources/common-services/STORM/0.10.0/configuration/ranger-storm-audit.xml
 b7cf4c5 
  
ambari-server/src/main/resources/common-services/STORM/0.10.0/configuration/ranger-storm-policymgr-ssl.xml
 9592914 
  
ambari-server/src/main/resources/common-services/STORM/0.10.0/configuration/ranger-storm-security.xml
 5f0e944 
  
ambari-server/src/main/resources/common-services/STORM/0.9.1/configuration/storm-site.xml
 ab7d9cb 
  
ambari-server/src/main/resources/common-services/STORM/0.9.3/configuration/ranger-storm-plugin-properties.xml
 9ba58b5 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/hbase-env.xml
 7b55225 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/hbase-site.xml
 3f860e0 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/ranger-hbase-plugin-properties.xml
 adcf116 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/configuration/ranger-hdfs-plugin-properties.xml
 1e65763 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/ranger-hive-plugin-properties.xml
 d82c8b4 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/KNOX/configuration/ranger-knox-plugin-properties.xml
 ad2b1e4 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/configuration/ranger-hbase-audit.xml
 f670d7e 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/configuration/ranger-hbase-policymgr-ssl.xml
 79370bc 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/configuration/ranger-hbase-security.xml
 7ef63d8 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/configuration/ranger-hdfs-audit.xml
 63154eb 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/configuration/ranger-hdfs-policymgr-ssl.xml
 e14a9e8 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/configuration/ranger-hdfs-security.xml
 a5f1e24 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/configuration/ranger-hive-audit.xml
 874d0d5 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/configuration/ranger-hive-policymgr-ssl.xml
 14e7b16 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/configuration/ranger-hive-security.xml
 61e008f 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/KAFKA/configuration/ranger-kafka-policymgr-ssl.xml
 2f4c121 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/KNOX/configuration/ranger-knox-audit.xml
 abdf2bd 
  

Re: Review Request 50653: AMBARI-17308 Ambari Logfeeder outputs a lot of errors due to parse date

[Oliver Szabo]

> On Aug. 1, 2016, 12:35 p.m., Oliver Szabo wrote:
> > Ship It!

i will test this out a bit more, my logs looks like this locally:

2016-08-01T17:41:29.739Z, User(null), RemoteIp(192.168.64.1), Operation(User 
login), Roles(
), Status(Failed), Reason(Authentication required) ... based on that the actual 
solution is working, so I check it could work in both cases


- Oliver


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50653/#review144326
---


On Aug. 1, 2016, 10:23 a.m., Masahiro Tanaka wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50653/
> ---
> 
> (Updated Aug. 1, 2016, 10:23 a.m.)
> 
> 
> Review request for Ambari, Daniel Gergely, Dmytro Sen, Oliver Szabo, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-17308
> https://issues.apache.org/jira/browse/AMBARI-17308
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> In logsearch_feeder service log, we got errors like below
> ```
> 2016-06-20 15:28:09,368 ERROR file=ambari-audit.log 
> org.apache.ambari.logfeeder.mapper.MapperDate LogFeederUtil.java:356 - Error 
> applying date transformation. isEpoch=false, 
> dateFormat=-MM-dd'T'HH:mm:ss.SSSZ, value=2016-06-20T15:28:08.000. 
> mapClass=map_date, input=input:source=file, 
> path=/var/log/ambari-server/ambari-audit.log, fieldName=logtime. Messages 
> suppressed before: 2
> java.text.ParseException: Unparseable date: "2016-06-20T15:28:08.000"
>   at java.text.DateFormat.parse(DateFormat.java:366)
>   at 
> org.apache.ambari.logfeeder.mapper.MapperDate.apply(MapperDate.java:83)
>   at org.apache.ambari.logfeeder.filter.Filter.apply(Filter.java:154)
>   at 
> org.apache.ambari.logfeeder.filter.FilterGrok.applyMessage(FilterGrok.java:291)
>   at 
> org.apache.ambari.logfeeder.filter.FilterGrok.flush(FilterGrok.java:320)
>   at org.apache.ambari.logfeeder.input.Input.flush(Input.java:125)
>   at 
> org.apache.ambari.logfeeder.input.InputFile.processFile(InputFile.java:430)
>   at org.apache.ambari.logfeeder.input.InputFile.start(InputFile.java:260)
>   at org.apache.ambari.logfeeder.input.Input.run(Input.java:100)
>   at java.lang.Thread.run(Thread.java:745)
> ```
> ambari-audit.log is like below
> ```
> 2016-07-21T01:52:49.875+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
> 2016-07-21T01:52:49.905+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu16/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu16), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu16/2.x/BUILDS/2.5.0.0-1025)
> 2016-07-21T01:52:50.015+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-UTILS-1.1.0.21),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-UTILS-1.1.0.21), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/ubuntu14)
> ```
> I think date format of the ambari-audit.log (2016-07-21T01:52:49.875+09) 
> should be like 2016-07-21T01:52:49.875+0900, since grok-pattern can't handle 
> 2016-07-21T01:52:49.875+09 format.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerDefaultImpl.java
>  ea6524f 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/input.config-ambari.json.j2
>  5f751bf 
> 
> Diff: https://reviews.apache.org/r/50653/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Masahiro Tanaka
> 
>

Re: Review Request 50666: AMBARI-10145 : Add support for tee to another Storage service?

[Ajit Kumar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50666/#review144370
---




ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricsAggregatorSink.java
 (line 27)


Some documentation?



ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricsAggregatorSink.java
 (line 29)


public keyword is not required in interfaces.



ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TimelineMetricsAggregatorMemorySink.java
 (line 37)


Any particular reason for having static block? Why not instantiated while 
declaring the variable?



ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TimelineMetricsAggregatorMemorySink.java
 (line 38)


You don't need to add type.
hostAggregateRecords = new HashMap<>(); should work.



ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TimelineMetricsAggregatorMemorySink.java
 (line 47)


As per our coding convention formatting for if else block should be 
if () {

} else {

}



ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TimelineMetricsAggregatorMemorySink.java
 (line 49)


Instead of storing hostAggregateMap, it should be copied and stored for 
following reason:
- if hostAggregateMap is immutable, putAll in subsequent 
saveHostAggregateRecords call will fail.
- Any add/delete operation on hostAggregateMap after calling 
saveHostAggregateRecords will be impact hostAggregateRecords.



ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TimelineMetricsAggregatorMemorySink.java
 (line 59)


same here



ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TimelineMetricsAggregatorMemorySink.java
 (line 68)


Any reason why getters are not part of interface?



ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TimelineMetricsAggregatorMemorySink.java
 (line 69)


Do you want to return copy or immutable map?



ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixHBaseAccessorTest.java
 (line 256)


Map clusterAggregateMap = 
new HashMap<>();


- Ajit Kumar


On Aug. 1, 2016, 6:31 p.m., Jameel Mohamed wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50666/
> ---
> 
> (Updated Aug. 1, 2016, 6:31 p.m.)
> 
> 
> Review request for Ambari, Aravindan Vijayan, Sumit Mohanty, and Sid Wagle.
> 
> 
> Bugs: AMBARI-10145
> https://issues.apache.org/jira/browse/AMBARI-10145
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Added new Interface to drain aggregated metrics to external storage. Added 
> unit test and sample implementation 
> (TimelineMetricsAggregatorMemorySink.java) to store metrics in memory.
> 
> 
> Diffs
> -
> 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixHBaseAccessor.java
>  dfd730d 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricConfiguration.java
>  e82d65a 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricsAggregatorSink.java
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TimelineMetricsAggregatorMemorySink.java
>  PRE-CREATION 
>   
> 

Re: Review Request 50656: Improvements/fixes for AMBARI_INFRA initial service definition

[Jayush Luniya]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50656/#review144371
---


Ship it!




Ship It!

- Jayush Luniya


On Aug. 1, 2016, 12:22 p.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50656/
> ---
> 
> (Updated Aug. 1, 2016, 12:22 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jayush Luniya, Robert 
> Nettleton, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17974
> https://issues.apache.org/jira/browse/AMBARI-17974
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> contains a few fixes/imporvements for AMBARI_INFRA service definition:
> - role command order fix
> - implemented service check (test with kerberos too ... check only one of the 
> solrs has started or not)
> - quick links for infra Solrs (adimin UI)
> - configuration fixes (1 display name is missing + adding configuration 
> properties during ambari upgrade is not needed)
> - set the order of AMBARI_INFRA service on 'Select Services' page on the UI
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/configuration/infra-solr-client-log4j.xml
>  f597492 
>   
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/configuration/infra-solr-env.xml
>  dfbaac9 
>   
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/configuration/infra-solr-log4j.xml
>  b192a28 
>   
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/configuration/infra-solr-xml.xml
>  1d9d8ef 
>   
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/metainfo.xml
>  a1cee49 
>   
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/package/scripts/params.py
>  898a35c 
>   
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/package/scripts/service_check.py
>  a5a94fc 
>   
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/quicklinks/quicklinks.json
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/AMBARI_INFRA/role_command_order.json
>  34654c2 
>   ambari-web/app/models/stack_service.js 3e2d5b4 
> 
> Diff: https://reviews.apache.org/r/50656/diff/
> 
> 
> Testing
> ---
> 
> Total run:1034
> Total errors:0
> Total failures:0
> OK
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 1:14.391s
> [INFO] Finished at: Mon Aug 01 14:16:11 CEST 2016
> [INFO] Final Memory: 59M/985M
> [INFO] ---
> 
> 
> Thanks,
> 
> Oliver Szabo
> 
>

Re: Review Request 50595: AMBARI-17955: Enabled download of berkeley DB jar file to be provisioned during install

[Venkat Ranganathan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50595/
---

(Updated Aug. 1, 2016, 11:38 a.m.)


Review request for Ambari and Alejandro Fernandez.


Changes
---

Fixed a trailing blank


Bugs: AMBARI-17955
https://issues.apache.org/jira/browse/AMBARI-17955


Repository: ambari


Description
---

Enabled the registration of optionally downloaded berkeley db jar file.   
Introduce Berekley DB as a jdbc-db (even though it is actually not.   Download 
je-5.0.73.jar if not already present and inform users on how to register

also, fixed Atlas integration changes with classname and packagename


Diffs (updated)
-

  ambari-server/src/main/python/ambari_server/dbConfiguration.py fc68a4c 
  ambari-server/src/main/python/ambari_server/dbConfiguration_linux.py d09bd77 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon.py
 dd1cda0 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py
 ebe3634 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
8d5cdc9 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
f8ba3e9 
  ambari-server/src/test/python/stacks/2.1/FALCON/test_falcon_server.py 2dd0da9 

Diff: https://reviews.apache.org/r/50595/diff/


Testing
---

Tested downloading of the jar file by registering with ambari server
Tested Falcon/Atlas component names


Thanks,

Venkat Ranganathan

Review Request 50666: AMBARI-10145 : Add support for tee to another Storage service?

[Jameel Mohamed]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50666/
---

Review request for Ambari, Aravindan Vijayan, Sumit Mohanty, and Sid Wagle.


Bugs: AMBARI-10145
https://issues.apache.org/jira/browse/AMBARI-10145


Repository: ambari


Description
---

Added new Interface to drain aggregated metrics to external storage. Added unit 
test and sample implementation (TimelineMetricsAggregatorMemorySink.java) to 
store metrics in memory.


Diffs
-

  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixHBaseAccessor.java
 dfd730d 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricConfiguration.java
 e82d65a 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricsAggregatorSink.java
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TimelineMetricsAggregatorMemorySink.java
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixHBaseAccessorTest.java
 0ea668a 

Diff: https://reviews.apache.org/r/50666/diff/


Testing
---

Manually tested.


Added unit test.


mvn clean test


Thanks,

Jameel Mohamed

Re: Review Request 50616: Coverity Scan Security Vulnerability - SQL injection

[Oliver Szabo]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50616/#review144362
---


Ship it!




Ship It!

- Oliver Szabo


On July 29, 2016, 9:19 p.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50616/
> ---
> 
> (Updated July 29, 2016, 9:19 p.m.)
> 
> 
> Review request for Ambari, Mahadev Konar, Sumit Mohanty, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-17962
> https://issues.apache.org/jira/browse/AMBARI-17962
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The Ambari coverity scan found two "High impact security" issues, both SQL 
> Injections.  They are both the same coding issue, but one is in 
> OracleConnector.java, and one is in the analogous method in 
> PostgresConnector.java.
> 
> This is the key description:
> ```
> CID 167755 (#1 of 1): SQL injection (SQLI)9. sql_taint: Insecure 
> concatenation of a SQL statement. The value searchClause is tainted.
> 
> Perform one of the following to guard against SQL injection attacks.
> * Parameterize the SQL statement using ? positional characters. Bind the 
> tainted values to the ? positional parameters using one of the 
> PreparedStatement.set* methods.
> * Validate user-supplied values against predefined constant values. 
> Concatenate these constant values into the SQL statement.
> * Cast tainted values to safe types such as integers. Concatenate these type 
> safe values into the statement.
> 
> [More 
> Information|https://scan3.coverity.com/doc/en/cov_checker_ref.html#id_sql_generic]
> ```
> 
> This is the one in OracleConnector.java, lines 32 -55:
> 
> ```
> 32  @Override
>   8. taint_path_param: Parameter searchClause receives the tainted data.
> 33  protected PreparedStatement getQualifiedPS(Statements statement, String 
> searchClause, Workflows.WorkflowDBEntry.WorkflowFields field, boolean 
> sortAscending, int offset, int limit) throws IOException {
> 34if (db == null)
> 35  throw new IOException("db not initialized");
> 36
> 37String order = " ORDER BY " + field.toString() + " " + (sortAscending ? 
> SORT_ASC : SORT_DESC);
> 38
> 39String query = "select * \n" +
> 40"  from ( select " +
> 41//"/*+ FIRST_ROWS(n) */ \n" +
> 42"  a.*, ROWNUM rnum \n" +
> 43"  from ("
>   CID 167755 (#1 of 1): SQL injection (SQLI)9. sql_taint: Insecure 
> concatenation of a SQL statement. The value searchClause is tainted.
>   Perform one of the following to guard against SQL injection attacks.
> 
> Parameterize the SQL statement using ? positional characters. Bind the 
> tainted values to the ? positional parameters using one of the 
> PreparedStatement.set* methods.
> Validate user-supplied values against predefined constant values. 
> Concatenate these constant values into the SQL statement.
> Cast tainted values to safe types such as integers. Concatenate these 
> type safe values into the statement.
> 
> More Information
> 44+ statement.getStatementString() + searchClause + order +
> 45") a \n" +
> 46"  where ROWNUM <= " + (offset + limit) + ") \n" +
> 47"where rnum  >= " + offset;
> 48
> 49try {
>   10. sql_sink: Passing the tainted value query to the SQL API 
> java.sql.Connection.prepareStatement(java.lang.String) may allow an attacker 
> to inject SQL.
> 50  return db.prepareStatement(query);
> 51} catch (SQLException e) {
> 52  throw new IOException(e);
> 53}
> 54
> 55  }
> ```
> This is the one in PostgresConnector.java, lines 495-504:
> 
> ``` 
>8. taint_path_param: Parameter searchClause receives the tainted data.
> 495  protected PreparedStatement getQualifiedPS(Statements statement, String 
> searchClause) throws IOException {
> 496if (db == null)
> 497  throw new IOException("postgres db not initialized");
> 498try {
> 499  // LOG.debug("preparing " + statement.getStatementString() + 
> searchClause);
>CID 167743 (#1 of 1): SQL injection (SQLI)9. sql_taint: Insecure 
> concatenation of a SQL statement. The value searchClause is tainted. Passing 
> the tainted command to the SQL API 
> java.sql.Connection.prepareStatement(java.lang.String) may allow an attacker 
> to inject SQL.
>Perform one of the following to guard against SQL injection attacks.
> 
> Parameterize the SQL statement using ? positional characters. Bind the 
> tainted values to the ? positional parameters using one of the 
> PreparedStatement.set* methods.
> Validate user-supplied values against predefined constant values. 
> Concatenate these constant values into the SQL statement.
> Cast tainted values to safe types such as integers. Concatenate these 
> type safe values into 

Re: Review Request 50595: AMBARI-17955: Enabled download of berkeley DB jar file to be provisioned during install

[Venkat Ranganathan]

> On July 31, 2016, 4:34 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/python/ambari_server/dbConfiguration_linux.py, line 
> > 1131
> > 
> >
> > We should remove the version from the jar name

This is the only version supported - like what we have for sqljdbc4


- Venkat


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50595/#review144291
---


On Aug. 1, 2016, 10:38 a.m., Venkat Ranganathan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50595/
> ---
> 
> (Updated Aug. 1, 2016, 10:38 a.m.)
> 
> 
> Review request for Ambari and Alejandro Fernandez.
> 
> 
> Bugs: AMBARI-17955
> https://issues.apache.org/jira/browse/AMBARI-17955
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Enabled the registration of optionally downloaded berkeley db jar file.   
> Introduce Berekley DB as a jdbc-db (even though it is actually not.   
> Download je-5.0.73.jar if not already present and inform users on how to 
> register
> 
> also, fixed Atlas integration changes with classname and packagename
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/python/ambari_server/dbConfiguration.py fc68a4c 
>   ambari-server/src/main/python/ambari_server/dbConfiguration_linux.py 
> d09bd77 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon.py
>  dd1cda0 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py
>  ebe3634 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 8d5cdc9 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> f8ba3e9 
> 
> Diff: https://reviews.apache.org/r/50595/diff/
> 
> 
> Testing
> ---
> 
> Tested downloading of the jar file by registering with ambari server
> Tested Falcon/Atlas component names
> 
> 
> Thanks,
> 
> Venkat Ranganathan
> 
>

Re: Review Request 50595: AMBARI-17955: Enabled download of berkeley DB jar file to be provisioned during install

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50595/#review144358
---




ambari-server/src/main/python/ambari_server/dbConfiguration.py (line 46)


Unit tests are failing, this is a dictionary so it needs the key and value.


- Alejandro Fernandez


On Aug. 1, 2016, 5:38 p.m., Venkat Ranganathan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50595/
> ---
> 
> (Updated Aug. 1, 2016, 5:38 p.m.)
> 
> 
> Review request for Ambari and Alejandro Fernandez.
> 
> 
> Bugs: AMBARI-17955
> https://issues.apache.org/jira/browse/AMBARI-17955
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Enabled the registration of optionally downloaded berkeley db jar file.   
> Introduce Berekley DB as a jdbc-db (even though it is actually not.   
> Download je-5.0.73.jar if not already present and inform users on how to 
> register
> 
> also, fixed Atlas integration changes with classname and packagename
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/python/ambari_server/dbConfiguration.py fc68a4c 
>   ambari-server/src/main/python/ambari_server/dbConfiguration_linux.py 
> d09bd77 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon.py
>  dd1cda0 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py
>  ebe3634 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 8d5cdc9 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> f8ba3e9 
> 
> Diff: https://reviews.apache.org/r/50595/diff/
> 
> 
> Testing
> ---
> 
> Tested downloading of the jar file by registering with ambari server
> Tested Falcon/Atlas component names
> 
> 
> Thanks,
> 
> Venkat Ranganathan
> 
>

Re: Review Request 50595: AMBARI-17955: Enabled download of berkeley DB jar file to be provisioned during install

[Venkat Ranganathan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50595/
---

(Updated Aug. 1, 2016, 10:38 a.m.)


Review request for Ambari and Alejandro Fernandez.


Changes
---

Fixed review comments


Bugs: AMBARI-17955
https://issues.apache.org/jira/browse/AMBARI-17955


Repository: ambari


Description
---

Enabled the registration of optionally downloaded berkeley db jar file.   
Introduce Berekley DB as a jdbc-db (even though it is actually not.   Download 
je-5.0.73.jar if not already present and inform users on how to register

also, fixed Atlas integration changes with classname and packagename


Diffs (updated)
-

  ambari-server/src/main/python/ambari_server/dbConfiguration.py fc68a4c 
  ambari-server/src/main/python/ambari_server/dbConfiguration_linux.py d09bd77 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon.py
 dd1cda0 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py
 ebe3634 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
8d5cdc9 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
f8ba3e9 

Diff: https://reviews.apache.org/r/50595/diff/


Testing
---

Tested downloading of the jar file by registering with ambari server
Tested Falcon/Atlas component names


Thanks,

Venkat Ranganathan

Re: Review Request 50646: Add kerberos.json file for Sqoop in HDP 2.5 needed by Atlas

[Swapan Shridhar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50646/#review144353
---


Ship it!




Ship It!

- Swapan Shridhar


On Aug. 1, 2016, 12:09 a.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50646/
> ---
> 
> (Updated Aug. 1, 2016, 12:09 a.m.)
> 
> 
> Review request for Ambari, Di Li, Jonathan Hurley, Jayush Luniya, Nate Cole, 
> Robert Levas, Swapan Shridhar, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-17972
> https://issues.apache.org/jira/browse/AMBARI-17972
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-17782 made changes so that HDP 2.5 with Sqoop in a kerberized cluster 
> would set configs in sqoop-atlas-application.properties:
> atlas.jaas.KafkaClient.option.useTicketCache=true
> atlas.jaas.KafkaClient.option.renewTicket=true
> 
> However, the kerberos.json file that was added broke some tests and had to be 
> reverted because it didn't contain any identities or components. Now, readd 
> the file and add some unit tests so that a kerberos.json file without any 
> components is supported.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  16266cf 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SQOOP/kerberos.json 
> PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
>  6387fe8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/KerberosServiceDescriptorTest.java
>  033a0e2 
> 
> Diff: https://reviews.apache.org/r/50646/diff/
> 
> 
> Testing
> ---
> 
> Unit tests passed.
> 
> KerberosHelperTest.java
> KerberosServiceDescriptorTest.java
> 
> 
> Python unit tests,
> Total run:1034
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Re: Review Request 50663: Add snappy package dependency for AMS collector

[Aravindan Vijayan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50663/#review144351
---


Ship it!




Ship It!

- Aravindan Vijayan


On Aug. 1, 2016, 4:53 p.m., Dmytro Sen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50663/
> ---
> 
> (Updated Aug. 1, 2016, 4:53 p.m.)
> 
> 
> Review request for Ambari, Aravindan Vijayan, Sid Wagle, and Vitalyi 
> Brodetskyi.
> 
> 
> Bugs: AMBARI-17980
> https://issues.apache.org/jira/browse/AMBARI-17980
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add snappy package dependency for AMS collector
> 
> 
> Diffs
> -
> 
>   ambari-metrics/ambari-metrics-assembly/pom.xml 0a4f4e2 
> 
> Diff: https://reviews.apache.org/r/50663/diff/
> 
> 
> Testing
> ---
> 
> Manual testing
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Re: Review Request 50663: Add snappy package dependency for AMS collector

[Dmytro Sen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50663/
---

(Updated Авг. 1, 2016, 4:53 п.п.)


Review request for Ambari, Aravindan Vijayan, Sid Wagle, and Vitalyi Brodetskyi.


Bugs: AMBARI-17980
https://issues.apache.org/jira/browse/AMBARI-17980


Repository: ambari


Description
---

Add snappy package dependency for AMS collector


Diffs (updated)
-

  ambari-metrics/ambari-metrics-assembly/pom.xml 0a4f4e2 

Diff: https://reviews.apache.org/r/50663/diff/


Testing
---

Manual testing


Thanks,

Dmytro Sen

Review Request 50663: Add snappy package dependency for AMS collector

[Dmytro Sen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50663/
---

Review request for Ambari, Aravindan Vijayan, Sid Wagle, and Vitalyi Brodetskyi.


Bugs: AMBARI-17980
https://issues.apache.org/jira/browse/AMBARI-17980


Repository: ambari


Description
---

Add snappy package dependency for AMS collector


Diffs
-

  ambari-metrics/ambari-metrics-assembly/pom.xml 0a4f4e2 

Diff: https://reviews.apache.org/r/50663/diff/


Testing
---

Manual testing


Thanks,

Dmytro Sen

Re: Review Request 50616: Coverity Scan Security Vulnerability - SQL injection

[Sumit Mohanty]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50616/#review144346
---


Ship it!




Ship It!

- Sumit Mohanty


On July 29, 2016, 9:19 p.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50616/
> ---
> 
> (Updated July 29, 2016, 9:19 p.m.)
> 
> 
> Review request for Ambari, Mahadev Konar, Sumit Mohanty, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-17962
> https://issues.apache.org/jira/browse/AMBARI-17962
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The Ambari coverity scan found two "High impact security" issues, both SQL 
> Injections.  They are both the same coding issue, but one is in 
> OracleConnector.java, and one is in the analogous method in 
> PostgresConnector.java.
> 
> This is the key description:
> ```
> CID 167755 (#1 of 1): SQL injection (SQLI)9. sql_taint: Insecure 
> concatenation of a SQL statement. The value searchClause is tainted.
> 
> Perform one of the following to guard against SQL injection attacks.
> * Parameterize the SQL statement using ? positional characters. Bind the 
> tainted values to the ? positional parameters using one of the 
> PreparedStatement.set* methods.
> * Validate user-supplied values against predefined constant values. 
> Concatenate these constant values into the SQL statement.
> * Cast tainted values to safe types such as integers. Concatenate these type 
> safe values into the statement.
> 
> [More 
> Information|https://scan3.coverity.com/doc/en/cov_checker_ref.html#id_sql_generic]
> ```
> 
> This is the one in OracleConnector.java, lines 32 -55:
> 
> ```
> 32  @Override
>   8. taint_path_param: Parameter searchClause receives the tainted data.
> 33  protected PreparedStatement getQualifiedPS(Statements statement, String 
> searchClause, Workflows.WorkflowDBEntry.WorkflowFields field, boolean 
> sortAscending, int offset, int limit) throws IOException {
> 34if (db == null)
> 35  throw new IOException("db not initialized");
> 36
> 37String order = " ORDER BY " + field.toString() + " " + (sortAscending ? 
> SORT_ASC : SORT_DESC);
> 38
> 39String query = "select * \n" +
> 40"  from ( select " +
> 41//"/*+ FIRST_ROWS(n) */ \n" +
> 42"  a.*, ROWNUM rnum \n" +
> 43"  from ("
>   CID 167755 (#1 of 1): SQL injection (SQLI)9. sql_taint: Insecure 
> concatenation of a SQL statement. The value searchClause is tainted.
>   Perform one of the following to guard against SQL injection attacks.
> 
> Parameterize the SQL statement using ? positional characters. Bind the 
> tainted values to the ? positional parameters using one of the 
> PreparedStatement.set* methods.
> Validate user-supplied values against predefined constant values. 
> Concatenate these constant values into the SQL statement.
> Cast tainted values to safe types such as integers. Concatenate these 
> type safe values into the statement.
> 
> More Information
> 44+ statement.getStatementString() + searchClause + order +
> 45") a \n" +
> 46"  where ROWNUM <= " + (offset + limit) + ") \n" +
> 47"where rnum  >= " + offset;
> 48
> 49try {
>   10. sql_sink: Passing the tainted value query to the SQL API 
> java.sql.Connection.prepareStatement(java.lang.String) may allow an attacker 
> to inject SQL.
> 50  return db.prepareStatement(query);
> 51} catch (SQLException e) {
> 52  throw new IOException(e);
> 53}
> 54
> 55  }
> ```
> This is the one in PostgresConnector.java, lines 495-504:
> 
> ``` 
>8. taint_path_param: Parameter searchClause receives the tainted data.
> 495  protected PreparedStatement getQualifiedPS(Statements statement, String 
> searchClause) throws IOException {
> 496if (db == null)
> 497  throw new IOException("postgres db not initialized");
> 498try {
> 499  // LOG.debug("preparing " + statement.getStatementString() + 
> searchClause);
>CID 167743 (#1 of 1): SQL injection (SQLI)9. sql_taint: Insecure 
> concatenation of a SQL statement. The value searchClause is tainted. Passing 
> the tainted command to the SQL API 
> java.sql.Connection.prepareStatement(java.lang.String) may allow an attacker 
> to inject SQL.
>Perform one of the following to guard against SQL injection attacks.
> 
> Parameterize the SQL statement using ? positional characters. Bind the 
> tainted values to the ? positional parameters using one of the 
> PreparedStatement.set* methods.
> Validate user-supplied values against predefined constant values. 
> Concatenate these constant values into the SQL statement.
> Cast tainted values to safe types such as integers. Concatenate these 
> type safe values into 

Re: Review Request 50616: Coverity Scan Security Vulnerability - SQL injection

[Mahadev Konar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50616/#review144343
---


Ship it!




Ship It!

- Mahadev Konar


On July 29, 2016, 9:19 p.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50616/
> ---
> 
> (Updated July 29, 2016, 9:19 p.m.)
> 
> 
> Review request for Ambari, Mahadev Konar, Sumit Mohanty, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-17962
> https://issues.apache.org/jira/browse/AMBARI-17962
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The Ambari coverity scan found two "High impact security" issues, both SQL 
> Injections.  They are both the same coding issue, but one is in 
> OracleConnector.java, and one is in the analogous method in 
> PostgresConnector.java.
> 
> This is the key description:
> ```
> CID 167755 (#1 of 1): SQL injection (SQLI)9. sql_taint: Insecure 
> concatenation of a SQL statement. The value searchClause is tainted.
> 
> Perform one of the following to guard against SQL injection attacks.
> * Parameterize the SQL statement using ? positional characters. Bind the 
> tainted values to the ? positional parameters using one of the 
> PreparedStatement.set* methods.
> * Validate user-supplied values against predefined constant values. 
> Concatenate these constant values into the SQL statement.
> * Cast tainted values to safe types such as integers. Concatenate these type 
> safe values into the statement.
> 
> [More 
> Information|https://scan3.coverity.com/doc/en/cov_checker_ref.html#id_sql_generic]
> ```
> 
> This is the one in OracleConnector.java, lines 32 -55:
> 
> ```
> 32  @Override
>   8. taint_path_param: Parameter searchClause receives the tainted data.
> 33  protected PreparedStatement getQualifiedPS(Statements statement, String 
> searchClause, Workflows.WorkflowDBEntry.WorkflowFields field, boolean 
> sortAscending, int offset, int limit) throws IOException {
> 34if (db == null)
> 35  throw new IOException("db not initialized");
> 36
> 37String order = " ORDER BY " + field.toString() + " " + (sortAscending ? 
> SORT_ASC : SORT_DESC);
> 38
> 39String query = "select * \n" +
> 40"  from ( select " +
> 41//"/*+ FIRST_ROWS(n) */ \n" +
> 42"  a.*, ROWNUM rnum \n" +
> 43"  from ("
>   CID 167755 (#1 of 1): SQL injection (SQLI)9. sql_taint: Insecure 
> concatenation of a SQL statement. The value searchClause is tainted.
>   Perform one of the following to guard against SQL injection attacks.
> 
> Parameterize the SQL statement using ? positional characters. Bind the 
> tainted values to the ? positional parameters using one of the 
> PreparedStatement.set* methods.
> Validate user-supplied values against predefined constant values. 
> Concatenate these constant values into the SQL statement.
> Cast tainted values to safe types such as integers. Concatenate these 
> type safe values into the statement.
> 
> More Information
> 44+ statement.getStatementString() + searchClause + order +
> 45") a \n" +
> 46"  where ROWNUM <= " + (offset + limit) + ") \n" +
> 47"where rnum  >= " + offset;
> 48
> 49try {
>   10. sql_sink: Passing the tainted value query to the SQL API 
> java.sql.Connection.prepareStatement(java.lang.String) may allow an attacker 
> to inject SQL.
> 50  return db.prepareStatement(query);
> 51} catch (SQLException e) {
> 52  throw new IOException(e);
> 53}
> 54
> 55  }
> ```
> This is the one in PostgresConnector.java, lines 495-504:
> 
> ``` 
>8. taint_path_param: Parameter searchClause receives the tainted data.
> 495  protected PreparedStatement getQualifiedPS(Statements statement, String 
> searchClause) throws IOException {
> 496if (db == null)
> 497  throw new IOException("postgres db not initialized");
> 498try {
> 499  // LOG.debug("preparing " + statement.getStatementString() + 
> searchClause);
>CID 167743 (#1 of 1): SQL injection (SQLI)9. sql_taint: Insecure 
> concatenation of a SQL statement. The value searchClause is tainted. Passing 
> the tainted command to the SQL API 
> java.sql.Connection.prepareStatement(java.lang.String) may allow an attacker 
> to inject SQL.
>Perform one of the following to guard against SQL injection attacks.
> 
> Parameterize the SQL statement using ? positional characters. Bind the 
> tainted values to the ? positional parameters using one of the 
> PreparedStatement.set* methods.
> Validate user-supplied values against predefined constant values. 
> Concatenate these constant values into the SQL statement.
> Cast tainted values to safe types such as integers. Concatenate these 
> type safe values into 

Review Request 50660: YARN restart icon appeared after long time after deleting ZK server

[Vitalyi Brodetskyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50660/
---

Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, Dmytro Sen, and 
Jonathan Hurley.


Bugs: AMBARI-17979
https://issues.apache.org/jira/browse/AMBARI-17979


Repository: ambari


Description
---

STR:
->On a RM HA Cluster, add a host. 
->Add ZK Server on that host. Wait for Restart indicators to appear (Indicators 
appeared in test)
->Remove ZK Server from that host. Wait for Restart indicators to appear 
(*Indicators did not appear until ~15 minutes after*). The test has 3 retries 
to look for restart indicators with 40 seconds sleep in each.

Test logs with timestamps
{code}
2016-07-26 12:49:15,479 INFO 
com.hw.ambari.ui.opm.monitoring.pages.HostDetailsPage.selectAction(): Clicked 
Action from comboBox with xpath://ul[contains(@class,'dropdown-menu')]//li[not 
(@class='hidden')]//a[span[text() ='Delete']]
2016-07-26 12:49:15,755 INFO com.hw.ambari.ui.opm.WebDriverHelper.confirm(): 
Confirm Selected Operation..
2016-07-26 12:49:15,868 INFO 
com.hw.ambari.ui.opm.elements.DialogWindow.clickConfirm(): Confirm Button found
2016-07-26 12:49:17,985 INFO 
com.hw.ambari.ui.opm.elements.DialogWindow.clickConfirm(): Confirm Button found
2016-07-26 12:49:18,010 INFO 
com.hw.ambari.ui.opm.elements.DialogWindow.clickConfirm(): Clicked Confirm 
Button
2016-07-26 12:49:18,010 DEBUG 
com.hw.ambari.ui.opm.WebDriverHelper.sleepSeconds(): Starting sleeping for 30 
seconds...
2016-07-26 12:49:18,010 DEBUG 
com.hw.ambari.ui.opm.WebDriverHelper.sleepSeconds(): Caller: 
com.hw.ambari.ui.tests.monitoring.admin_page.TestEnableRmHa.deleteZookeeperServerFromExistingHost(TestEnableRmHa.java:620)
{code}


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
 f9ce8e6 
  
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 6e3eef7 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ConfigGroupResourceProvider.java
 3ba8e69 
  ambari-server/src/main/java/org/apache/ambari/server/state/ConfigHelper.java 
16edc74 
  ambari-server/src/main/java/org/apache/ambari/server/state/HostConfig.java 
0f451ce 
  
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
 53059dc 
  
ambari-server/src/main/java/org/apache/ambari/server/state/svccomphost/ServiceComponentHostImpl.java
 78f3751 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ConfigGroupResourceProviderTest.java
 2a4b949 
  
ambari-server/src/test/java/org/apache/ambari/server/state/ConfigHelperTest.java
 f3d35d9 
  
ambari-server/src/test/java/org/apache/ambari/server/state/svccomphost/ServiceComponentHostTest.java
 44afdaa 

Diff: https://reviews.apache.org/r/50660/diff/


Testing
---

mvn clean test


Thanks,

Vitalyi Brodetskyi

Re: Review Request 50656: Improvements/fixes for AMBARI_INFRA initial service definition

[Robert Nettleton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50656/#review144335
---


Ship it!




Ship It!

- Robert Nettleton


On Aug. 1, 2016, 12:22 p.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50656/
> ---
> 
> (Updated Aug. 1, 2016, 12:22 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jayush Luniya, Robert 
> Nettleton, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17974
> https://issues.apache.org/jira/browse/AMBARI-17974
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> contains a few fixes/imporvements for AMBARI_INFRA service definition:
> - role command order fix
> - implemented service check (test with kerberos too ... check only one of the 
> solrs has started or not)
> - quick links for infra Solrs (adimin UI)
> - configuration fixes (1 display name is missing + adding configuration 
> properties during ambari upgrade is not needed)
> - set the order of AMBARI_INFRA service on 'Select Services' page on the UI
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/configuration/infra-solr-client-log4j.xml
>  f597492 
>   
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/configuration/infra-solr-env.xml
>  dfbaac9 
>   
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/configuration/infra-solr-log4j.xml
>  b192a28 
>   
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/configuration/infra-solr-xml.xml
>  1d9d8ef 
>   
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/metainfo.xml
>  a1cee49 
>   
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/package/scripts/params.py
>  898a35c 
>   
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/package/scripts/service_check.py
>  a5a94fc 
>   
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/quicklinks/quicklinks.json
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/AMBARI_INFRA/role_command_order.json
>  34654c2 
>   ambari-web/app/models/stack_service.js 3e2d5b4 
> 
> Diff: https://reviews.apache.org/r/50656/diff/
> 
> 
> Testing
> ---
> 
> Total run:1034
> Total errors:0
> Total failures:0
> OK
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 1:14.391s
> [INFO] Finished at: Mon Aug 01 14:16:11 CEST 2016
> [INFO] Final Memory: 59M/985M
> [INFO] ---
> 
> 
> Thanks,
> 
> Oliver Szabo
> 
>

Re: Review Request 50297: AMBARI-17788: Refactor spooler code in OutputHDFSFile to be reusable for OutputS3File

[Oliver Szabo]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50297/#review144332
---


Ship it!




Ship It!

- Oliver Szabo


On Aug. 1, 2016, 1:10 p.m., Hemanth Yamijala wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50297/
> ---
> 
> (Updated Aug. 1, 2016, 1:10 p.m.)
> 
> 
> Review request for Ambari, Hayat Behlim, Dharmesh Makwana, Miklos Gergely, 
> and Oliver Szabo.
> 
> 
> Bugs: AMBARI-17788
> https://issues.apache.org/jira/browse/AMBARI-17788
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> This patch extracts spooling functions outside OutputHDFSFile so that it can 
> be used for AMBARI-17785. The patch allows callbacks to check the condition 
> on which spool files should be rolled over and what to do on rollover.
> 
> 
> Diffs
> -
> 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputHDFSFile.java
>  87cc0eb 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/spool/LogSpooler.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/spool/LogSpoolerContext.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/spool/LogSpoolerException.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/spool/RolloverCondition.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/spool/RolloverHandler.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/spool/LogSpoolerTest.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/50297/diff/
> 
> 
> Testing
> ---
> 
> * New unit tests.
> * Have tested by installing new RPM on an existing Ambari installation and 
> configuring the output for one of the services to write to HDFS. Tested that 
> upload to HDFS, rollover is working correctly for this service.
> 
> 
> Thanks,
> 
> Hemanth Yamijala
> 
>

Re: Review Request 50653: AMBARI-17308 Ambari Logfeeder outputs a lot of errors due to parse date

[Oliver Szabo]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50653/#review144326
---


Ship it!




Ship It!

- Oliver Szabo


On Aug. 1, 2016, 10:23 a.m., Masahiro Tanaka wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50653/
> ---
> 
> (Updated Aug. 1, 2016, 10:23 a.m.)
> 
> 
> Review request for Ambari, Daniel Gergely, Dmytro Sen, Oliver Szabo, and 
> Sebastian Toader.
> 
> 
> Bugs: AMBARI-17308
> https://issues.apache.org/jira/browse/AMBARI-17308
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> In logsearch_feeder service log, we got errors like below
> ```
> 2016-06-20 15:28:09,368 ERROR file=ambari-audit.log 
> org.apache.ambari.logfeeder.mapper.MapperDate LogFeederUtil.java:356 - Error 
> applying date transformation. isEpoch=false, 
> dateFormat=-MM-dd'T'HH:mm:ss.SSSZ, value=2016-06-20T15:28:08.000. 
> mapClass=map_date, input=input:source=file, 
> path=/var/log/ambari-server/ambari-audit.log, fieldName=logtime. Messages 
> suppressed before: 2
> java.text.ParseException: Unparseable date: "2016-06-20T15:28:08.000"
>   at java.text.DateFormat.parse(DateFormat.java:366)
>   at 
> org.apache.ambari.logfeeder.mapper.MapperDate.apply(MapperDate.java:83)
>   at org.apache.ambari.logfeeder.filter.Filter.apply(Filter.java:154)
>   at 
> org.apache.ambari.logfeeder.filter.FilterGrok.applyMessage(FilterGrok.java:291)
>   at 
> org.apache.ambari.logfeeder.filter.FilterGrok.flush(FilterGrok.java:320)
>   at org.apache.ambari.logfeeder.input.Input.flush(Input.java:125)
>   at 
> org.apache.ambari.logfeeder.input.InputFile.processFile(InputFile.java:430)
>   at org.apache.ambari.logfeeder.input.InputFile.start(InputFile.java:260)
>   at org.apache.ambari.logfeeder.input.Input.run(Input.java:100)
>   at java.lang.Thread.run(Thread.java:745)
> ```
> ambari-audit.log is like below
> ```
> 2016-07-21T01:52:49.875+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
> 2016-07-21T01:52:49.905+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu16/repositories/HDP-2.5),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu16), Repo 
> id(HDP-2.5), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu16/2.x/BUILDS/2.5.0.0-1025)
> 2016-07-21T01:52:50.015+09, User(admin), RemoteIp(192.168.72.1), 
> Operation(Repository update), RequestType(PUT), 
> url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-UTILS-1.1.0.21),
>  ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
> id(HDP-UTILS-1.1.0.21), Base 
> URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/ubuntu14)
> ```
> I think date format of the ambari-audit.log (2016-07-21T01:52:49.875+09) 
> should be like 2016-07-21T01:52:49.875+0900, since grok-pattern can't handle 
> 2016-07-21T01:52:49.875+09 format.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerDefaultImpl.java
>  ea6524f 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/input.config-ambari.json.j2
>  5f751bf 
> 
> Diff: https://reviews.apache.org/r/50653/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Masahiro Tanaka
> 
>

Review Request 50656: Improvements/fixes for AMBARI_INFRA initial service definition

[Oliver Szabo]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50656/
---

Review request for Ambari, Alejandro Fernandez, Jayush Luniya, Robert 
Nettleton, and Sumit Mohanty.


Bugs: AMBARI-17974
https://issues.apache.org/jira/browse/AMBARI-17974


Repository: ambari


Description
---

contains a few fixes/imporvements for AMBARI_INFRA service definition:
- role command order fix
- implemented service check (test with kerberos too ... check only one of the 
solrs has started or not)
- quick links for infra Solrs (adimin UI)
- configuration fixes (1 display name is missing + adding configuration 
properties during ambari upgrade is not needed)
- set the order of AMBARI_INFRA service on 'Select Services' page on the UI


Diffs
-

  
ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/configuration/infra-solr-client-log4j.xml
 f597492 
  
ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/configuration/infra-solr-env.xml
 dfbaac9 
  
ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/configuration/infra-solr-log4j.xml
 b192a28 
  
ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/configuration/infra-solr-xml.xml
 1d9d8ef 
  
ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/metainfo.xml
 a1cee49 
  
ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/package/scripts/params.py
 898a35c 
  
ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/package/scripts/service_check.py
 a5a94fc 
  
ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/quicklinks/quicklinks.json
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/AMBARI_INFRA/role_command_order.json
 34654c2 
  ambari-web/app/models/stack_service.js 3e2d5b4 

Diff: https://reviews.apache.org/r/50656/diff/


Testing
---

Total run:1034
Total errors:0
Total failures:0
OK
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 1:14.391s
[INFO] Finished at: Mon Aug 01 14:16:11 CEST 2016
[INFO] Final Memory: 59M/985M
[INFO] ---


Thanks,

Oliver Szabo

Re: Review Request 50647: Changed oozie.authentication.kerberos.principal and oozie.authentication.kerberos.keytab are reverted while regenerating keytab files

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50647/#review144325
---


Ship it!




Ship It!

- Jonathan Hurley


On July 31, 2016, 10 p.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50647/
> ---
> 
> (Updated July 31, 2016, 10 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jonathan 
> Hurley, Jayush Luniya, Sumit Mohanty, Vitalyi Brodetskyi, and Venkat 
> Ranganathan.
> 
> 
> Bugs: AMBARI-17968
> https://issues.apache.org/jira/browse/AMBARI-17968
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Changed `oozie-site/oozie.authentication.kerberos.principal` and 
> `oozie-site/oozie.authentication.kerberos.keytab` properties are reverted 
> while regenerating keytab files.
> 
> The changed properties are needed to support Oozie in high availability (HA) 
> mode with failover enabled via a load balancing proxy server.
> 
> # Cause
> The relevant part of the Kerberos descriptor for Oozie is:
> ```
> {
>   "name": "/spnego",
>   "principal": {
> "configuration": 
> "oozie-site/oozie.authentication.kerberos.principal"
>   },
>   "keytab": {
> "configuration": 
> "oozie-site/oozie.authentication.kerberos.keytab"
>   }
> }
> ```
> 
> Because of this, certain Kerberos-related operations (like Regenerate 
> Keytabs) resets the values of 
> `oozie-site/oozie.authentication.kerberos.principal` and 
> `oozie-site/oozie.authentication.kerberos.keytab` to match the principal name 
> and keytab file of the Kerberos identity definition for `/spnego`.
> 
> However, in HA, the properties need to be something like:
> ```
> oozie.authentication.kerberos.prinipal= "*"
> oozie.authentication.kerberos.keytab = "/path/to/oozie_ha.keytab"
> ```
> 
> # Solution
> After enabling HA and either before or after enabling Kerberos, the following 
> `oozie-site` properties may be set:
> * `oozie.ha.authentication.kerberos.principal`
> * `oozie.ha.authentication.kerberos.keytab`
> 
> If either exist when configuring Oozie, the value of the property will be 
> used to update the relevant `oozie.authentication.kerberos.*` property.
> 
> For example:
> * if `oozie.ha.authentication.kerberos.principal` is set, its value will set 
> used to set `oozie.authentication.kerberos.principal`
> * if `oozie.ha.authentication.kerberos.keytab` is set, its value will set 
> used to set `oozie.authentication.kerberos.keytab`
> Note: One or both may be set. 
> 
> So even though `oozie.authentication.kerberos.principal` will contain a 
> principal name like `HTTP/_HOST@SOME.REALM`, when writing the oozie-site.xml 
> file, the value for `oozie.authentication.kerberos.principal` will be written 
> out as the value set for `oozie.ha.authentication.kerberos.principal`, which 
> would typically be "*", when HA is enabled for Oozie.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
>  23c9edc 
>   
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py
>  00229d3 
>   ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 
> 99d6dec 
> 
> Diff: https://reviews.apache.org/r/50647/diff/
> 
> 
> Testing
> ---
> 
> Manually tested
> 
> # Local test results: 
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 1:19.684s
> [INFO] Finished at: Sun Jul 31 21:36:24 EDT 2016
> [INFO] Final Memory: 71M/1705M
> [INFO] 
> 
> 
> # Jenkins test results: PENDING
> 
> 
> Thanks,
> 
> Robert Levas
> 
>

Re: Review Request 50646: Add kerberos.json file for Sqoop in HDP 2.5 needed by Atlas

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50646/#review144323
---


Ship it!




Ship It!

- Jonathan Hurley


On July 31, 2016, 8:09 p.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50646/
> ---
> 
> (Updated July 31, 2016, 8:09 p.m.)
> 
> 
> Review request for Ambari, Di Li, Jonathan Hurley, Jayush Luniya, Nate Cole, 
> Robert Levas, Swapan Shridhar, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-17972
> https://issues.apache.org/jira/browse/AMBARI-17972
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-17782 made changes so that HDP 2.5 with Sqoop in a kerberized cluster 
> would set configs in sqoop-atlas-application.properties:
> atlas.jaas.KafkaClient.option.useTicketCache=true
> atlas.jaas.KafkaClient.option.renewTicket=true
> 
> However, the kerberos.json file that was added broke some tests and had to be 
> reverted because it didn't contain any identities or components. Now, readd 
> the file and add some unit tests so that a kerberos.json file without any 
> components is supported.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  16266cf 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SQOOP/kerberos.json 
> PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
>  6387fe8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/KerberosServiceDescriptorTest.java
>  033a0e2 
> 
> Diff: https://reviews.apache.org/r/50646/diff/
> 
> 
> Testing
> ---
> 
> Unit tests passed.
> 
> KerberosHelperTest.java
> KerberosServiceDescriptorTest.java
> 
> 
> Python unit tests,
> Total run:1034
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Re: Review Request 50654: Coverity Scan Security Vulnerability - Resource Leak defects

[Andrew Onischuk]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50654/#review144321
---


Ship it!




Ship It!

- Andrew Onischuk


On Aug. 1, 2016, 11:05 a.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50654/
> ---
> 
> (Updated Aug. 1, 2016, 11:05 a.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, and Dmytro 
> Sen.
> 
> 
> Bugs: AMBARI-17922
> https://issues.apache.org/jira/browse/AMBARI-17922
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix "Resource Leak" issues that were found by coverity scan.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/java/org/apache/ambari/server/view/ViewRegistry.java 
> f49f604 
> 
> Diff: https://reviews.apache.org/r/50654/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Re: Review Request 50651: Zeppelin start fails in non-root

[Vitalyi Brodetskyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50651/#review144320
---


Ship it!




Ship It!

- Vitalyi Brodetskyi


On Сер. 1, 2016, 7:50 до полудня, Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50651/
> ---
> 
> (Updated Сер. 1, 2016, 7:50 до полудня)
> 
> 
> Review request for Ambari and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-17973
> https://issues.apache.org/jira/browse/AMBARI-17973
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Traceback (most recent call last):
>   File 
> "/var/lib/ambari-agent/cache/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/master.py",
>  line 293, in 
> Master().execute()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 280, in execute
> method(env)
>   File 
> "/var/lib/ambari-agent/cache/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/master.py",
>  line 167, in start
> self.update_zeppelin_interpreter()
>   File 
> "/var/lib/ambari-agent/cache/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/master.py",
>  line 254, in update_zeppelin_interpreter
> config_data = self.get_interpreter_settings()
>   File 
> "/var/lib/ambari-agent/cache/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/master.py",
>  line 195, in get_interpreter_settings
> interpreter_config_file = open(interpreter_config, "r")
> IOError: [Errno 13] Permission denied: 
> '/etc/zeppelin/conf/interpreter.json'
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/master.py
>  531e613 
> 
> Diff: https://reviews.apache.org/r/50651/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>

Review Request 50654: Coverity Scan Security Vulnerability - Resource Leak defects

[Vitalyi Brodetskyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50654/
---

Review request for Ambari, Dmitro Lisnichenko and Dmytro Sen.


Bugs: AMBARI-17922
https://issues.apache.org/jira/browse/AMBARI-17922


Repository: ambari


Description
---

Fix "Resource Leak" issues that were found by coverity scan.


Diffs
-

  ambari-server/src/main/java/org/apache/ambari/server/view/ViewRegistry.java 
f49f604 

Diff: https://reviews.apache.org/r/50654/diff/


Testing
---

mvn clean test


Thanks,

Vitalyi Brodetskyi

Review Request 50653: AMBARI-17308 Ambari Logfeeder outputs a lot of errors due to parse date

[Masahiro Tanaka]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50653/
---

Review request for Ambari, Daniel Gergely, Dmytro Sen, Oliver Szabo, and 
Sebastian Toader.


Bugs: AMBARI-17308
https://issues.apache.org/jira/browse/AMBARI-17308


Repository: ambari


Description
---

In logsearch_feeder service log, we got errors like below
```
2016-06-20 15:28:09,368 ERROR file=ambari-audit.log 
org.apache.ambari.logfeeder.mapper.MapperDate LogFeederUtil.java:356 - Error 
applying date transformation. isEpoch=false, 
dateFormat=-MM-dd'T'HH:mm:ss.SSSZ, value=2016-06-20T15:28:08.000. 
mapClass=map_date, input=input:source=file, 
path=/var/log/ambari-server/ambari-audit.log, fieldName=logtime. Messages 
suppressed before: 2
java.text.ParseException: Unparseable date: "2016-06-20T15:28:08.000"
at java.text.DateFormat.parse(DateFormat.java:366)
at 
org.apache.ambari.logfeeder.mapper.MapperDate.apply(MapperDate.java:83)
at org.apache.ambari.logfeeder.filter.Filter.apply(Filter.java:154)
at 
org.apache.ambari.logfeeder.filter.FilterGrok.applyMessage(FilterGrok.java:291)
at 
org.apache.ambari.logfeeder.filter.FilterGrok.flush(FilterGrok.java:320)
at org.apache.ambari.logfeeder.input.Input.flush(Input.java:125)
at 
org.apache.ambari.logfeeder.input.InputFile.processFile(InputFile.java:430)
at org.apache.ambari.logfeeder.input.InputFile.start(InputFile.java:260)
at org.apache.ambari.logfeeder.input.Input.run(Input.java:100)
at java.lang.Thread.run(Thread.java:745)
```
ambari-audit.log is like below
```
2016-07-21T01:52:49.875+09, User(admin), RemoteIp(192.168.72.1), 
Operation(Repository update), RequestType(PUT), 
url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-2.5),
 ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
id(HDP-2.5), Base 
URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu14/2.x/BUILDS/2.5.0.0-1025)
2016-07-21T01:52:49.905+09, User(admin), RemoteIp(192.168.72.1), 
Operation(Repository update), RequestType(PUT), 
url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu16/repositories/HDP-2.5),
 ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu16), Repo 
id(HDP-2.5), Base 
URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP/ubuntu16/2.x/BUILDS/2.5.0.0-1025)
2016-07-21T01:52:50.015+09, User(admin), RemoteIp(192.168.72.1), 
Operation(Repository update), RequestType(PUT), 
url(http://192.168.72.101:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/ubuntu14/repositories/HDP-UTILS-1.1.0.21),
 ResultStatus(200 OK), Stack(HDP), Stack version(2.5), OS(ubuntu14), Repo 
id(HDP-UTILS-1.1.0.21), Base 
URL(http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/ubuntu14)
```
I think date format of the ambari-audit.log (2016-07-21T01:52:49.875+09) should 
be like 2016-07-21T01:52:49.875+0900, since grok-pattern can't handle 
2016-07-21T01:52:49.875+09 format.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/audit/AuditLoggerDefaultImpl.java
 ea6524f 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/input.config-ambari.json.j2
 5f751bf 

Diff: https://reviews.apache.org/r/50653/diff/


Testing
---

mvn clean test


Thanks,

Masahiro Tanaka

Re: Review Request 50646: Add kerberos.json file for Sqoop in HDP 2.5 needed by Atlas

[Robert Levas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50646/#review144313
---


Ship it!




Ship It!

- Robert Levas


On July 31, 2016, 8:09 p.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50646/
> ---
> 
> (Updated July 31, 2016, 8:09 p.m.)
> 
> 
> Review request for Ambari, Di Li, Jonathan Hurley, Jayush Luniya, Nate Cole, 
> Robert Levas, Swapan Shridhar, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-17972
> https://issues.apache.org/jira/browse/AMBARI-17972
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-17782 made changes so that HDP 2.5 with Sqoop in a kerberized cluster 
> would set configs in sqoop-atlas-application.properties:
> atlas.jaas.KafkaClient.option.useTicketCache=true
> atlas.jaas.KafkaClient.option.renewTicket=true
> 
> However, the kerberos.json file that was added broke some tests and had to be 
> reverted because it didn't contain any identities or components. Now, readd 
> the file and add some unit tests so that a kerberos.json file without any 
> components is supported.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  16266cf 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SQOOP/kerberos.json 
> PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
>  6387fe8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/KerberosServiceDescriptorTest.java
>  033a0e2 
> 
> Diff: https://reviews.apache.org/r/50646/diff/
> 
> 
> Testing
> ---
> 
> Unit tests passed.
> 
> KerberosHelperTest.java
> KerberosServiceDescriptorTest.java
> 
> 
> Python unit tests,
> Total run:1034
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Review Request 50651: Zeppelin start fails in non-root

[Andrew Onischuk]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50651/
---

Review request for Ambari and Vitalyi Brodetskyi.


Bugs: AMBARI-17973
https://issues.apache.org/jira/browse/AMBARI-17973


Repository: ambari


Description
---

Traceback (most recent call last):
  File 
"/var/lib/ambari-agent/cache/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/master.py",
 line 293, in 
Master().execute()
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
 line 280, in execute
method(env)
  File 
"/var/lib/ambari-agent/cache/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/master.py",
 line 167, in start
self.update_zeppelin_interpreter()
  File 
"/var/lib/ambari-agent/cache/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/master.py",
 line 254, in update_zeppelin_interpreter
config_data = self.get_interpreter_settings()
  File 
"/var/lib/ambari-agent/cache/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/master.py",
 line 195, in get_interpreter_settings
interpreter_config_file = open(interpreter_config, "r")
IOError: [Errno 13] Permission denied: '/etc/zeppelin/conf/interpreter.json'


Diffs
-

  
ambari-server/src/main/resources/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/master.py
 531e613 

Diff: https://reviews.apache.org/r/50651/diff/


Testing
---

mvn clean test


Thanks,

Andrew Onischuk