Re: Review Request 50636: Ambari Sends Cached Configurations On Initial Task Execution Attempt

2016-07-31 Thread Robert Levas 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50636/#review144280
---


Ship it!




Ship It!

- Robert Levas


On July 30, 2016, 9:04 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50636/
> ---
> 
> (Updated July 30, 2016, 9:04 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, and 
> Sid Wagle.
> 
> 
> Bugs: AMBARI-17970
> https://issues.apache.org/jira/browse/AMBARI-17970
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> This was an EclipseLink cache problem discovered on downgrade testing. When 
> downgrading Ranger Admin from HDP 2.5 to HDP 2.4, the python code correctly 
> detects HDP 2.4 is the version it should use for "stack feature" checking. On 
> HDP 2.4, Ranger used the audit DB and therefore looks at the configs for 
> audit DB related data.
> 
> However, the data was not there. The problem is complex, yet subtle: 
> - The client thread creating the downgrade changes the desired configurations 
> of the cluster (that's good!)
> - The action scheduler picks up the first task to run during downgrade and 
> sees that configs need to be refresh (that's good!)
> - The ClusterEntity that the action scheduler sees from its entity manager 
> doesn't have the updates applied by the client thread (that's bad!)
> 
> It takes EclipseLink up to 1000ms to invalidate the L1 persistence context 
> cache of the action scheduler's EntityManager. As a result, there's a 
> fraction of a second where the scheduler could see the first downgrade task 
> and try to schedule it using stale configurations. In this case, the 
> configurations from HDP 2.5.
> 
> The solution I chose was to use a synchronous event bus to instruct 
> interested DAOs to invalidate particular entities. The entity manager bound 
> to the action scheduler thread was the problem, but since that thread uses so 
> many DAOs indirectly, it was easier to broadcast an event to interested DAOs, 
> keeping the scheduler thread out of it.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ExecutionCommandWrapper.java
>  2f6ab02 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/jpa/EntityManagerCacheInvalidationEvent.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/jpa/JPAEvent.java 
> PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/publishers/JPAEventPublisher.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterDAO.java 
> 7f499f0 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
>  53059dc 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/cluster/ClusterTest.java
>  7c0e17c 
> 
> Diff: https://reviews.apache.org/r/50636/diff/
> 
> 
> Testing
> ---
> 
> PENDING UNIT TESTS
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Review Request 50645: Ambari upgrade to 2.4.0 adds several config properties that are not necessary and some are incorrect

2016-07-31 Thread Sumit Mohanty 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50645/
---

Review request for Ambari, Alejandro Fernandez, Dmitro Lisnichenko, Jaimin 
Jetly, and Mugdha Varadkar.


Bugs: AMBARI-17971
https://issues.apache.org/jira/browse/AMBARI-17971


Repository: ambari


Description
---

Ambari upgrade to 2.4.0 adds several config properties that are not necessary 
and some are incorrect. (interim patch)


Diffs
-

  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/application-properties.xml
 108c7b4 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/atlas-log4j.xml
 a071442 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/atlas-solrconfig.xml
 35e9f58 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
 1a0e797 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
 8006158 
  
ambari-server/src/main/resources/common-services/ATLAS/0.7.0.2.5/configuration/atlas-solrconfig.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/configuration/falcon-atlas-application.properties.xml
 e653651 
  
ambari-server/src/main/resources/common-services/KAFKA/0.9.0/configuration/kafka-broker.xml
 0275358 
  
ambari-server/src/main/resources/common-services/KAFKA/0.9.0/configuration/ranger-kafka-audit.xml
 5257549 
  
ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/configuration/ranger-kms-audit.xml
 50a997c 
  
ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/configuration/sqoop-site.xml
 f7823d2 
  
ambari-server/src/main/resources/common-services/STORM/0.10.0/configuration/ranger-storm-audit.xml
 b7cf4c5 
  
ambari-server/src/main/resources/common-services/STORM/0.9.1/configuration/storm-site.xml
 ab7d9cb 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/hbase-env.xml
 7b55225 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/hbase-site.xml
 3f860e0 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/configuration/ranger-hbase-audit.xml
 f670d7e 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/configuration/ranger-hdfs-audit.xml
 63154eb 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/configuration/ranger-hive-audit.xml
 874d0d5 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/KNOX/configuration/ranger-knox-audit.xml
 abdf2bd 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/configuration/ranger-yarn-audit.xml
 8237f1c 
  
ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/configuration/yarn-site.xml
 5daa9fd 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/ranger-atlas-audit.xml
 efeea5f 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/FALCON/configuration/falcon-atlas-application.properties.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/RANGER_KMS/configuration/ranger-kms-audit.xml
 86b963b 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/yarn-site.xml
 ea7a7af 

Diff: https://reviews.apache.org/r/50645/diff/


Testing
---

Manually run. Waiting for further tests.


Thanks,

Sumit Mohanty

Re: Review Request 50645: Ambari upgrade to 2.4.0 adds several config properties that are not necessary and some are incorrect

2016-07-31 Thread Sumit Mohanty 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50645/#review144287
---




ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
 (line 166)


This config type is unavailable in older version of ATLAS.



ambari-server/src/main/resources/common-services/KAFKA/0.9.0/configuration/kafka-broker.xml
 (line 168)


Seems it should have been marked as deleted. 
https://github.com/apache/ambari/blob/branch-2.2.2/ambari-server/src/main/resources/stacks/HDP/2.3/services/KAFKA/configuration/kafka-broker.xml#L122



ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/configuration/yarn-site.xml
 


Support for this property was added only starting HDP-2.5



ambari-server/src/main/resources/stacks/HDP/2.5/services/FALCON/configuration/falcon-atlas-application.properties.xml
 (line 1)


This should be added starting HDP-2.5


- Sumit Mohanty


On July 31, 2016, 9:26 p.m., Sumit Mohanty wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50645/
> ---
> 
> (Updated July 31, 2016, 9:26 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmitro Lisnichenko, Jaimin 
> Jetly, and Mugdha Varadkar.
> 
> 
> Bugs: AMBARI-17971
> https://issues.apache.org/jira/browse/AMBARI-17971
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Ambari upgrade to 2.4.0 adds several config properties that are not necessary 
> and some are incorrect. (interim patch)
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/application-properties.xml
>  108c7b4 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/atlas-log4j.xml
>  a071442 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/atlas-solrconfig.xml
>  35e9f58 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
>  1a0e797 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
>  8006158 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.7.0.2.5/configuration/atlas-solrconfig.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/configuration/falcon-atlas-application.properties.xml
>  e653651 
>   
> ambari-server/src/main/resources/common-services/KAFKA/0.9.0/configuration/kafka-broker.xml
>  0275358 
>   
> ambari-server/src/main/resources/common-services/KAFKA/0.9.0/configuration/ranger-kafka-audit.xml
>  5257549 
>   
> ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/configuration/ranger-kms-audit.xml
>  50a997c 
>   
> ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/configuration/sqoop-site.xml
>  f7823d2 
>   
> ambari-server/src/main/resources/common-services/STORM/0.10.0/configuration/ranger-storm-audit.xml
>  b7cf4c5 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/configuration/storm-site.xml
>  ab7d9cb 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/hbase-env.xml
>  7b55225 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/hbase-site.xml
>  3f860e0 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/configuration/ranger-hbase-audit.xml
>  f670d7e 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/configuration/ranger-hdfs-audit.xml
>  63154eb 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/configuration/ranger-hive-audit.xml
>  874d0d5 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/KNOX/configuration/ranger-knox-audit.xml
>  abdf2bd 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/configuration/ranger-yarn-audit.xml
>  8237f1c 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/configuration/yarn-site.xml
>  5daa9fd 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/ranger-atlas-audit.xml
>  efeea5f 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/FALCON/configuration/falcon-atlas-application.properties.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/RANGER_KMS/configuration/ranger-kms-audit.xml
>  86b963b 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/yarn-site.xml
>  ea7a7af 
> 
> Diff: https://reviews.apache.org/r/50645/diff/
> 
> 
> Tes

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

2016-07-31 Thread Sumit Mohanty 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144288
---




ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
 (line 128)


I think the previous code was safer - spark_javaopts_properties = 
default("/configurations/spark-javaopts-properties/content", " ")

Any reason to switch to if ... else ...



ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
 (line 133)


Use delete option with File(). The helper method do the necessary work for 
non-root deployments.

e.g.
File(params.infra_solr_pidfile,
   action="delete"
   )



ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
 (line 25)


Remove the line 



ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
 (line 30)


Remove the line 



ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
 (line 24)


Remove the line 



ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
 (line 30)


Remove the line 


- Sumit Mohanty


On July 31, 2016, 3:44 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated July 31, 2016, 3:44 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50431: AMBARI-17896: Fix falcon hook classname and classpath for atlas

2016-07-31 Thread Alejandro Fernandez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50431/#review144290
---



Please close this review

- Alejandro Fernandez


On July 27, 2016, 1:07 a.m., Venkat Ranganathan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50431/
> ---
> 
> (Updated July 27, 2016, 1:07 a.m.)
> 
> 
> Review request for Ambari and Alejandro Fernandez.
> 
> 
> Bugs: AMBARI-17896
> https://issues.apache.org/jira/browse/AMBARI-17896
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fixed stack advisor for 2.5 and also the param_linux for setting the right 
> conf path
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py
>  ebe3634 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 8d5cdc9 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> f8ba3e9 
> 
> Diff: https://reviews.apache.org/r/50431/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Venkat Ranganathan
> 
>

Re: Review Request 50595: AMBARI-17955: Enabled download of berkeley DB jar file to be provisioned during install

2016-07-31 Thread Alejandro Fernandez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50595/#review144291
---




ambari-server/src/main/python/ambari_server/dbConfiguration_linux.py (line 1131)


We should remove the version from the jar name



ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py (line 
742)


it's just hasattr(self, "atlasFalconHookClassName"), no need to prepend 
"self."


- Alejandro Fernandez


On July 29, 2016, 8:14 p.m., Venkat Ranganathan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50595/
> ---
> 
> (Updated July 29, 2016, 8:14 p.m.)
> 
> 
> Review request for Ambari and Alejandro Fernandez.
> 
> 
> Bugs: AMBARI-17955
> https://issues.apache.org/jira/browse/AMBARI-17955
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Enabled the registration of optionally downloaded berkeley db jar file.   
> Introduce Berekley DB as a jdbc-db (even though it is actually not.   
> Download je-5.0.73.jar if not already present and inform users on how to 
> register
> 
> also, fixed Atlas integration changes with classname and packagename
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/python/ambari_server/dbConfiguration.py fc68a4c 
>   ambari-server/src/main/python/ambari_server/dbConfiguration_linux.py 
> d09bd77 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon.py
>  dd1cda0 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py
>  ebe3634 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 8d5cdc9 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> f8ba3e9 
> 
> Diff: https://reviews.apache.org/r/50595/diff/
> 
> 
> Testing
> ---
> 
> Tested downloading of the jar file by registering with ambari server
> Tested Falcon/Atlas component names
> 
> 
> Thanks,
> 
> Venkat Ranganathan
> 
>

Review Request 50646: Add kerberos.json file for Sqoop in HDP 2.5 needed by Atlas

2016-07-31 Thread Alejandro Fernandez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50646/
---

Review request for Ambari, Di Li, Jonathan Hurley, Jayush Luniya, Nate Cole, 
Robert Levas, Swapan Shridhar, and Tim Thorpe.


Bugs: AMBARI-17972
https://issues.apache.org/jira/browse/AMBARI-17972


Repository: ambari


Description
---

AMBARI-17782 made changes so that HDP 2.5 with Sqoop in a kerberized cluster 
would set configs in sqoop-atlas-application.properties:
atlas.jaas.KafkaClient.option.useTicketCache=true
atlas.jaas.KafkaClient.option.renewTicket=true

However, the kerberos.json file that was added broke some tests and had to be 
reverted because it didn't contain any identities or components. Now, readd the 
file and add some unit tests so that a kerberos.json file without any 
components is supported.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
 16266cf 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/SQOOP/kerberos.json 
PRE-CREATION 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
 6387fe8 
  
ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/KerberosServiceDescriptorTest.java
 033a0e2 

Diff: https://reviews.apache.org/r/50646/diff/


Testing
---

Unit tests passed.

KerberosHelperTest.java
KerberosServiceDescriptorTest.java


Python unit tests,
Total run:1034
Total errors:0
Total failures:0
OK


Thanks,

Alejandro Fernandez

Re: Review Request 50646: Add kerberos.json file for Sqoop in HDP 2.5 needed by Atlas

2016-07-31 Thread Alejandro Fernandez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50646/#review144292
---




ambari-server/src/main/resources/stacks/HDP/2.5/services/SQOOP/kerberos.json 
(line 13)


Technically, a component is no longer necessary. But added it just to be 
safe.



ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
 (line 2574)


Verify that a kerberos descriptor can exist without a "component" section.


- Alejandro Fernandez


On Aug. 1, 2016, 12:09 a.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50646/
> ---
> 
> (Updated Aug. 1, 2016, 12:09 a.m.)
> 
> 
> Review request for Ambari, Di Li, Jonathan Hurley, Jayush Luniya, Nate Cole, 
> Robert Levas, Swapan Shridhar, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-17972
> https://issues.apache.org/jira/browse/AMBARI-17972
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-17782 made changes so that HDP 2.5 with Sqoop in a kerberized cluster 
> would set configs in sqoop-atlas-application.properties:
> atlas.jaas.KafkaClient.option.useTicketCache=true
> atlas.jaas.KafkaClient.option.renewTicket=true
> 
> However, the kerberos.json file that was added broke some tests and had to be 
> reverted because it didn't contain any identities or components. Now, readd 
> the file and add some unit tests so that a kerberos.json file without any 
> components is supported.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  16266cf 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SQOOP/kerberos.json 
> PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
>  6387fe8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/KerberosServiceDescriptorTest.java
>  033a0e2 
> 
> Diff: https://reviews.apache.org/r/50646/diff/
> 
> 
> Testing
> ---
> 
> Unit tests passed.
> 
> KerberosHelperTest.java
> KerberosServiceDescriptorTest.java
> 
> 
> Python unit tests,
> Total run:1034
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

2016-07-31 Thread Saisai Shao 


> On July 31, 2016, 10:02 p.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py,
> >  line 129
> > 
> >
> > I think the previous code was safer - spark_javaopts_properties = 
> > default("/configurations/spark-javaopts-properties/content", " ")
> > 
> > Any reason to switch to if ... else ...

I see, I will revert it back.


- Saisai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144288
---


On July 31, 2016, 3:44 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated July 31, 2016, 3:44 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

2016-07-31 Thread Saisai Shao 


> On July 31, 2016, 10:02 p.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml,
> >  line 30
> > 
> >
> > Remove the line 

Done


> On July 31, 2016, 10:02 p.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml,
> >  line 24
> > 
> >
> > Remove the line 

Done


> On July 31, 2016, 10:02 p.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml,
> >  line 30
> > 
> >
> > Remove the line 

Done


> On July 31, 2016, 10:02 p.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml,
> >  line 25
> > 
> >
> > Remove the line 

Done


> On July 31, 2016, 10:02 p.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py,
> >  line 140
> > 
> >
> > Use delete option with File(). The helper method do the necessary work 
> > for non-root deployments.
> > 
> > e.g.
> > File(params.infra_solr_pidfile,
> >action="delete"
> >)

Done


- Saisai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144288
---


On July 31, 2016, 3:44 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated July 31, 2016, 3:44 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

2016-07-31 Thread Saisai Shao 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/
---

(Updated Aug. 1, 2016, 1:22 a.m.)


Review request for Ambari, Jayush Luniya and Sumit Mohanty.


Changes
---

Fix Spark hdp.version issues in upgrading and fresh install


Bugs: AMBARI-17954
https://issues.apache.org/jira/browse/AMBARI-17954


Repository: ambari


Description
---

Fix Spark hdp.version issues in upgrading and fresh install


Diffs (updated)
-

  
ambari-common/src/main/python/resource_management/libraries/functions/constants.py
 42ecf6a 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
 5495655 
  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
 646239e 
  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
 86aeb70 
  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
 2b5d36b 
  
ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
 e27c986 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
 6d26052 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
b295cc9 
  
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
 5b20a72 
  
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
 5d2cc54 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
c9f9823 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
d0332ed 
  
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
 b3d05d8 
  
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
 34ca199 
  ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
b2ed5df 
  ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
  ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
a1abdfa 

Diff: https://reviews.apache.org/r/50594/diff/


Testing
---

Manual test with different scenarios:

1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
2. Upgrade for 2.3.6 to 2.5.0.
3. Downgrade from 2.5.0 to 2.3.6.


Thanks,

Saisai Shao

Review Request 50647: Changed oozie.authentication.kerberos.principal and oozie.authentication.kerberos.keytab are reverted while regenerating keytab files

2016-07-31 Thread Robert Levas 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50647/
---

Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jonathan 
Hurley, Jayush Luniya, Sumit Mohanty, Vitalyi Brodetskyi, and Venkat 
Ranganathan.


Bugs: AMBARI-17968
https://issues.apache.org/jira/browse/AMBARI-17968


Repository: ambari


Description
---

Changed `oozie-site/oozie.authentication.kerberos.principal` and 
`oozie-site/oozie.authentication.kerberos.keytab` properties are reverted while 
regenerating keytab files.

The changed properties are needed to support Oozie in high availability (HA) 
mode with failover enabled via a load balancing proxy server.

*Cause*
The relevant part of the Kerberos descriptor for Oozie is:
```
{
  "name": "/spnego",
  "principal": {
"configuration": 
"oozie-site/oozie.authentication.kerberos.principal"
  },
  "keytab": {
"configuration": 
"oozie-site/oozie.authentication.kerberos.keytab"
  }
}
```

Because of this, certain Kerberos-related operations (like Regenerate Keytabs) 
resets the values of `oozie-site/oozie.authentication.kerberos.principal` and 
`oozie-site/oozie.authentication.kerberos.keytab` to match the principal name 
and keytab file of the Kerberos identity definition for `/spnego`.

However, in HA, the properties need to be something like:
```
oozie.authentication.kerberos.prinipal= "*"
oozie.authentication.kerberos.keytab = "/path/to/oozie_ha.keytab"
```

*Solution*
After enabling HA and either before or after enabling Kerberos, the following 
`oozie-site` properties may be set:
* `oozie.ha.authentication.kerberos.principal`
* `oozie.ha.authentication.kerberos.keytab`

If either exist when configuring Oozie, the value of the property will be used 
to update the relevant `oozie.authentication.kerberos.*` property.

For example:
* if `oozie.ha.authentication.kerberos.principal` is set, its value will set 
used to set `oozie.authentication.kerberos.principal`
* if `oozie.ha.authentication.kerberos.keytab` is set, its value will set used 
to set `oozie.authentication.kerberos.keytab`
Note: One or both may be set. 

So even though `oozie.authentication.kerberos.principal` will contain a 
principal name like `HTTP/_HOST@SOME.REALM`, when writing the oozie-site.xml 
file, the value for `oozie.authentication.kerberos.principal` will be written 
out as the value set for `oozie.ha.authentication.kerberos.principal`, which 
would typically be "*", when HA is enabled for Oozie.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
 23c9edc 
  
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py
 00229d3 
  ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 99d6dec 

Diff: https://reviews.apache.org/r/50647/diff/


Testing
---

Manually tested

# Local test results: 

[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 1:19.684s
[INFO] Finished at: Sun Jul 31 21:36:24 EDT 2016
[INFO] Final Memory: 71M/1705M
[INFO] 

# Jenkins test results: PENDING


Thanks,

Robert Levas

Re: Review Request 50647: Changed oozie.authentication.kerberos.principal and oozie.authentication.kerberos.keytab are reverted while regenerating keytab files

2016-07-31 Thread Robert Levas 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50647/
---

(Updated July 31, 2016, 10 p.m.)


Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jonathan 
Hurley, Jayush Luniya, Sumit Mohanty, Vitalyi Brodetskyi, and Venkat 
Ranganathan.


Bugs: AMBARI-17968
https://issues.apache.org/jira/browse/AMBARI-17968


Repository: ambari


Description (updated)
---

Changed `oozie-site/oozie.authentication.kerberos.principal` and 
`oozie-site/oozie.authentication.kerberos.keytab` properties are reverted while 
regenerating keytab files.

The changed properties are needed to support Oozie in high availability (HA) 
mode with failover enabled via a load balancing proxy server.

# Cause
The relevant part of the Kerberos descriptor for Oozie is:
```
{
  "name": "/spnego",
  "principal": {
"configuration": 
"oozie-site/oozie.authentication.kerberos.principal"
  },
  "keytab": {
"configuration": 
"oozie-site/oozie.authentication.kerberos.keytab"
  }
}
```

Because of this, certain Kerberos-related operations (like Regenerate Keytabs) 
resets the values of `oozie-site/oozie.authentication.kerberos.principal` and 
`oozie-site/oozie.authentication.kerberos.keytab` to match the principal name 
and keytab file of the Kerberos identity definition for `/spnego`.

However, in HA, the properties need to be something like:
```
oozie.authentication.kerberos.prinipal= "*"
oozie.authentication.kerberos.keytab = "/path/to/oozie_ha.keytab"
```

# Solution
After enabling HA and either before or after enabling Kerberos, the following 
`oozie-site` properties may be set:
* `oozie.ha.authentication.kerberos.principal`
* `oozie.ha.authentication.kerberos.keytab`

If either exist when configuring Oozie, the value of the property will be used 
to update the relevant `oozie.authentication.kerberos.*` property.

For example:
* if `oozie.ha.authentication.kerberos.principal` is set, its value will set 
used to set `oozie.authentication.kerberos.principal`
* if `oozie.ha.authentication.kerberos.keytab` is set, its value will set used 
to set `oozie.authentication.kerberos.keytab`
Note: One or both may be set. 

So even though `oozie.authentication.kerberos.principal` will contain a 
principal name like `HTTP/_HOST@SOME.REALM`, when writing the oozie-site.xml 
file, the value for `oozie.authentication.kerberos.principal` will be written 
out as the value set for `oozie.ha.authentication.kerberos.principal`, which 
would typically be "*", when HA is enabled for Oozie.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
 23c9edc 
  
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py
 00229d3 
  ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 99d6dec 

Diff: https://reviews.apache.org/r/50647/diff/


Testing
---

Manually tested

# Local test results: 

[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 1:19.684s
[INFO] Finished at: Sun Jul 31 21:36:24 EDT 2016
[INFO] Final Memory: 71M/1705M
[INFO] 

# Jenkins test results: PENDING


Thanks,

Robert Levas

Re: Review Request 50647: Changed oozie.authentication.kerberos.principal and oozie.authentication.kerberos.keytab are reverted while regenerating keytab files

2016-07-31 Thread Sumit Mohanty 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50647/#review144299
---




ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py
 (line 150)


Will the removal of the check cause issues for Ambari upgrade for a 
deployed 2.1 or 2.2 cluster?


- Sumit Mohanty


On Aug. 1, 2016, 2 a.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50647/
> ---
> 
> (Updated Aug. 1, 2016, 2 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jonathan 
> Hurley, Jayush Luniya, Sumit Mohanty, Vitalyi Brodetskyi, and Venkat 
> Ranganathan.
> 
> 
> Bugs: AMBARI-17968
> https://issues.apache.org/jira/browse/AMBARI-17968
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Changed `oozie-site/oozie.authentication.kerberos.principal` and 
> `oozie-site/oozie.authentication.kerberos.keytab` properties are reverted 
> while regenerating keytab files.
> 
> The changed properties are needed to support Oozie in high availability (HA) 
> mode with failover enabled via a load balancing proxy server.
> 
> # Cause
> The relevant part of the Kerberos descriptor for Oozie is:
> ```
> {
>   "name": "/spnego",
>   "principal": {
> "configuration": 
> "oozie-site/oozie.authentication.kerberos.principal"
>   },
>   "keytab": {
> "configuration": 
> "oozie-site/oozie.authentication.kerberos.keytab"
>   }
> }
> ```
> 
> Because of this, certain Kerberos-related operations (like Regenerate 
> Keytabs) resets the values of 
> `oozie-site/oozie.authentication.kerberos.principal` and 
> `oozie-site/oozie.authentication.kerberos.keytab` to match the principal name 
> and keytab file of the Kerberos identity definition for `/spnego`.
> 
> However, in HA, the properties need to be something like:
> ```
> oozie.authentication.kerberos.prinipal= "*"
> oozie.authentication.kerberos.keytab = "/path/to/oozie_ha.keytab"
> ```
> 
> # Solution
> After enabling HA and either before or after enabling Kerberos, the following 
> `oozie-site` properties may be set:
> * `oozie.ha.authentication.kerberos.principal`
> * `oozie.ha.authentication.kerberos.keytab`
> 
> If either exist when configuring Oozie, the value of the property will be 
> used to update the relevant `oozie.authentication.kerberos.*` property.
> 
> For example:
> * if `oozie.ha.authentication.kerberos.principal` is set, its value will set 
> used to set `oozie.authentication.kerberos.principal`
> * if `oozie.ha.authentication.kerberos.keytab` is set, its value will set 
> used to set `oozie.authentication.kerberos.keytab`
> Note: One or both may be set. 
> 
> So even though `oozie.authentication.kerberos.principal` will contain a 
> principal name like `HTTP/_HOST@SOME.REALM`, when writing the oozie-site.xml 
> file, the value for `oozie.authentication.kerberos.principal` will be written 
> out as the value set for `oozie.ha.authentication.kerberos.principal`, which 
> would typically be "*", when HA is enabled for Oozie.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
>  23c9edc 
>   
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py
>  00229d3 
>   ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 
> 99d6dec 
> 
> Diff: https://reviews.apache.org/r/50647/diff/
> 
> 
> Testing
> ---
> 
> Manually tested
> 
> # Local test results: 
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 1:19.684s
> [INFO] Finished at: Sun Jul 31 21:36:24 EDT 2016
> [INFO] Final Memory: 71M/1705M
> [INFO] 
> 
> 
> # Jenkins test results: PENDING
> 
> 
> Thanks,
> 
> Robert Levas
> 
>

Re: Review Request 50647: Changed oozie.authentication.kerberos.principal and oozie.authentication.kerberos.keytab are reverted while regenerating keytab files

2016-07-31 Thread Sumit Mohanty 


> On Aug. 1, 2016, 3:17 a.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py,
> >  line 150
> > 
> >
> > Will the removal of the check cause issues for Ambari upgrade for a 
> > deployed 2.1 or 2.2 cluster?

Sorry missed the whole change - ignore.


- Sumit


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50647/#review144299
---


On Aug. 1, 2016, 2 a.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50647/
> ---
> 
> (Updated Aug. 1, 2016, 2 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jonathan 
> Hurley, Jayush Luniya, Sumit Mohanty, Vitalyi Brodetskyi, and Venkat 
> Ranganathan.
> 
> 
> Bugs: AMBARI-17968
> https://issues.apache.org/jira/browse/AMBARI-17968
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Changed `oozie-site/oozie.authentication.kerberos.principal` and 
> `oozie-site/oozie.authentication.kerberos.keytab` properties are reverted 
> while regenerating keytab files.
> 
> The changed properties are needed to support Oozie in high availability (HA) 
> mode with failover enabled via a load balancing proxy server.
> 
> # Cause
> The relevant part of the Kerberos descriptor for Oozie is:
> ```
> {
>   "name": "/spnego",
>   "principal": {
> "configuration": 
> "oozie-site/oozie.authentication.kerberos.principal"
>   },
>   "keytab": {
> "configuration": 
> "oozie-site/oozie.authentication.kerberos.keytab"
>   }
> }
> ```
> 
> Because of this, certain Kerberos-related operations (like Regenerate 
> Keytabs) resets the values of 
> `oozie-site/oozie.authentication.kerberos.principal` and 
> `oozie-site/oozie.authentication.kerberos.keytab` to match the principal name 
> and keytab file of the Kerberos identity definition for `/spnego`.
> 
> However, in HA, the properties need to be something like:
> ```
> oozie.authentication.kerberos.prinipal= "*"
> oozie.authentication.kerberos.keytab = "/path/to/oozie_ha.keytab"
> ```
> 
> # Solution
> After enabling HA and either before or after enabling Kerberos, the following 
> `oozie-site` properties may be set:
> * `oozie.ha.authentication.kerberos.principal`
> * `oozie.ha.authentication.kerberos.keytab`
> 
> If either exist when configuring Oozie, the value of the property will be 
> used to update the relevant `oozie.authentication.kerberos.*` property.
> 
> For example:
> * if `oozie.ha.authentication.kerberos.principal` is set, its value will set 
> used to set `oozie.authentication.kerberos.principal`
> * if `oozie.ha.authentication.kerberos.keytab` is set, its value will set 
> used to set `oozie.authentication.kerberos.keytab`
> Note: One or both may be set. 
> 
> So even though `oozie.authentication.kerberos.principal` will contain a 
> principal name like `HTTP/_HOST@SOME.REALM`, when writing the oozie-site.xml 
> file, the value for `oozie.authentication.kerberos.principal` will be written 
> out as the value set for `oozie.ha.authentication.kerberos.principal`, which 
> would typically be "*", when HA is enabled for Oozie.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
>  23c9edc 
>   
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py
>  00229d3 
>   ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 
> 99d6dec 
> 
> Diff: https://reviews.apache.org/r/50647/diff/
> 
> 
> Testing
> ---
> 
> Manually tested
> 
> # Local test results: 
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 1:19.684s
> [INFO] Finished at: Sun Jul 31 21:36:24 EDT 2016
> [INFO] Final Memory: 71M/1705M
> [INFO] 
> 
> 
> # Jenkins test results: PENDING
> 
> 
> Thanks,
> 
> Robert Levas
> 
>

Re: Review Request 50647: Changed oozie.authentication.kerberos.principal and oozie.authentication.kerberos.keytab are reverted while regenerating keytab files

2016-07-31 Thread Sumit Mohanty 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50647/#review144302
---


Ship it!




Ship It!

- Sumit Mohanty


On Aug. 1, 2016, 2 a.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50647/
> ---
> 
> (Updated Aug. 1, 2016, 2 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Andrew Onischuk, Jonathan 
> Hurley, Jayush Luniya, Sumit Mohanty, Vitalyi Brodetskyi, and Venkat 
> Ranganathan.
> 
> 
> Bugs: AMBARI-17968
> https://issues.apache.org/jira/browse/AMBARI-17968
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Changed `oozie-site/oozie.authentication.kerberos.principal` and 
> `oozie-site/oozie.authentication.kerberos.keytab` properties are reverted 
> while regenerating keytab files.
> 
> The changed properties are needed to support Oozie in high availability (HA) 
> mode with failover enabled via a load balancing proxy server.
> 
> # Cause
> The relevant part of the Kerberos descriptor for Oozie is:
> ```
> {
>   "name": "/spnego",
>   "principal": {
> "configuration": 
> "oozie-site/oozie.authentication.kerberos.principal"
>   },
>   "keytab": {
> "configuration": 
> "oozie-site/oozie.authentication.kerberos.keytab"
>   }
> }
> ```
> 
> Because of this, certain Kerberos-related operations (like Regenerate 
> Keytabs) resets the values of 
> `oozie-site/oozie.authentication.kerberos.principal` and 
> `oozie-site/oozie.authentication.kerberos.keytab` to match the principal name 
> and keytab file of the Kerberos identity definition for `/spnego`.
> 
> However, in HA, the properties need to be something like:
> ```
> oozie.authentication.kerberos.prinipal= "*"
> oozie.authentication.kerberos.keytab = "/path/to/oozie_ha.keytab"
> ```
> 
> # Solution
> After enabling HA and either before or after enabling Kerberos, the following 
> `oozie-site` properties may be set:
> * `oozie.ha.authentication.kerberos.principal`
> * `oozie.ha.authentication.kerberos.keytab`
> 
> If either exist when configuring Oozie, the value of the property will be 
> used to update the relevant `oozie.authentication.kerberos.*` property.
> 
> For example:
> * if `oozie.ha.authentication.kerberos.principal` is set, its value will set 
> used to set `oozie.authentication.kerberos.principal`
> * if `oozie.ha.authentication.kerberos.keytab` is set, its value will set 
> used to set `oozie.authentication.kerberos.keytab`
> Note: One or both may be set. 
> 
> So even though `oozie.authentication.kerberos.principal` will contain a 
> principal name like `HTTP/_HOST@SOME.REALM`, when writing the oozie-site.xml 
> file, the value for `oozie.authentication.kerberos.principal` will be written 
> out as the value set for `oozie.ha.authentication.kerberos.principal`, which 
> would typically be "*", when HA is enabled for Oozie.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
>  23c9edc 
>   
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py
>  00229d3 
>   ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 
> 99d6dec 
> 
> Diff: https://reviews.apache.org/r/50647/diff/
> 
> 
> Testing
> ---
> 
> Manually tested
> 
> # Local test results: 
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 1:19.684s
> [INFO] Finished at: Sun Jul 31 21:36:24 EDT 2016
> [INFO] Final Memory: 71M/1705M
> [INFO] 
> 
> 
> # Jenkins test results: PENDING
> 
> 
> Thanks,
> 
> Robert Levas
> 
>