Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

2016-08-03 Thread Jayush Luniya 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144625
---


Ship it!




Ship It!

- Jayush Luniya


On Aug. 2, 2016, 8:37 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated Aug. 2, 2016, 8:37 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  6537e43 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  b297fa6 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  e046849 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> 6390f1f 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

2016-08-02 Thread Saisai Shao 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/
---

(Updated Aug. 2, 2016, 8:37 a.m.)


Review request for Ambari, Jayush Luniya and Sumit Mohanty.


Bugs: AMBARI-17954
https://issues.apache.org/jira/browse/AMBARI-17954


Repository: ambari


Description
---

Fix Spark hdp.version issues in upgrading and fresh install


Diffs (updated)
-

  
ambari-common/src/main/python/resource_management/libraries/functions/constants.py
 6537e43 
  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
 646239e 
  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
 86aeb70 
  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
 2b5d36b 
  
ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
 e27c986 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
 b297fa6 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
b295cc9 
  
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
 5b20a72 
  
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
 e046849 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
c9f9823 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
6390f1f 
  
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
 b3d05d8 
  ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
b2ed5df 
  ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
  ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
a1abdfa 

Diff: https://reviews.apache.org/r/50594/diff/


Testing
---

Manual test with different scenarios:

1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
2. Upgrade for 2.3.6 to 2.5.0.
3. Downgrade from 2.5.0 to 2.3.6.


Thanks,

Saisai Shao

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

2016-08-01 Thread Saisai Shao 


> On Aug. 2, 2016, 12:34 a.m., Jayush Luniya wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java,
> >  line 2273
> > 
> >
> > We need to replace hdp_full_version with full_stack_version. The SPARK 
> > code has been cleaned up to remove hdp hardcodings in Ambari 2.4. If you 
> > see params.py the variable hdp_full_version is not even defined. So when 
> > you upgrade ambari-server from 2.1.0 or 2.2.0 with my hotfix the config 
> > will remain -Dhdp.version={{hdp_full_version}}.
> 
> Saisai Shao wrote:
> From my understanding, you mean that in the params.py we should also take 
> care of {{hdp_full_version}} if amabri is upgraded from lower version. Can 
> you please explain more about it.
> 
> Jayush Luniya wrote:
> If you look at 
> https://github.com/apache/ambari/blob/branch-2.1.2/ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py
>  and 
> https://github.com/apache/ambari/blob/branch-2.1.2/ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-defaults.xml#L128-L133
>  you will see that we use to set these properties as  
> -Dhdp.version={{hdp_full_version}}. So when you upgrade Ambari 2.1.X -> 
> 2.4.0, these config properties will continue to be 
> -Dhdp.version={{hdp_full_version}}. However params.py doesnt have this 
> variable defined anymore. It has been renamed to full_stack_version. So when 
> we do an ambari-server upgrade we should update these config properties to 
> find and replace hdp_full_version with full_stack_version.

I understand now. So you mean that we still need to keep these codes rather 
than delete it, is that right?

Or put the logic into params.py?


- Saisai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144418
---


On Aug. 1, 2016, 1:22 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated Aug. 1, 2016, 1:22 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

2016-08-01 Thread Jayush Luniya 


> On Aug. 2, 2016, 12:59 a.m., Jayush Luniya wrote:
> > It doesnt look like you have verified ambari-server upgrade scenarios with 
> > the patch. This is very important to validate. You need to ensure backward 
> > compatibility is maintained
> > 
> > Please test the following scenarios
> > Scenario 1:
> > 1. Install Ambari 2.1.0
> > 2. Deploy HDP-2.3.2
> > 3. Upgrade to Ambari 2.4.0
> > 4. Restart SPARK
> > 5. Verify SPARK restarts
> > 6. Upgrade HDP to 2.3.4 and verify upgrade succeeds
> > 7. Also upgrade to HDP 2.4+ and verify upgrade succeeds.
> > 
> > Scenario 2:
> > 1. Install Ambari 2.1.0
> > 2. Deploy HDP-2.3.2
> > 3. Upgrade to Ambari 2.2.0
> > 4. Restart Spark
> > 5. Confirm that the Spark configs are not getting written correctly due to 
> > the backward compatibility issue. Restart spark will cause issues.
> > 6. Upgrade to Hotfix build 2.2.0.13-1
> > 7. Update java_opts using following command 
> > ./configs.sh set   
> > spark-javaopts-properties content "-Dhdp.version=hdp_full_version"
> > 8. Restart SPARK service
> > 9. Verify that the Spark configs are getting written correctly.
> > 10. Now upgrade Ambari to Ambari 2.4.0
> > 11. Restart Spark
> > 12. Verify SPARK restarts
> > 13. Upgrade to HDP 2.3.4 
> > 14. Also upgrade to HDP-2.4+
> 
> Saisai Shao wrote:
> Where to get the specific version of Ambari and how to upgrade to the 
> specific version?

http://docs.hortonworks.com/index.html  has information about how to install a 
specific Ambari version and how to upgrade Ambari

Installing Ambari 2.1.2.1
http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.1/bk_Installing_HDP_AMB/content/ch_Installing_Ambari.html

Upgrading to Ambari-2.2.0
http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_upgrading_Ambari/content/_upgrading_ambari.html


- Jayush


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144425
---


On Aug. 1, 2016, 1:22 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated Aug. 1, 2016, 1:22 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

2016-08-01 Thread Jayush Luniya 


> On Aug. 2, 2016, 12:34 a.m., Jayush Luniya wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java,
> >  line 2273
> > 
> >
> > We need to replace hdp_full_version with full_stack_version. The SPARK 
> > code has been cleaned up to remove hdp hardcodings in Ambari 2.4. If you 
> > see params.py the variable hdp_full_version is not even defined. So when 
> > you upgrade ambari-server from 2.1.0 or 2.2.0 with my hotfix the config 
> > will remain -Dhdp.version={{hdp_full_version}}.
> 
> Saisai Shao wrote:
> From my understanding, you mean that in the params.py we should also take 
> care of {{hdp_full_version}} if amabri is upgraded from lower version. Can 
> you please explain more about it.

If you look at 
https://github.com/apache/ambari/blob/branch-2.1.2/ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py
 and 
https://github.com/apache/ambari/blob/branch-2.1.2/ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/configuration/spark-defaults.xml#L128-L133
 you will see that we use to set these properties as  
-Dhdp.version={{hdp_full_version}}. So when you upgrade Ambari 2.1.X -> 2.4.0, 
these config properties will continue to be -Dhdp.version={{hdp_full_version}}. 
However params.py doesnt have this variable defined anymore. It has been 
renamed to full_stack_version. So when we do an ambari-server upgrade we should 
update these config properties to find and replace hdp_full_version with 
full_stack_version.


- Jayush


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144418
---


On Aug. 1, 2016, 1:22 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated Aug. 1, 2016, 1:22 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

2016-08-01 Thread Saisai Shao 


> On Aug. 2, 2016, 12:59 a.m., Jayush Luniya wrote:
> > It doesnt look like you have verified ambari-server upgrade scenarios with 
> > the patch. This is very important to validate. You need to ensure backward 
> > compatibility is maintained
> > 
> > Please test the following scenarios
> > Scenario 1:
> > 1. Install Ambari 2.1.0
> > 2. Deploy HDP-2.3.2
> > 3. Upgrade to Ambari 2.4.0
> > 4. Restart SPARK
> > 5. Verify SPARK restarts
> > 6. Upgrade HDP to 2.3.4 and verify upgrade succeeds
> > 7. Also upgrade to HDP 2.4+ and verify upgrade succeeds.
> > 
> > Scenario 2:
> > 1. Install Ambari 2.1.0
> > 2. Deploy HDP-2.3.2
> > 3. Upgrade to Ambari 2.2.0
> > 4. Restart Spark
> > 5. Confirm that the Spark configs are not getting written correctly due to 
> > the backward compatibility issue. Restart spark will cause issues.
> > 6. Upgrade to Hotfix build 2.2.0.13-1
> > 7. Update java_opts using following command 
> > ./configs.sh set   
> > spark-javaopts-properties content "-Dhdp.version=hdp_full_version"
> > 8. Restart SPARK service
> > 9. Verify that the Spark configs are getting written correctly.
> > 10. Now upgrade Ambari to Ambari 2.4.0
> > 11. Restart Spark
> > 12. Verify SPARK restarts
> > 13. Upgrade to HDP 2.3.4 
> > 14. Also upgrade to HDP-2.4+

Where to get the specific version of Ambari and how to upgrade to the specific 
version?


- Saisai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144425
---


On Aug. 1, 2016, 1:22 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated Aug. 1, 2016, 1:22 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

2016-08-01 Thread Jayush Luniya 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144425
---



It doesnt look like you have verified ambari-server upgrade scenarios with the 
patch. This is very important to validate. You need to ensure backward 
compatibility is maintained

Please test the following scenarios
Scenario 1:
1. Install Ambari 2.1.0
2. Deploy HDP-2.3.2
3. Upgrade to Ambari 2.4.0
4. Restart SPARK
5. Verify SPARK restarts
6. Upgrade HDP to 2.3.4 and verify upgrade succeeds
7. Also upgrade to HDP 2.4+ and verify upgrade succeeds.

Scenario 2:
1. Install Ambari 2.1.0
2. Deploy HDP-2.3.2
3. Upgrade to Ambari 2.2.0
4. Restart Spark
5. Confirm that the Spark configs are not getting written correctly due to the 
backward compatibility issue. Restart spark will cause issues.
6. Upgrade to Hotfix build 2.2.0.13-1
7. Update java_opts using following command 
./configs.sh set   
spark-javaopts-properties content "-Dhdp.version=hdp_full_version"
8. Restart SPARK service
9. Verify that the Spark configs are getting written correctly.
10. Now upgrade Ambari to Ambari 2.4.0
11. Restart Spark
12. Verify SPARK restarts
13. Upgrade to HDP 2.3.4 
14. Also upgrade to HDP-2.4+

- Jayush Luniya


On Aug. 1, 2016, 1:22 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated Aug. 1, 2016, 1:22 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

2016-08-01 Thread Jayush Luniya 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144421
---




ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
 (line 284)


You can set this to 2.3.4.0 so that we dont create this for HDP-2.3.4+ 
instead of HDP-2.4+. Also if you you want to keep it as < HDP-2.4, you can set 
max_version as "2.4.0.0" instead of "2.3.99.99". The max_version field is 
exclusive 
i.e. [min_version, max_version)


- Jayush Luniya


On Aug. 1, 2016, 1:22 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated Aug. 1, 2016, 1:22 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

2016-08-01 Thread Saisai Shao 


> On Aug. 2, 2016, 12:34 a.m., Jayush Luniya wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java,
> >  line 2273
> > 
> >
> > We need to replace hdp_full_version with full_stack_version. The SPARK 
> > code has been cleaned up to remove hdp hardcodings in Ambari 2.4. If you 
> > see params.py the variable hdp_full_version is not even defined. So when 
> > you upgrade ambari-server from 2.1.0 or 2.2.0 with my hotfix the config 
> > will remain -Dhdp.version={{hdp_full_version}}.

>From my understanding, you mean that in the params.py we should also take care 
>of {{hdp_full_version}} if amabri is upgraded from lower version. Can you 
>please explain more about it.


- Saisai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144418
---


On Aug. 1, 2016, 1:22 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated Aug. 1, 2016, 1:22 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

2016-08-01 Thread Jayush Luniya 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144418
---




ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
 


We need to replace hdp_full_version with full_stack_version. The SPARK code 
has been cleaned up to remove hdp hardcodings in Ambari 2.4. If you see 
params.py the variable hdp_full_version is not even defined. So when you 
upgrade ambari-server from 2.1.0 or 2.2.0 with my hotfix the config will remain 
-Dhdp.version={{hdp_full_version}}.



ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
 


Same as above



ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
 


Same as above


- Jayush Luniya


On Aug. 1, 2016, 1:22 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated Aug. 1, 2016, 1:22 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

2016-08-01 Thread Sumit Mohanty 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144416
---


Ship it!




Ship It!

- Sumit Mohanty


On Aug. 1, 2016, 1:22 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated Aug. 1, 2016, 1:22 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

2016-08-01 Thread Saisai Shao 


> On Aug. 1, 2016, 9:27 p.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py,
> >  line 130
> > 
> >
> > The addition of -Dhdp.version should also be under condition 
> > check_stack_feature(StackFeature.SPARK_JAVA_OPTS_SUPPORT, 
> > effective_version).
> > 
> > I assume -Dhdp.version is to be added only for HDP-2.3 and below.
> 
> Saisai Shao wrote:
> I think it's fine to leave as it is, since in HDP-2.4 above version, we 
> will remove this configuration and not generate the java-opts file.

Also this java-opts file is not used for any other user-level configuration, so 
it is safe to delete the file.


- Saisai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144388
---


On Aug. 1, 2016, 1:22 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated Aug. 1, 2016, 1:22 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

2016-08-01 Thread Saisai Shao 


> On Aug. 1, 2016, 9:27 p.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py,
> >  line 130
> > 
> >
> > The addition of -Dhdp.version should also be under condition 
> > check_stack_feature(StackFeature.SPARK_JAVA_OPTS_SUPPORT, 
> > effective_version).
> > 
> > I assume -Dhdp.version is to be added only for HDP-2.3 and below.

I think it's fine to leave as it is, since in HDP-2.4 above version, we will 
remove this configuration and not generate the java-opts file.


- Saisai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144388
---


On Aug. 1, 2016, 1:22 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated Aug. 1, 2016, 1:22 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

2016-08-01 Thread Sumit Mohanty 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144388
---




ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
 (line 129)


The addition of -Dhdp.version should also be under condition 
check_stack_feature(StackFeature.SPARK_JAVA_OPTS_SUPPORT, effective_version).

I assume -Dhdp.version is to be added only for HDP-2.3 and below.


- Sumit Mohanty


On Aug. 1, 2016, 1:22 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated Aug. 1, 2016, 1:22 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

2016-07-31 Thread Saisai Shao 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/
---

(Updated Aug. 1, 2016, 1:22 a.m.)


Review request for Ambari, Jayush Luniya and Sumit Mohanty.


Changes
---

Fix Spark hdp.version issues in upgrading and fresh install


Bugs: AMBARI-17954
https://issues.apache.org/jira/browse/AMBARI-17954


Repository: ambari


Description
---

Fix Spark hdp.version issues in upgrading and fresh install


Diffs (updated)
-

  
ambari-common/src/main/python/resource_management/libraries/functions/constants.py
 42ecf6a 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
 5495655 
  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
 646239e 
  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
 86aeb70 
  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
 2b5d36b 
  
ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
 e27c986 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
 6d26052 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
b295cc9 
  
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
 5b20a72 
  
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
 5d2cc54 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
c9f9823 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
d0332ed 
  
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
 b3d05d8 
  
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
 34ca199 
  ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
b2ed5df 
  ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
  ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
a1abdfa 

Diff: https://reviews.apache.org/r/50594/diff/


Testing
---

Manual test with different scenarios:

1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
2. Upgrade for 2.3.6 to 2.5.0.
3. Downgrade from 2.5.0 to 2.3.6.


Thanks,

Saisai Shao

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

2016-07-31 Thread Saisai Shao 


> On July 31, 2016, 10:02 p.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml,
> >  line 30
> > 
> >
> > Remove the line 

Done


> On July 31, 2016, 10:02 p.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml,
> >  line 24
> > 
> >
> > Remove the line 

Done


> On July 31, 2016, 10:02 p.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml,
> >  line 30
> > 
> >
> > Remove the line 

Done


> On July 31, 2016, 10:02 p.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml,
> >  line 25
> > 
> >
> > Remove the line 

Done


> On July 31, 2016, 10:02 p.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py,
> >  line 140
> > 
> >
> > Use delete option with File(). The helper method do the necessary work 
> > for non-root deployments.
> > 
> > e.g.
> > File(params.infra_solr_pidfile,
> >action="delete"
> >)

Done


- Saisai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144288
---


On July 31, 2016, 3:44 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated July 31, 2016, 3:44 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

2016-07-31 Thread Saisai Shao 


> On July 31, 2016, 10:02 p.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py,
> >  line 129
> > 
> >
> > I think the previous code was safer - spark_javaopts_properties = 
> > default("/configurations/spark-javaopts-properties/content", " ")
> > 
> > Any reason to switch to if ... else ...

I see, I will revert it back.


- Saisai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144288
---


On July 31, 2016, 3:44 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated July 31, 2016, 3:44 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

2016-07-31 Thread Sumit Mohanty 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/#review144288
---




ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
 (line 128)


I think the previous code was safer - spark_javaopts_properties = 
default("/configurations/spark-javaopts-properties/content", " ")

Any reason to switch to if ... else ...



ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
 (line 133)


Use delete option with File(). The helper method do the necessary work for 
non-root deployments.

e.g.
File(params.infra_solr_pidfile,
   action="delete"
   )



ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
 (line 25)


Remove the line 



ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
 (line 30)


Remove the line 



ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
 (line 24)


Remove the line 



ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
 (line 30)


Remove the line 


- Sumit Mohanty


On July 31, 2016, 3:44 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50594/
> ---
> 
> (Updated July 31, 2016, 3:44 a.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17954
> https://issues.apache.org/jira/browse/AMBARI-17954
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fix Spark hdp.version issues in upgrading and fresh install
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  42ecf6a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  5495655 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
>  646239e 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  86aeb70 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
>  2b5d36b 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
>  e27c986 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  6d26052 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> b295cc9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
>  5b20a72 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  5d2cc54 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
> c9f9823 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> d0332ed 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
>  b3d05d8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
>  34ca199 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
> b2ed5df 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
> a1abdfa 
> 
> Diff: https://reviews.apache.org/r/50594/diff/
> 
> 
> Testing
> ---
> 
> Manual test with different scenarios:
> 
> 1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
> 2. Upgrade for 2.3.6 to 2.5.0.
> 3. Downgrade from 2.5.0 to 2.3.6.
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 50594: Fix Spark hdp.version issues in upgrading and fresh install

2016-07-30 Thread Saisai Shao 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50594/
---

(Updated July 31, 2016, 3:44 a.m.)


Review request for Ambari, Jayush Luniya and Sumit Mohanty.


Changes
---

Fix Spark hdp.version issues in upgrading and fresh install


Bugs: AMBARI-17954
https://issues.apache.org/jira/browse/AMBARI-17954


Repository: ambari


Description
---

Fix Spark hdp.version issues in upgrading and fresh install


Diffs (updated)
-

  
ambari-common/src/main/python/resource_management/libraries/functions/constants.py
 42ecf6a 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
 5495655 
  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/configuration/spark-defaults.xml
 646239e 
  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
 86aeb70 
  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
 2b5d36b 
  
ambari-server/src/main/resources/common-services/SPARK/1.5.2/configuration/spark-thrift-sparkconf.xml
 e27c986 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
 6d26052 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
b295cc9 
  
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml
 5b20a72 
  
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
 5d2cc54 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 
c9f9823 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
d0332ed 
  
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-defaults.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.4/services/SPARK/configuration/spark-thrift-sparkconf.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/configuration/spark-defaults.xml
 b3d05d8 
  
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
 34ca199 
  ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 
b2ed5df 
  ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py ed331ea 
  ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 
a1abdfa 

Diff: https://reviews.apache.org/r/50594/diff/


Testing
---

Manual test with different scenarios:

1. Fresh install of HDP 2.3.6, 2.4.3, 2.5.0
2. Upgrade for 2.3.6 to 2.5.0.
3. Downgrade from 2.5.0 to 2.3.6.


Thanks,

Saisai Shao