Re: Review Request 53334: AMBARI-18758. Livy configuration change for upgrade

[Sumit Mohanty]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53334/#review155227
---


Ship it!




Ship It!

- Sumit Mohanty


On Nov. 8, 2016, 12:05 a.m., Jeff Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53334/
> ---
> 
> (Updated Nov. 8, 2016, 12:05 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Robert Levas, and Sumit 
> Mohanty.
> 
> 
> Bugs: AMBARI-18758
> https://issues.apache.org/jira/browse/AMBARI-18758
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> 2 configuration key are changed.
> 
> livy.server.kerberos.keytab  --> livy.server.launch.kerberos.keytab
> livy.server.kerberos.principal  --> livy.server.launch.kerberos.principal
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  3a1c482 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/53334/diff/
> 
> 
> Testing
> ---
> 
> I have verified it in HDP 2.6. But havn't done for the upgrade.
> 
> @afernandez, do you know how to change the configuration key for upgrade ? I 
> didn't find samples in config-upgrade.xml
> 
> 
> Thanks,
> 
> Jeff Zhang
> 
>

Re: Review Request 53334: AMBARI-18758. Livy configuration change for upgrade

[Jeff Zhang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53334/
---

(Updated Nov. 8, 2016, 12:05 a.m.)


Review request for Ambari, Alejandro Fernandez, Robert Levas, and Sumit Mohanty.


Bugs: AMBARI-18758
https://issues.apache.org/jira/browse/AMBARI-18758


Repository: ambari


Description
---

2 configuration key are changed.

livy.server.kerberos.keytab  --> livy.server.launch.kerberos.keytab
livy.server.kerberos.principal  --> livy.server.launch.kerberos.principal


Diffs (updated)
-

  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
 3a1c482 
  ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json 
PRE-CREATION 

Diff: https://reviews.apache.org/r/53334/diff/


Testing
---

I have verified it in HDP 2.6. But havn't done for the upgrade.

@afernandez, do you know how to change the configuration key for upgrade ? I 
didn't find samples in config-upgrade.xml


Thanks,

Jeff Zhang

Re: Review Request 53334: AMBARI-18758. Livy configuration change for upgrade

[Jeff Zhang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53334/
---

(Updated Nov. 4, 2016, 6:04 a.m.)


Review request for Ambari, Alejandro Fernandez, Robert Levas, and Sumit Mohanty.


Bugs: AMBARI-18758
https://issues.apache.org/jira/browse/AMBARI-18758


Repository: ambari


Description
---

2 configuration key are changed.

livy.server.kerberos.keytab  --> livy.server.launch.kerberos.keytab
livy.server.kerberos.principal  --> livy.server.launch.kerberos.principal


Diffs (updated)
-

  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
 3a1c482 
  ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json 
PRE-CREATION 

Diff: https://reviews.apache.org/r/53334/diff/


Testing
---

I have verified it in HDP 2.6. But havn't done for the upgrade.

@afernandez, do you know how to change the configuration key for upgrade ? I 
didn't find samples in config-upgrade.xml


Thanks,

Jeff Zhang

Re: Review Request 53334: AMBARI-18758. Livy configuration change for upgrade

[Robert Levas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53334/#review154728
---


Fix it, then Ship it!




To update existng configurations during an stack upgrade you may need to add 
tasks to the upgrade descrptors.


ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json 
(lines 51 - 59)


Use the newer form of a referenced identitiy:

```
{
  "name": "livy_spnego",
  "reference": "/spnego",
  "principal": {
"configuration": "livy-conf/livy.server.auth.kerberos.principal"
  },
  "keytab": {
"configuration": "livy-conf/livy.server.auth.kerberos.keytab"
  }
}
```



ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json 
(line 92)


Use the newer form for referenced identities... see commmet about `/spnego` 
above



ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json 
(line 103)


Use the newer form for referenced identities... see commmet about `/spnego` 
above



ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json 
(line 106)


Use the newer form for referenced identities... see commmet about `/spnego` 
above



ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json 
(line 114)


Use the newer form for referenced identities... see commmet about `/spnego` 
above


- Robert Levas


On Oct. 31, 2016, 9:52 p.m., Jeff Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53334/
> ---
> 
> (Updated Oct. 31, 2016, 9:52 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-18758
> https://issues.apache.org/jira/browse/AMBARI-18758
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> 2 configuration key are changed.
> 
> livy.server.kerberos.keytab  --> livy.server.launch.kerberos.keytab
> livy.server.kerberos.principal  --> livy.server.launch.kerberos.principal
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  3a1c482 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/53334/diff/
> 
> 
> Testing
> ---
> 
> I have verified it in HDP 2.6. But havn't done for the upgrade.
> 
> @afernandez, do you know how to change the configuration key for upgrade ? I 
> didn't find samples in config-upgrade.xml
> 
> 
> Thanks,
> 
> Jeff Zhang
> 
>

Re: Review Request 53334: AMBARI-18758. Livy configuration change for upgrade

[Jeff Zhang]

> On Nov. 3, 2016, 1:21 a.m., Sumit Mohanty wrote:
> > Lets have a bug open to fix it in the upgrade pack.
> > 
> > Pls talk to Robert Levas and see if the persisted kerberos.json need to be 
> > modified or it will get auto modified when stack is upgraded.

Thanks Sumit, create https://issues.apache.org/jira/browse/AMBARI-18780 for 
upgrade fix.


- Jeff


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53334/#review154666
---


On Nov. 1, 2016, 1:52 a.m., Jeff Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53334/
> ---
> 
> (Updated Nov. 1, 2016, 1:52 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-18758
> https://issues.apache.org/jira/browse/AMBARI-18758
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> 2 configuration key are changed.
> 
> livy.server.kerberos.keytab  --> livy.server.launch.kerberos.keytab
> livy.server.kerberos.principal  --> livy.server.launch.kerberos.principal
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  3a1c482 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/53334/diff/
> 
> 
> Testing
> ---
> 
> I have verified it in HDP 2.6. But havn't done for the upgrade.
> 
> @afernandez, do you know how to change the configuration key for upgrade ? I 
> didn't find samples in config-upgrade.xml
> 
> 
> Thanks,
> 
> Jeff Zhang
> 
>

Re: Review Request 53334: AMBARI-18758. Livy configuration change for upgrade

[Sumit Mohanty]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53334/#review154666
---



Lets have a bug open to fix it in the upgrade pack.

Pls talk to Robert Levas and see if the persisted kerberos.json need to be 
modified or it will get auto modified when stack is upgraded.

- Sumit Mohanty


On Nov. 1, 2016, 1:52 a.m., Jeff Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53334/
> ---
> 
> (Updated Nov. 1, 2016, 1:52 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-18758
> https://issues.apache.org/jira/browse/AMBARI-18758
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> 2 configuration key are changed.
> 
> livy.server.kerberos.keytab  --> livy.server.launch.kerberos.keytab
> livy.server.kerberos.principal  --> livy.server.launch.kerberos.principal
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  3a1c482 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/53334/diff/
> 
> 
> Testing
> ---
> 
> I have verified it in HDP 2.6. But havn't done for the upgrade.
> 
> @afernandez, do you know how to change the configuration key for upgrade ? I 
> didn't find samples in config-upgrade.xml
> 
> 
> Thanks,
> 
> Jeff Zhang
> 
>

Re: Review Request 53334: AMBARI-18758. Livy configuration change for upgrade

[Jeff Zhang]

> On Nov. 1, 2016, 5:36 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py,
> >  line 234
> > 
> >
> > Is this meant to account for a different Spark version in HDP stack?
> > If so, we can use stack_features.json and the check_stack_feature 
> > function to make it more clear why this is the case.
> 
> Jeff Zhang wrote:
> yes, this is only for HDP 2.6 or later version.  This is not feature, 
> just a compablity break, so I think it is not necessary to put it in 
> stack_features.json
> 
> Jeff Zhang wrote:
> BTW, do you know how to change the configuration key for upgrade ? I 
> didn't find samples in config-upgrade.xml
> 
> Alejandro Fernandez wrote:
> We still don't have upgrade packs and config packs for HDP 2.5 to 2.6

Do you mean currently upgrade from 2.5 to 2.6 is not supported, and I can add 
that to config-upgrade.xml when it is supported ?


- Jeff


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53334/#review154422
---


On Nov. 1, 2016, 1:52 a.m., Jeff Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53334/
> ---
> 
> (Updated Nov. 1, 2016, 1:52 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-18758
> https://issues.apache.org/jira/browse/AMBARI-18758
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> 2 configuration key are changed.
> 
> livy.server.kerberos.keytab  --> livy.server.launch.kerberos.keytab
> livy.server.kerberos.principal  --> livy.server.launch.kerberos.principal
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  3a1c482 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/53334/diff/
> 
> 
> Testing
> ---
> 
> I have verified it in HDP 2.6. But havn't done for the upgrade.
> 
> @afernandez, do you know how to change the configuration key for upgrade ? I 
> didn't find samples in config-upgrade.xml
> 
> 
> Thanks,
> 
> Jeff Zhang
> 
>

Re: Review Request 53334: AMBARI-18758. Livy configuration change for upgrade

[Alejandro Fernandez]

> On Nov. 1, 2016, 5:36 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py,
> >  line 234
> > 
> >
> > Is this meant to account for a different Spark version in HDP stack?
> > If so, we can use stack_features.json and the check_stack_feature 
> > function to make it more clear why this is the case.
> 
> Jeff Zhang wrote:
> yes, this is only for HDP 2.6 or later version.  This is not feature, 
> just a compablity break, so I think it is not necessary to put it in 
> stack_features.json
> 
> Jeff Zhang wrote:
> BTW, do you know how to change the configuration key for upgrade ? I 
> didn't find samples in config-upgrade.xml

We still don't have upgrade packs and config packs for HDP 2.5 to 2.6


- Alejandro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53334/#review154422
---


On Nov. 1, 2016, 1:52 a.m., Jeff Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53334/
> ---
> 
> (Updated Nov. 1, 2016, 1:52 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-18758
> https://issues.apache.org/jira/browse/AMBARI-18758
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> 2 configuration key are changed.
> 
> livy.server.kerberos.keytab  --> livy.server.launch.kerberos.keytab
> livy.server.kerberos.principal  --> livy.server.launch.kerberos.principal
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  3a1c482 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/53334/diff/
> 
> 
> Testing
> ---
> 
> I have verified it in HDP 2.6. But havn't done for the upgrade.
> 
> @afernandez, do you know how to change the configuration key for upgrade ? I 
> didn't find samples in config-upgrade.xml
> 
> 
> Thanks,
> 
> Jeff Zhang
> 
>

Re: Review Request 53334: AMBARI-18758. Livy configuration change for upgrade

[Jeff Zhang]

> On Nov. 1, 2016, 5:36 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py,
> >  line 234
> > 
> >
> > Is this meant to account for a different Spark version in HDP stack?
> > If so, we can use stack_features.json and the check_stack_feature 
> > function to make it more clear why this is the case.
> 
> Jeff Zhang wrote:
> yes, this is only for HDP 2.6 or later version.  This is not feature, 
> just a compablity break, so I think it is not necessary to put it in 
> stack_features.json

BTW, do you know how to change the configuration key for upgrade ? I didn't 
find samples in config-upgrade.xml


- Jeff


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53334/#review154422
---


On Nov. 1, 2016, 1:52 a.m., Jeff Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53334/
> ---
> 
> (Updated Nov. 1, 2016, 1:52 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-18758
> https://issues.apache.org/jira/browse/AMBARI-18758
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> 2 configuration key are changed.
> 
> livy.server.kerberos.keytab  --> livy.server.launch.kerberos.keytab
> livy.server.kerberos.principal  --> livy.server.launch.kerberos.principal
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  3a1c482 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/53334/diff/
> 
> 
> Testing
> ---
> 
> I have verified it in HDP 2.6. But havn't done for the upgrade.
> 
> @afernandez, do you know how to change the configuration key for upgrade ? I 
> didn't find samples in config-upgrade.xml
> 
> 
> Thanks,
> 
> Jeff Zhang
> 
>

Re: Review Request 53334: AMBARI-18758. Livy configuration change for upgrade

[Jeff Zhang]

> On Nov. 1, 2016, 5:36 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py,
> >  line 234
> > 
> >
> > Is this meant to account for a different Spark version in HDP stack?
> > If so, we can use stack_features.json and the check_stack_feature 
> > function to make it more clear why this is the case.

yes, this is only for HDP 2.6 or later version.  This is not feature, just a 
compablity break, so I think it is not necessary to put it in 
stack_features.json


- Jeff


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53334/#review154422
---


On Nov. 1, 2016, 1:52 a.m., Jeff Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53334/
> ---
> 
> (Updated Nov. 1, 2016, 1:52 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-18758
> https://issues.apache.org/jira/browse/AMBARI-18758
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> 2 configuration key are changed.
> 
> livy.server.kerberos.keytab  --> livy.server.launch.kerberos.keytab
> livy.server.kerberos.principal  --> livy.server.launch.kerberos.principal
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  3a1c482 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/53334/diff/
> 
> 
> Testing
> ---
> 
> I have verified it in HDP 2.6. But havn't done for the upgrade.
> 
> @afernandez, do you know how to change the configuration key for upgrade ? I 
> didn't find samples in config-upgrade.xml
> 
> 
> Thanks,
> 
> Jeff Zhang
> 
>

Re: Review Request 53334: AMBARI-18758. Livy configuration change for upgrade

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53334/#review154422
---




ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
 (line 234)


Is this meant to account for a different Spark version in HDP stack?
If so, we can use stack_features.json and the check_stack_feature function 
to make it more clear why this is the case.


- Alejandro Fernandez


On Nov. 1, 2016, 1:52 a.m., Jeff Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53334/
> ---
> 
> (Updated Nov. 1, 2016, 1:52 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-18758
> https://issues.apache.org/jira/browse/AMBARI-18758
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> 2 configuration key are changed.
> 
> livy.server.kerberos.keytab  --> livy.server.launch.kerberos.keytab
> livy.server.kerberos.principal  --> livy.server.launch.kerberos.principal
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
>  3a1c482 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/53334/diff/
> 
> 
> Testing
> ---
> 
> I have verified it in HDP 2.6. But havn't done for the upgrade.
> 
> @afernandez, do you know how to change the configuration key for upgrade ? I 
> didn't find samples in config-upgrade.xml
> 
> 
> Thanks,
> 
> Jeff Zhang
> 
>

Review Request 53334: AMBARI-18758. Livy configuration change for upgrade

[Jeff Zhang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53334/
---

Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.


Bugs: AMBARI-18758
https://issues.apache.org/jira/browse/AMBARI-18758


Repository: ambari


Description
---

2 configuration key are changed.

livy.server.kerberos.keytab  --> livy.server.launch.kerberos.keytab
livy.server.kerberos.principal  --> livy.server.launch.kerberos.principal


Diffs
-

  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
 3a1c482 
  ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json 
PRE-CREATION 

Diff: https://reviews.apache.org/r/53334/diff/


Testing
---

I have verified it in HDP 2.6. But havn't done for the upgrade.


Thanks,

Jeff Zhang

Re: Review Request 53334: AMBARI-18758. Livy configuration change for upgrade

[Jeff Zhang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53334/
---

(Updated Nov. 1, 2016, 1:52 a.m.)


Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.


Bugs: AMBARI-18758
https://issues.apache.org/jira/browse/AMBARI-18758


Repository: ambari


Description
---

2 configuration key are changed.

livy.server.kerberos.keytab  --> livy.server.launch.kerberos.keytab
livy.server.kerberos.principal  --> livy.server.launch.kerberos.principal


Diffs
-

  
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
 3a1c482 
  ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json 
PRE-CREATION 

Diff: https://reviews.apache.org/r/53334/diff/


Testing (updated)
---

I have verified it in HDP 2.6. But havn't done for the upgrade.

@afernandez, do you know how to change the configuration key for upgrade ? I 
didn't find samples in config-upgrade.xml


Thanks,

Jeff Zhang