date:20170725

Re: Review Request 61129: AMBARI-21573. Kafka service failed to start during regenerate keytab after upgrade from Biginsights 4.2.5, 4.2.0 to HDP 2.6.2

2017-07-25 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61129/#review181416
---


Ship it!




Ship It!

- Jonathan Hurley


On July 25, 2017, 8:42 p.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61129/
> ---
> 
> (Updated July 25, 2017, 8:42 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Di Li, Jonathan Hurley, Sumit 
> Mohanty, Sid Wagle, Tim Thorpe, and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-21573
> https://issues.apache.org/jira/browse/AMBARI-21573
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Steps to Reproduce
> 
> 1. Install IOP 4.2.5 with Kafka
> 2. Kerberize the cluster
> 3. Upgrade Ambari to 2.5.2
> 4. Register HDP 2.6
> 5. Remove iop-select
> 6. Install HDP 2.6.
> 7. Express Upgrade from BigInsights to HDP
> 8. Regenerate Keytab
> 
> After regenerate keytab, Kafka will go down.
> 
> The problem is that kafka-broker config type has 
> ```
> listeners: "SASL_PLAINTEXT://localhost:6667"
> security.inter.broker.protocol: "SASL_PLAINTEXT"
> ```
> 
> and it needs to replace "SASL_PLAINTEXT" with "PLAINTEXTSASL"
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
>  87a2aef 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  a7ddd5c 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
>  6d00a90 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  cedc90f 
> 
> 
> Diff: https://reviews.apache.org/r/61129/diff/2/
> 
> 
> Testing
> ---
> 
> Verified on a live cluster.
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Re: Review Request 61129: AMBARI-21573. Kafka service failed to start during regenerate keytab after upgrade from Biginsights 4.2.5, 4.2.0 to HDP 2.6.2

2017-07-25 Thread Alejandro Fernandez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61129/
---

(Updated July 26, 2017, 12:42 a.m.)


Review request for Ambari, Dmytro Grinenko, Di Li, Jonathan Hurley, Sumit 
Mohanty, Sid Wagle, Tim Thorpe, and Vitalyi Brodetskyi.


Bugs: AMBARI-21573
https://issues.apache.org/jira/browse/AMBARI-21573


Repository: ambari


Description
---

Steps to Reproduce

1. Install IOP 4.2.5 with Kafka
2. Kerberize the cluster
3. Upgrade Ambari to 2.5.2
4. Register HDP 2.6
5. Remove iop-select
6. Install HDP 2.6.
7. Express Upgrade from BigInsights to HDP
8. Regenerate Keytab

After regenerate keytab, Kafka will go down.

The problem is that kafka-broker config type has 
```
listeners: "SASL_PLAINTEXT://localhost:6667"
security.inter.broker.protocol: "SASL_PLAINTEXT"
```

and it needs to replace "SASL_PLAINTEXT" with "PLAINTEXTSASL"


Diffs (updated)
-

  
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
 87a2aef 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
 a7ddd5c 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
 6d00a90 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
 cedc90f 


Diff: https://reviews.apache.org/r/61129/diff/2/

Changes: https://reviews.apache.org/r/61129/diff/1-2/


Testing
---

Verified on a live cluster.


Thanks,

Alejandro Fernandez

Re: Review Request 61129: AMBARI-21573. Kafka service failed to start during regenerate keytab after upgrade from Biginsights 4.2.5, 4.2.0 to HDP 2.6.2

2017-07-25 Thread Sid Wagle


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61129/#review181414
---


Ship it!




Ship It!

- Sid Wagle


On July 26, 2017, 12:17 a.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61129/
> ---
> 
> (Updated July 26, 2017, 12:17 a.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Di Li, Jonathan Hurley, Sumit 
> Mohanty, Sid Wagle, Tim Thorpe, and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-21573
> https://issues.apache.org/jira/browse/AMBARI-21573
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Steps to Reproduce
> 
> 1. Install IOP 4.2.5 with Kafka
> 2. Kerberize the cluster
> 3. Upgrade Ambari to 2.5.2
> 4. Register HDP 2.6
> 5. Remove iop-select
> 6. Install HDP 2.6.
> 7. Express Upgrade from BigInsights to HDP
> 8. Regenerate Keytab
> 
> After regenerate keytab, Kafka will go down.
> 
> The problem is that kafka-broker config type has 
> ```
> listeners: "SASL_PLAINTEXT://localhost:6667"
> security.inter.broker.protocol: "SASL_PLAINTEXT"
> ```
> 
> and it needs to replace "SASL_PLAINTEXT" with "PLAINTEXTSASL"
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
>  87a2aef 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  a7ddd5c 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
>  6d00a90 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  cedc90f 
> 
> 
> Diff: https://reviews.apache.org/r/61129/diff/1/
> 
> 
> Testing
> ---
> 
> Verified on a live cluster.
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Review Request 61129: AMBARI-21573. Kafka service failed to start during regenerate keytab after upgrade from Biginsights 4.2.5, 4.2.0 to HDP 2.6.2

2017-07-25 Thread Alejandro Fernandez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61129/
---

Review request for Ambari, Dmytro Grinenko, Di Li, Jonathan Hurley, Sumit 
Mohanty, Sid Wagle, Tim Thorpe, and Vitalyi Brodetskyi.


Bugs: AMBARI-21573
https://issues.apache.org/jira/browse/AMBARI-21573


Repository: ambari


Description
---

Steps to Reproduce

1. Install IOP 4.2.5 with Kafka
2. Kerberize the cluster
3. Upgrade Ambari to 2.5.2
4. Register HDP 2.6
5. Remove iop-select
6. Install HDP 2.6.
7. Express Upgrade from BigInsights to HDP
8. Regenerate Keytab

After regenerate keytab, Kafka will go down.

The problem is that kafka-broker config type has 
```
listeners: "SASL_PLAINTEXT://localhost:6667"
security.inter.broker.protocol: "SASL_PLAINTEXT"
```

and it needs to replace "SASL_PLAINTEXT" with "PLAINTEXTSASL"


Diffs
-

  
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
 87a2aef 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
 a7ddd5c 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
 6d00a90 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
 cedc90f 


Diff: https://reviews.apache.org/r/61129/diff/1/


Testing
---

Verified on a live cluster.


Thanks,

Alejandro Fernandez

Re: Review Request 61107: Migrate custom extension support

2017-07-25 Thread Alejandro Fernandez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61107/#review181409
---



This will also need to be ported over to trunk for Hadoop 3.0.


ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/custom_extensions.py
Lines 38 (patched)


Why is iop hardcoded here?


- Alejandro Fernandez


On July 25, 2017, 11:29 p.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61107/
> ---
> 
> (Updated July 25, 2017, 11:29 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Sumit Mohanty, and Sid Wagle.
> 
> 
> Bugs: AMBARI-21570
> https://issues.apache.org/jira/browse/AMBARI-21570
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Support custom extension to load third party libraries for integration with 
> other vendors
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  c74e029 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/custom_extensions.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py
>  f7705c4 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py
>  2e7f5cd 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  31cf0c8 
>   ambari-server/src/main/resources/stacks/HDP/2.6/role_command_order.json 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/HBASE/configuration/hbase-env.xml
>  9afaecb 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/HDFS/configuration/hadoop-env.xml
>  04b9304 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/configuration/hive-env.xml
>  929c10d 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/YARN/configuration-mapred/mapred-site.xml
>  f03125b 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/YARN/configuration/yarn-site.xml
>  5f8de49 
> 
> 
> Diff: https://reviews.apache.org/r/61107/diff/3/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Re: Review Request 61107: Migrate custom extension support

2017-07-25 Thread Sumit Mohanty


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61107/#review181405
---




ambari-server/src/main/resources/stacks/HDP/2.6/role_command_order.json
Lines 6 (patched)


This, we can drop as we are not planning to add HBASE_REST_SERVER as a 
component.



ambari-server/src/main/resources/stacks/HDP/2.6/services/HBASE/configuration/hbase-env.xml
Lines 122 (patched)


This should not be true after Ambari upgrade. EU upgrade pack should edit 
the config property and append the path.



ambari-server/src/main/resources/stacks/HDP/2.6/services/YARN/configuration-mapred/mapred-site.xml
Lines 28 (patched)


Lets not make it true on-ambari-upgrade but we need a change in the EU 
upgrade pack to append the path to the config property.



ambari-server/src/main/resources/stacks/HDP/2.6/services/YARN/configuration/yarn-site.xml
Line 22 (original), 22 (patched)


Do we need this change during the EU?


- Sumit Mohanty


On July 25, 2017, 11:29 p.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61107/
> ---
> 
> (Updated July 25, 2017, 11:29 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Sumit Mohanty, and Sid Wagle.
> 
> 
> Bugs: AMBARI-21570
> https://issues.apache.org/jira/browse/AMBARI-21570
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Support custom extension to load third party libraries for integration with 
> other vendors
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  c74e029 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/custom_extensions.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py
>  f7705c4 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py
>  2e7f5cd 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  31cf0c8 
>   ambari-server/src/main/resources/stacks/HDP/2.6/role_command_order.json 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/HBASE/configuration/hbase-env.xml
>  9afaecb 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/HDFS/configuration/hadoop-env.xml
>  04b9304 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/configuration/hive-env.xml
>  929c10d 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/YARN/configuration-mapred/mapred-site.xml
>  f03125b 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/YARN/configuration/yarn-site.xml
>  5f8de49 
> 
> 
> Diff: https://reviews.apache.org/r/61107/diff/3/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Re: Review Request 61107: Migrate custom extension support

2017-07-25 Thread Vitalyi Brodetskyi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61107/
---

(Updated Липень 25, 2017, 11:29 після полудня)


Review request for Ambari, Andrew Onischuk, Sumit Mohanty, and Sid Wagle.


Bugs: AMBARI-21570
https://issues.apache.org/jira/browse/AMBARI-21570


Repository: ambari


Description
---

Support custom extension to load third party libraries for integration with 
other vendors


Diffs (updated)
-

  
ambari-common/src/main/python/resource_management/libraries/functions/constants.py
 c74e029 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/custom_extensions.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py
 f7705c4 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py
 2e7f5cd 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
 31cf0c8 
  ambari-server/src/main/resources/stacks/HDP/2.6/role_command_order.json 
PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.6/services/HBASE/configuration/hbase-env.xml
 9afaecb 
  
ambari-server/src/main/resources/stacks/HDP/2.6/services/HDFS/configuration/hadoop-env.xml
 04b9304 
  
ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/configuration/hive-env.xml
 929c10d 
  
ambari-server/src/main/resources/stacks/HDP/2.6/services/YARN/configuration-mapred/mapred-site.xml
 f03125b 
  
ambari-server/src/main/resources/stacks/HDP/2.6/services/YARN/configuration/yarn-site.xml
 5f8de49 


Diff: https://reviews.apache.org/r/61107/diff/3/

Changes: https://reviews.apache.org/r/61107/diff/2-3/


Testing
---

mvn clean test


Thanks,

Vitalyi Brodetskyi

Re: Review Request 61102: AMBARI-21568. Pig service check fails after upgrade

2017-07-25 Thread Alejandro Fernandez



> On July 25, 2017, 8:31 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
> > Lines 75 (patched)
> > 
> >
> > HDP 2.6 defaults to exectype=tez, so why was this needed?
> 
> Attila Doroszlai wrote:
> Because BI doesn't have Tez, nor does HDP after the upgrade.

Can you make this conditional if one of the tez properties does not exist?
Otherwise, we lose the context of why this is important.


- Alejandro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61102/#review181372
---


On July 25, 2017, 2:11 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61102/
> ---
> 
> (Updated July 25, 2017, 2:11 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush 
> Luniya, Oliver Szabo, Sumit Mohanty, Sebastian Toader, and Sid Wagle.
> 
> 
> Bugs: AMBARI-21568
> https://issues.apache.org/jira/browse/AMBARI-21568
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Execute Tez-specific part in Pig service check only if Tez is present in 
> the cluster.
> * HDP 2.6 comes with `exectype=tez`.  Change it back to `exectype=mapreduce` 
> for upgrade from BI.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/params_linux.py
>  5af127414288b29114212684d099301b6ba35357 
>   
> ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/service_check.py
>  ba4082e61a073cfb92ca992e01a6464d15eb2bd1 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
>  87a2aef4a612189510a9761760d8973959680511 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  a7ddd5c1a6cadf14acc4714c88a775fceb8e2a8d 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
>  6d00a90b679cef19c956ae9cb154c77574d88aec 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  cedc90fb47ccd6824bb36a9bd62f89429a48582a 
> 
> 
> Diff: https://reviews.apache.org/r/61102/diff/1/
> 
> 
> Testing
> ---
> 
> * Pig service check on cluster upgraded from BI 4.2:
> 
> ```
> ...
> 2017-07-24 19:16:05,079 [main] INFO  org.apache.pig.Main - Pig script 
> completed in 34 seconds and 863 milliseconds (34863 ms)
> 2017-07-24 19:16:06,008 - ExecuteHadoop['fs -test -e 
> /user/ambari-qa/pigsmoke.out'] {'bin_dir': 
> '/usr/hdp/current/hadoop-client/bin', 'user': 'ambari-qa', 'conf_dir': 
> '/usr/hdp/current/hadoop-client/conf'}
> 2017-07-24 19:16:06,012 - Execute['hadoop --config 
> /usr/hdp/current/hadoop-client/conf fs -test -e 
> /user/ambari-qa/pigsmoke.out'] {'logoutput': None, 'try_sleep': 0, 
> 'environment': {}, 'tries': 1, 'user': 'ambari-qa', 'path': 
> ['/usr/hdp/current/hadoop-client/bin']}
> 
> Command completed successfully!
> ```
> 
> * Pig service check on cluster upgraded from BI 4.2 with Tez installed after 
> the upgrade:
> 
> ```
> ...
> 2017-07-25 07:13:45,352 [pool-1-thread-1] INFO  
> org.apache.tez.client.TezClient - Shutting down Tez Session, 
> sessionName=PigLatin:pigSmoke.sh, applicationId=application_1500966646772_0002
> 2017-07-25 07:13:46,049 - ExecuteHadoop['fs -test -e 
> /user/ambari-qa/pigsmoke.out'] {'bin_dir': 
> '/usr/hdp/current/hadoop-client/bin', 'user': 'ambari-qa', 'conf_dir': 
> '/usr/hdp/current/hadoop-client/conf'}
> 2017-07-25 07:13:46,052 - Execute['hadoop --config 
> /usr/hdp/current/hadoop-client/conf fs -test -e 
> /user/ambari-qa/pigsmoke.out'] {'logoutput': None, 'try_sleep': 0, 
> 'environment': {}, 'tries': 1, 'user': 'ambari-qa', 'path': 
> ['/usr/hdp/current/hadoop-client/bin']}
> 
> Command completed successfully!
> ```
> 
> * Pig service check on cluster upgraded from BI 4.2.5:
> 
> ```
> 2017-07-25 08:16:17,511 [main] INFO  org.apache.pig.Main - Pig script 
> completed in 34 seconds and 276 milliseconds (34276 ms)
> 2017-07-25 08:16:18,209 - ExecuteHadoop['fs -test -e 
> /user/ambari-qa/pigsmoke.out'] {'bin_dir': 
> '/usr/hdp/current/hadoop-client/bin', 'user': 'ambari-qa', 'conf_dir': 
> '/usr/hdp/current/hadoop-client/conf'}
> 2017-07-25 08:16:18,212 - Execute['hadoop --config 
> /usr/hdp/current/hadoop-client/conf fs -test -e 
> /user/ambari-qa/pigsmoke.out'] {'logoutput': None, 'try_sleep': 0, 
> 'environment': {}, 'tries': 1, 'user': 'ambari-qa', 'path': 
> ['/usr/hdp/current/hadoop-client/bin']}
> 
> Command completed successfully!
> ```
> 
> * Pig service check on fresh HDP 2.6 install.
> 
> 
> Thanks,
> 
>

Re: Review Request 61102: AMBARI-21568. Pig service check fails after upgrade

2017-07-25 Thread Attila Doroszlai



> On July 25, 2017, 10:31 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/params_linux.py
> > Lines 43 (patched)
> > 
> >
> > We should instead look if any hosts have Tez Client.

What's the condition for that?


> On July 25, 2017, 10:31 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
> > Lines 75 (patched)
> > 
> >
> > HDP 2.6 defaults to exectype=tez, so why was this needed?

Because BI doesn't have Tez, nor does HDP after the upgrade.


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61102/#review181372
---


On July 25, 2017, 4:11 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61102/
> ---
> 
> (Updated July 25, 2017, 4:11 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush 
> Luniya, Oliver Szabo, Sumit Mohanty, Sebastian Toader, and Sid Wagle.
> 
> 
> Bugs: AMBARI-21568
> https://issues.apache.org/jira/browse/AMBARI-21568
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Execute Tez-specific part in Pig service check only if Tez is present in 
> the cluster.
> * HDP 2.6 comes with `exectype=tez`.  Change it back to `exectype=mapreduce` 
> for upgrade from BI.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/params_linux.py
>  5af127414288b29114212684d099301b6ba35357 
>   
> ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/service_check.py
>  ba4082e61a073cfb92ca992e01a6464d15eb2bd1 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
>  87a2aef4a612189510a9761760d8973959680511 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  a7ddd5c1a6cadf14acc4714c88a775fceb8e2a8d 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
>  6d00a90b679cef19c956ae9cb154c77574d88aec 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  cedc90fb47ccd6824bb36a9bd62f89429a48582a 
> 
> 
> Diff: https://reviews.apache.org/r/61102/diff/1/
> 
> 
> Testing
> ---
> 
> * Pig service check on cluster upgraded from BI 4.2:
> 
> ```
> ...
> 2017-07-24 19:16:05,079 [main] INFO  org.apache.pig.Main - Pig script 
> completed in 34 seconds and 863 milliseconds (34863 ms)
> 2017-07-24 19:16:06,008 - ExecuteHadoop['fs -test -e 
> /user/ambari-qa/pigsmoke.out'] {'bin_dir': 
> '/usr/hdp/current/hadoop-client/bin', 'user': 'ambari-qa', 'conf_dir': 
> '/usr/hdp/current/hadoop-client/conf'}
> 2017-07-24 19:16:06,012 - Execute['hadoop --config 
> /usr/hdp/current/hadoop-client/conf fs -test -e 
> /user/ambari-qa/pigsmoke.out'] {'logoutput': None, 'try_sleep': 0, 
> 'environment': {}, 'tries': 1, 'user': 'ambari-qa', 'path': 
> ['/usr/hdp/current/hadoop-client/bin']}
> 
> Command completed successfully!
> ```
> 
> * Pig service check on cluster upgraded from BI 4.2 with Tez installed after 
> the upgrade:
> 
> ```
> ...
> 2017-07-25 07:13:45,352 [pool-1-thread-1] INFO  
> org.apache.tez.client.TezClient - Shutting down Tez Session, 
> sessionName=PigLatin:pigSmoke.sh, applicationId=application_1500966646772_0002
> 2017-07-25 07:13:46,049 - ExecuteHadoop['fs -test -e 
> /user/ambari-qa/pigsmoke.out'] {'bin_dir': 
> '/usr/hdp/current/hadoop-client/bin', 'user': 'ambari-qa', 'conf_dir': 
> '/usr/hdp/current/hadoop-client/conf'}
> 2017-07-25 07:13:46,052 - Execute['hadoop --config 
> /usr/hdp/current/hadoop-client/conf fs -test -e 
> /user/ambari-qa/pigsmoke.out'] {'logoutput': None, 'try_sleep': 0, 
> 'environment': {}, 'tries': 1, 'user': 'ambari-qa', 'path': 
> ['/usr/hdp/current/hadoop-client/bin']}
> 
> Command completed successfully!
> ```
> 
> * Pig service check on cluster upgraded from BI 4.2.5:
> 
> ```
> 2017-07-25 08:16:17,511 [main] INFO  org.apache.pig.Main - Pig script 
> completed in 34 seconds and 276 milliseconds (34276 ms)
> 2017-07-25 08:16:18,209 - ExecuteHadoop['fs -test -e 
> /user/ambari-qa/pigsmoke.out'] {'bin_dir': 
> '/usr/hdp/current/hadoop-client/bin', 'user': 'ambari-qa', 'conf_dir': 
> '/usr/hdp/current/hadoop-client/conf'}
> 2017-07-25 08:16:18,212 - Execute['hadoop --config 
> /usr/hdp/current/hadoop-client/conf fs -test -e 
> /user/ambari-qa/pigsmoke.out'] {'logoutput': None, 'try_sleep': 0, 
> 'environment': {}, 'tries

Re: Review Request 61102: AMBARI-21568. Pig service check fails after upgrade

2017-07-25 Thread Alejandro Fernandez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61102/#review181372
---




ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/params_linux.py
Lines 43 (patched)


We should instead look if any hosts have Tez Client.



ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
Lines 75 (patched)


HDP 2.6 defaults to exectype=tez, so why was this needed?


- Alejandro Fernandez


On July 25, 2017, 2:11 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61102/
> ---
> 
> (Updated July 25, 2017, 2:11 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush 
> Luniya, Oliver Szabo, Sumit Mohanty, Sebastian Toader, and Sid Wagle.
> 
> 
> Bugs: AMBARI-21568
> https://issues.apache.org/jira/browse/AMBARI-21568
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Execute Tez-specific part in Pig service check only if Tez is present in 
> the cluster.
> * HDP 2.6 comes with `exectype=tez`.  Change it back to `exectype=mapreduce` 
> for upgrade from BI.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/params_linux.py
>  5af127414288b29114212684d099301b6ba35357 
>   
> ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/service_check.py
>  ba4082e61a073cfb92ca992e01a6464d15eb2bd1 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
>  87a2aef4a612189510a9761760d8973959680511 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  a7ddd5c1a6cadf14acc4714c88a775fceb8e2a8d 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
>  6d00a90b679cef19c956ae9cb154c77574d88aec 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  cedc90fb47ccd6824bb36a9bd62f89429a48582a 
> 
> 
> Diff: https://reviews.apache.org/r/61102/diff/1/
> 
> 
> Testing
> ---
> 
> * Pig service check on cluster upgraded from BI 4.2:
> 
> ```
> ...
> 2017-07-24 19:16:05,079 [main] INFO  org.apache.pig.Main - Pig script 
> completed in 34 seconds and 863 milliseconds (34863 ms)
> 2017-07-24 19:16:06,008 - ExecuteHadoop['fs -test -e 
> /user/ambari-qa/pigsmoke.out'] {'bin_dir': 
> '/usr/hdp/current/hadoop-client/bin', 'user': 'ambari-qa', 'conf_dir': 
> '/usr/hdp/current/hadoop-client/conf'}
> 2017-07-24 19:16:06,012 - Execute['hadoop --config 
> /usr/hdp/current/hadoop-client/conf fs -test -e 
> /user/ambari-qa/pigsmoke.out'] {'logoutput': None, 'try_sleep': 0, 
> 'environment': {}, 'tries': 1, 'user': 'ambari-qa', 'path': 
> ['/usr/hdp/current/hadoop-client/bin']}
> 
> Command completed successfully!
> ```
> 
> * Pig service check on cluster upgraded from BI 4.2 with Tez installed after 
> the upgrade:
> 
> ```
> ...
> 2017-07-25 07:13:45,352 [pool-1-thread-1] INFO  
> org.apache.tez.client.TezClient - Shutting down Tez Session, 
> sessionName=PigLatin:pigSmoke.sh, applicationId=application_1500966646772_0002
> 2017-07-25 07:13:46,049 - ExecuteHadoop['fs -test -e 
> /user/ambari-qa/pigsmoke.out'] {'bin_dir': 
> '/usr/hdp/current/hadoop-client/bin', 'user': 'ambari-qa', 'conf_dir': 
> '/usr/hdp/current/hadoop-client/conf'}
> 2017-07-25 07:13:46,052 - Execute['hadoop --config 
> /usr/hdp/current/hadoop-client/conf fs -test -e 
> /user/ambari-qa/pigsmoke.out'] {'logoutput': None, 'try_sleep': 0, 
> 'environment': {}, 'tries': 1, 'user': 'ambari-qa', 'path': 
> ['/usr/hdp/current/hadoop-client/bin']}
> 
> Command completed successfully!
> ```
> 
> * Pig service check on cluster upgraded from BI 4.2.5:
> 
> ```
> 2017-07-25 08:16:17,511 [main] INFO  org.apache.pig.Main - Pig script 
> completed in 34 seconds and 276 milliseconds (34276 ms)
> 2017-07-25 08:16:18,209 - ExecuteHadoop['fs -test -e 
> /user/ambari-qa/pigsmoke.out'] {'bin_dir': 
> '/usr/hdp/current/hadoop-client/bin', 'user': 'ambari-qa', 'conf_dir': 
> '/usr/hdp/current/hadoop-client/conf'}
> 2017-07-25 08:16:18,212 - Execute['hadoop --config 
> /usr/hdp/current/hadoop-client/conf fs -test -e 
> /user/ambari-qa/pigsmoke.out'] {'logoutput': None, 'try_sleep': 0, 
> 'environment': {}, 'tries': 1, 'user': 'ambari-qa', 'path': 
> ['/usr/hdp/current/hadoop-client/bin']}
> 
> Command completed successfully!
> ```
> 
> * Pig service check on fresh HDP 2.6 install.
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 61083: AMBARI-21563 : Copy Hadoop core-site.xml to Atlas conf when hadoop is configured for LDAP/AD.

2017-07-25 Thread Sumit Mohanty


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61083/#review181373
---




ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
Lines 210 (patched)


Multiple copies of core-site is fine isn't it. Assuming Atlas is now marked 
for restart when core-site is changed.


- Sumit Mohanty


On July 24, 2017, 2:51 p.m., Vishal Suvagia wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61083/
> ---
> 
> (Updated July 24, 2017, 2:51 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jayush Luniya, 
> Madhan Neethiraj, Mugdha Varadkar, Nixon Rodrigues, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-21563
> https://issues.apache.org/jira/browse/AMBARI-21563
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Atlas use user groups synced in hadoopUGI for authentication and 
> authorization process for LDAP/Kerberos type.
> As core-site.xml was available in HBase conf which is in Atlas classpath and 
> so core.site.xml configs were available. But now as core-site.xml is not 
> available in Hbase conf, to fix this issue hadoop core-site.xml need to be 
> copied to Atlas conf directory by Ambari.
> 
> PS: Attached patch is applicable for trunk branch, patch for branch-2.5 is 
> attached on Apache JIRA.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  c74e029 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
>  d424f5b 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
>  3ed469a 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  31cf0c8 
>   ambari-server/src/main/resources/stacks/HDP/2.6/services/ATLAS/metainfo.xml 
> 6e1a52e 
>   ambari-server/src/test/python/stacks/2.3/ATLAS/test_metadata_server.py 
> d1d8b7f 
>   ambari-server/src/test/python/stacks/2.5/ATLAS/test_atlas_server.py 539bef5 
> 
> 
> Diff: https://reviews.apache.org/r/61083/diff/1/
> 
> 
> Testing
> ---
> 
> Verified with installation on CentOS 6.
> 
> 
> Thanks,
> 
> Vishal Suvagia
> 
>

Re: Review Request 61105: Auto populate zookeeper.connect in Ranger Kafka plugin properties

2017-07-25 Thread Alejandro Fernandez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61105/#review181371
---


Ship it!




Ship It!

- Alejandro Fernandez


On July 25, 2017, 10:28 a.m., Mugdha Varadkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61105/
> ---
> 
> (Updated July 25, 2017, 10:28 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez and Gautam Borad.
> 
> 
> Bugs: AMBARI-21560
> https://issues.apache.org/jira/browse/AMBARI-21560
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Auto populate zookeeper.connect in ranger-kafka-plugin-properties which 
> always points to the localhost:2181, that cause test connection and resource 
> lookup failure in Kafka service created on Ranger Admin.
> 
> 
> Patch for trunk is attached on 
> https://issues.apache.org/jira/secure/attachment/12878770/AMBARI-21560-trunk.patch
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/KAFKA/0.9.0/configuration/ranger-kafka-plugin-properties.xml
>  3949402 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> d0f82af 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> fe3d26d 
> 
> 
> Diff: https://reviews.apache.org/r/61105/diff/1/
> 
> 
> Testing
> ---
> 
> Unit Test Report:
> 
> test_recommendKAFKAConfigurations (test_stack_advisor.TestHDP23StackAdvisor) 
> ...  ok
> 
> 
> Thanks,
> 
> Mugdha Varadkar
> 
>

Re: Review Request 61083: AMBARI-21563 : Copy Hadoop core-site.xml to Atlas conf when hadoop is configured for LDAP/AD.

2017-07-25 Thread Alejandro Fernandez



> On July 24, 2017, 4:22 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
> > Lines 210 (patched)
> > 
> >
> > Why do we need to delete?
> 
> Vishal Suvagia wrote:
> Hi Jayush, 
> The core-site.xml file is present in the HBase conf directory for 
> previous stacks, so if we create core-site for previous stacks then this may 
> lead to duplicate configurations for core-site.xml i.e one core-site.xml in 
> HBase conf directory and one in Atlas conf directory, hence the deletion.

Can you add a comment explaining this logic? Then ship it.


- Alejandro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61083/#review181221
---


On July 24, 2017, 2:51 p.m., Vishal Suvagia wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61083/
> ---
> 
> (Updated July 24, 2017, 2:51 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jayush Luniya, 
> Madhan Neethiraj, Mugdha Varadkar, Nixon Rodrigues, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-21563
> https://issues.apache.org/jira/browse/AMBARI-21563
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Atlas use user groups synced in hadoopUGI for authentication and 
> authorization process for LDAP/Kerberos type.
> As core-site.xml was available in HBase conf which is in Atlas classpath and 
> so core.site.xml configs were available. But now as core-site.xml is not 
> available in Hbase conf, to fix this issue hadoop core-site.xml need to be 
> copied to Atlas conf directory by Ambari.
> 
> PS: Attached patch is applicable for trunk branch, patch for branch-2.5 is 
> attached on Apache JIRA.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  c74e029 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
>  d424f5b 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
>  3ed469a 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  31cf0c8 
>   ambari-server/src/main/resources/stacks/HDP/2.6/services/ATLAS/metainfo.xml 
> 6e1a52e 
>   ambari-server/src/test/python/stacks/2.3/ATLAS/test_metadata_server.py 
> d1d8b7f 
>   ambari-server/src/test/python/stacks/2.5/ATLAS/test_atlas_server.py 539bef5 
> 
> 
> Diff: https://reviews.apache.org/r/61083/diff/1/
> 
> 
> Testing
> ---
> 
> Verified with installation on CentOS 6.
> 
> 
> Thanks,
> 
> Vishal Suvagia
> 
>

Re: Review Request 61107: Migrate custom extension support

2017-07-25 Thread Vitalyi Brodetskyi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61107/
---

(Updated Липень 25, 2017, 7:31 після полудня)


Review request for Ambari, Andrew Onischuk, Sumit Mohanty, and Sid Wagle.


Bugs: AMBARI-21570
https://issues.apache.org/jira/browse/AMBARI-21570


Repository: ambari


Description
---

Support custom extension to load third party libraries for integration with 
other vendors


Diffs (updated)
-

  
ambari-common/src/main/python/resource_management/libraries/functions/constants.py
 c74e029 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/custom_extensions.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py
 f7705c4 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py
 2e7f5cd 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
 31cf0c8 
  ambari-server/src/main/resources/stacks/HDP/2.6/role_command_order.json 
PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.6/services/HDFS/configuration/hadoop-env.xml
 04b9304 
  
ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/configuration/hive-env.xml
 929c10d 
  
ambari-server/src/main/resources/stacks/HDP/2.6/services/YARN/configuration-mapred/mapred-site.xml
 f03125b 
  
ambari-server/src/main/resources/stacks/HDP/2.6/services/YARN/configuration/yarn-site.xml
 5f8de49 


Diff: https://reviews.apache.org/r/61107/diff/2/

Changes: https://reviews.apache.org/r/61107/diff/1-2/


Testing
---

mvn clean test


Thanks,

Vitalyi Brodetskyi

Re: Review Request 60403: Add Kerberos HTTP SPNEGO authentication support to Ambari Metrics Monitor

2017-07-25 Thread Qin Liu



> On July 5, 2017, 6:38 p.m., Robert Levas wrote:
> > ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/spnego_kerberos_auth.py
> > Lines 33 (patched)
> > 
> >
> > Would it be possible to use 
> > `resource_management.libraries.functions.curl_krb_request.curl_krb_request` 
> > instead?

Hi Robert, 
Thank you so much for your review! I just came back from vacation and will 
re-visit it ASAP.


- Qin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60403/#review179673
---


On June 23, 2017, 10:49 p.m., Qin Liu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60403/
> ---
> 
> (Updated June 23, 2017, 10:49 p.m.)
> 
> 
> Review request for Ambari, Aravindan Vijayan, Di Li, Dmytro Sen, Robert 
> Levas, Sid Wagle, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-17898
> https://issues.apache.org/jira/browse/AMBARI-17898
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> This is a subtask of AMBARI-14384 "Ambari Metrics doesn't use SPNEGO to 
> authenticate".
> In a Kerberos enabled cluster with SPNEGO enabled on Hadoop APIs, Ambari 
> Metrics Collector web-console will be Kerberos HTTP SPNEGO enabled too. But 
> Ambari Metrics Monitor, a client of Ambari Metrics Collector, currently does 
> not support Kerberos HTTP SPNEGO authentication.
> /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out:
> 2015-12-15 13:26:30,663 [INFO] emitter.py:101 - server: 
> http://metrics-collector:6188/ws/v1/timeline/metrics
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:84 - Error sending metrics to 
> server. HTTP Error 401: Authentication required
> 2015-12-15 13:26:30,671 [WARNING] emitter.py:90 - Retrying after 5 ...
> 
> 
> Diffs
> -
> 
>   ambari-metrics/ambari-metrics-host-monitoring/conf/unix/metric_monitor.ini 
> 7fe7397 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/config_reader.py
>  d1429ed 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/emitter.py 
> 77b8c23 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/krberr.py 
> PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/spnego_kerberos_auth.py
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/test/python/core/TestEmitter.py
>  4056ae3 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/kerberos.json
>  51f541f 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/metainfo.xml
>  9031b46 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/ams.py
>  f49d47d 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
>  b8c14f4 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/templates/metric_monitor.ini.j2
>  b7dee50 
> 
> 
> Diff: https://reviews.apache.org/r/60403/diff/1/
> 
> 
> Testing
> ---
> 
> 1. Unit tests passed
> 2. manually tested with HDP/trunk
> 
> 
> Thanks,
> 
> Qin Liu
> 
>

Re: Review Request 61102: AMBARI-21568. Pig service check fails after upgrade

2017-07-25 Thread Sebastian Toader


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61102/#review181345
---


Ship it!




Ship It!

- Sebastian Toader


On July 25, 2017, 4:11 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61102/
> ---
> 
> (Updated July 25, 2017, 4:11 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush 
> Luniya, Oliver Szabo, Sumit Mohanty, Sebastian Toader, and Sid Wagle.
> 
> 
> Bugs: AMBARI-21568
> https://issues.apache.org/jira/browse/AMBARI-21568
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Execute Tez-specific part in Pig service check only if Tez is present in 
> the cluster.
> * HDP 2.6 comes with `exectype=tez`.  Change it back to `exectype=mapreduce` 
> for upgrade from BI.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/params_linux.py
>  5af127414288b29114212684d099301b6ba35357 
>   
> ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/service_check.py
>  ba4082e61a073cfb92ca992e01a6464d15eb2bd1 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
>  87a2aef4a612189510a9761760d8973959680511 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  a7ddd5c1a6cadf14acc4714c88a775fceb8e2a8d 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
>  6d00a90b679cef19c956ae9cb154c77574d88aec 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  cedc90fb47ccd6824bb36a9bd62f89429a48582a 
> 
> 
> Diff: https://reviews.apache.org/r/61102/diff/1/
> 
> 
> Testing
> ---
> 
> * Pig service check on cluster upgraded from BI 4.2:
> 
> ```
> ...
> 2017-07-24 19:16:05,079 [main] INFO  org.apache.pig.Main - Pig script 
> completed in 34 seconds and 863 milliseconds (34863 ms)
> 2017-07-24 19:16:06,008 - ExecuteHadoop['fs -test -e 
> /user/ambari-qa/pigsmoke.out'] {'bin_dir': 
> '/usr/hdp/current/hadoop-client/bin', 'user': 'ambari-qa', 'conf_dir': 
> '/usr/hdp/current/hadoop-client/conf'}
> 2017-07-24 19:16:06,012 - Execute['hadoop --config 
> /usr/hdp/current/hadoop-client/conf fs -test -e 
> /user/ambari-qa/pigsmoke.out'] {'logoutput': None, 'try_sleep': 0, 
> 'environment': {}, 'tries': 1, 'user': 'ambari-qa', 'path': 
> ['/usr/hdp/current/hadoop-client/bin']}
> 
> Command completed successfully!
> ```
> 
> * Pig service check on cluster upgraded from BI 4.2 with Tez installed after 
> the upgrade:
> 
> ```
> ...
> 2017-07-25 07:13:45,352 [pool-1-thread-1] INFO  
> org.apache.tez.client.TezClient - Shutting down Tez Session, 
> sessionName=PigLatin:pigSmoke.sh, applicationId=application_1500966646772_0002
> 2017-07-25 07:13:46,049 - ExecuteHadoop['fs -test -e 
> /user/ambari-qa/pigsmoke.out'] {'bin_dir': 
> '/usr/hdp/current/hadoop-client/bin', 'user': 'ambari-qa', 'conf_dir': 
> '/usr/hdp/current/hadoop-client/conf'}
> 2017-07-25 07:13:46,052 - Execute['hadoop --config 
> /usr/hdp/current/hadoop-client/conf fs -test -e 
> /user/ambari-qa/pigsmoke.out'] {'logoutput': None, 'try_sleep': 0, 
> 'environment': {}, 'tries': 1, 'user': 'ambari-qa', 'path': 
> ['/usr/hdp/current/hadoop-client/bin']}
> 
> Command completed successfully!
> ```
> 
> * Pig service check on cluster upgraded from BI 4.2.5:
> 
> ```
> 2017-07-25 08:16:17,511 [main] INFO  org.apache.pig.Main - Pig script 
> completed in 34 seconds and 276 milliseconds (34276 ms)
> 2017-07-25 08:16:18,209 - ExecuteHadoop['fs -test -e 
> /user/ambari-qa/pigsmoke.out'] {'bin_dir': 
> '/usr/hdp/current/hadoop-client/bin', 'user': 'ambari-qa', 'conf_dir': 
> '/usr/hdp/current/hadoop-client/conf'}
> 2017-07-25 08:16:18,212 - Execute['hadoop --config 
> /usr/hdp/current/hadoop-client/conf fs -test -e 
> /user/ambari-qa/pigsmoke.out'] {'logoutput': None, 'try_sleep': 0, 
> 'environment': {}, 'tries': 1, 'user': 'ambari-qa', 'path': 
> ['/usr/hdp/current/hadoop-client/bin']}
> 
> Command completed successfully!
> ```
> 
> * Pig service check on fresh HDP 2.6 install.
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 61107: Migrate custom extension support

2017-07-25 Thread Andrew Onischuk


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61107/#review181336
---




ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/custom_extensions.py
Lines 147 (patched)


This might not work with non-root.
Needs to be tested



ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/custom_extensions.py
Lines 167 (patched)


We don't need to force ownership upon deleting


- Andrew Onischuk


On July 25, 2017, 12:36 p.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61107/
> ---
> 
> (Updated July 25, 2017, 12:36 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Sumit Mohanty, and Sid Wagle.
> 
> 
> Bugs: AMBARI-21570
> https://issues.apache.org/jira/browse/AMBARI-21570
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Support custom extension to load third party libraries for integration with 
> other vendors
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  c74e029 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/custom_extensions.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py
>  f7705c4 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py
>  2e7f5cd 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  31cf0c8 
>   ambari-server/src/main/resources/stacks/HDP/2.6/role_command_order.json 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/HDFS/configuration/hadoop-env.xml
>  04b9304 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/configuration/hive-env.xml
>  929c10d 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/YARN/configuration-mapred/mapred-site.xml
>  f03125b 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/YARN/configuration/yarn-site.xml
>  5f8de49 
> 
> 
> Diff: https://reviews.apache.org/r/61107/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Re: Review Request 61107: Migrate custom extension support

2017-07-25 Thread Andrew Onischuk


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61107/#review181335
---




ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/custom_extensions.py
Lines 118 (patched)


Some cutomers might have /tmp not-executable.
So we avoid using it by agent or server.
Can we use format("{tmp_dir}/custom_extensions") instead?


- Andrew Onischuk


On July 25, 2017, 12:36 p.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61107/
> ---
> 
> (Updated July 25, 2017, 12:36 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Sumit Mohanty, and Sid Wagle.
> 
> 
> Bugs: AMBARI-21570
> https://issues.apache.org/jira/browse/AMBARI-21570
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Support custom extension to load third party libraries for integration with 
> other vendors
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  c74e029 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/custom_extensions.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py
>  f7705c4 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py
>  2e7f5cd 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  31cf0c8 
>   ambari-server/src/main/resources/stacks/HDP/2.6/role_command_order.json 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/HDFS/configuration/hadoop-env.xml
>  04b9304 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/configuration/hive-env.xml
>  929c10d 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/YARN/configuration-mapred/mapred-site.xml
>  f03125b 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/YARN/configuration/yarn-site.xml
>  5f8de49 
> 
> 
> Diff: https://reviews.apache.org/r/61107/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Review Request 61107: Migrate custom extension support

2017-07-25 Thread Vitalyi Brodetskyi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61107/
---

Review request for Ambari, Andrew Onischuk, Sumit Mohanty, and Sid Wagle.


Bugs: AMBARI-21570
https://issues.apache.org/jira/browse/AMBARI-21570


Repository: ambari


Description
---

Support custom extension to load third party libraries for integration with 
other vendors


Diffs
-

  
ambari-common/src/main/python/resource_management/libraries/functions/constants.py
 c74e029 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/custom_extensions.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py
 f7705c4 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py
 2e7f5cd 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
 31cf0c8 
  ambari-server/src/main/resources/stacks/HDP/2.6/role_command_order.json 
PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.6/services/HDFS/configuration/hadoop-env.xml
 04b9304 
  
ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/configuration/hive-env.xml
 929c10d 
  
ambari-server/src/main/resources/stacks/HDP/2.6/services/YARN/configuration-mapred/mapred-site.xml
 f03125b 
  
ambari-server/src/main/resources/stacks/HDP/2.6/services/YARN/configuration/yarn-site.xml
 5f8de49 


Diff: https://reviews.apache.org/r/61107/diff/1/


Testing
---

mvn clean test


Thanks,

Vitalyi Brodetskyi

Re: Review Request 61079: AMBARI-21559. Alert definition paths should be updated on upgrade

2017-07-25 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61079/#review181334
---


Ship it!




Ship It!

- Jonathan Hurley


On July 25, 2017, 6:06 a.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61079/
> ---
> 
> (Updated July 25, 2017, 6:06 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Sumit 
> Mohanty, and Sid Wagle.
> 
> 
> Bugs: AMBARI-21559
> https://issues.apache.org/jira/browse/AMBARI-21559
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Update script paths during Ambari upgrade.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  2d13cbaa764dfdf8338f0fcd2aaf48083515f825 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java
>  e8c986b9a51ad557f1f600ff77ad89fc38f5707c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/alert/AlertDefinitionFactory.java
>  acbb881264ae2c89a371063f57214e42c7ca7001 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/alert/ScriptSource.java
>  d1b70701ae26c3117f7b9fb8ece2676c0f3d45cb 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/AbstractFinalUpgradeCatalog.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/FinalUpgradeCatalog.java
>  1ac011862bff70742e99d627068225617d73d66d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/SchemaUpgradeHelper.java
>  af2a7d1cce7cfbc590415fff431be2d6d7308078 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpdateAlertScriptPaths.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java
>  bc1c19af6d637af5d2284f65fe05fa7f53bc4f90 
> 
> 
> Diff: https://reviews.apache.org/r/61079/diff/2/
> 
> 
> Testing
> ---
> 
> Tested manually on local cluster.  No alerts are skipped due to non-existent 
> scripts.
> 
> ```
> 24 Jul 2017 11:17:54,753  INFO [main] AmbariMetaInfo:1186 - Updating script 
> path for the alert named namenode_ha_health from 
> 'HDFS/2.7.2.4.2/package/alerts/alert_ha_namenode_health.py' to 
> 'BigInsights/4.2/services/HDFS/package/alerts/alert_ha_namenode_health.py'
> 24 Jul 2017 11:17:54,753  INFO [main] AmbariMetaInfo:1186 - Updating script 
> path for the alert named upgrade_finalized_state from 
> 'HDFS/2.7.2.4.2/package/alerts/alert_upgrade_finalized.py' to 
> 'BigInsights/4.2/services/HDFS/package/alerts/alert_upgrade_finalized.py'
> 24 Jul 2017 11:17:54,753  INFO [main] AmbariMetaInfo:1186 - Updating script 
> path for the alert named datanode_unmounted_data_dir from 
> 'HDFS/2.7.2.4.2/package/alerts/alert_datanode_unmounted_data_dir.py' to 
> 'BigInsights/4.2/services/HDFS/package/alerts/alert_datanode_unmounted_data_dir.py'
> 24 Jul 2017 11:17:54,754  INFO [main] AmbariMetaInfo:1186 - Updating script 
> path for the alert named namenode_last_checkpoint from 
> 'HDFS/2.7.2.4.2/package/alerts/alert_checkpoint_time.py' to 
> 'BigInsights/4.2/services/HDFS/package/alerts/alert_checkpoint_time.py'
> 24 Jul 2017 11:17:54,754  INFO [main] AmbariMetaInfo:1186 - Updating script 
> path for the alert named flume_agent_status from 
> 'FLUME/1.6.0.4.2/package/alerts/alert_flume_agent_status.py' to 
> 'BigInsights/4.2/services/FLUME/package/alerts/alert_flume_agent_status.py'
> 24 Jul 2017 11:17:54,754  INFO [main] AmbariMetaInfo:1186 - Updating script 
> path for the alert named ams_metrics_monitor_process from 
> 'AMBARI_METRICS/0.1.0.4.1/package/alerts/alert_ambari_metrics_monitor.py' to 
> 'BigInsights/4.2/services/AMBARI_METRICS/package/alerts/alert_ambari_metrics_monitor.py'
> ```
> 
> Existing unit tests:
> 
> ```
> $ mvn -am -pl ambari-server clean test
> ...
> Tests run: 5003, Failures: 0, Errors: 0, Skipped: 39
> ...
> Total run:1162
> Total errors:0
> Total failures:0
> OK
> ...
> [INFO] BUILD SUCCESS
> ```
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Review Request 61105: Auto populate zookeeper.connect in Ranger Kafka plugin properties

2017-07-25 Thread Mugdha Varadkar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61105/
---

Review request for Ambari, Alejandro Fernandez and Gautam Borad.


Bugs: AMBARI-21560
https://issues.apache.org/jira/browse/AMBARI-21560


Repository: ambari


Description
---

Auto populate zookeeper.connect in ranger-kafka-plugin-properties which always 
points to the localhost:2181, that cause test connection and resource lookup 
failure in Kafka service created on Ranger Admin.


Patch for trunk is attached on 
https://issues.apache.org/jira/secure/attachment/12878770/AMBARI-21560-trunk.patch


Diffs
-

  
ambari-server/src/main/resources/common-services/KAFKA/0.9.0/configuration/ranger-kafka-plugin-properties.xml
 3949402 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
d0f82af 
  ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py fe3d26d 


Diff: https://reviews.apache.org/r/61105/diff/1/


Testing
---

Unit Test Report:

test_recommendKAFKAConfigurations (test_stack_advisor.TestHDP23StackAdvisor) 
...  ok


Thanks,

Mugdha Varadkar

Re: Review Request 61079: AMBARI-21559. Alert definition paths should be updated on upgrade

2017-07-25 Thread Attila Doroszlai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61079/
---

(Updated July 25, 2017, 12:06 p.m.)


Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Sumit Mohanty, 
and Sid Wagle.


Changes
---

keep most of `source`, only update `path`


Bugs: AMBARI-21559
https://issues.apache.org/jira/browse/AMBARI-21559


Repository: ambari


Description
---

Update script paths during Ambari upgrade.


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
 2d13cbaa764dfdf8338f0fcd2aaf48083515f825 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java
 e8c986b9a51ad557f1f600ff77ad89fc38f5707c 
  
ambari-server/src/main/java/org/apache/ambari/server/state/alert/AlertDefinitionFactory.java
 acbb881264ae2c89a371063f57214e42c7ca7001 
  
ambari-server/src/main/java/org/apache/ambari/server/state/alert/ScriptSource.java
 d1b70701ae26c3117f7b9fb8ece2676c0f3d45cb 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/AbstractFinalUpgradeCatalog.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/FinalUpgradeCatalog.java
 1ac011862bff70742e99d627068225617d73d66d 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/SchemaUpgradeHelper.java
 af2a7d1cce7cfbc590415fff431be2d6d7308078 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpdateAlertScriptPaths.java
 PRE-CREATION 
  
ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java
 bc1c19af6d637af5d2284f65fe05fa7f53bc4f90 


Diff: https://reviews.apache.org/r/61079/diff/2/

Changes: https://reviews.apache.org/r/61079/diff/1-2/


Testing
---

Tested manually on local cluster.  No alerts are skipped due to non-existent 
scripts.

```
24 Jul 2017 11:17:54,753  INFO [main] AmbariMetaInfo:1186 - Updating script 
path for the alert named namenode_ha_health from 
'HDFS/2.7.2.4.2/package/alerts/alert_ha_namenode_health.py' to 
'BigInsights/4.2/services/HDFS/package/alerts/alert_ha_namenode_health.py'
24 Jul 2017 11:17:54,753  INFO [main] AmbariMetaInfo:1186 - Updating script 
path for the alert named upgrade_finalized_state from 
'HDFS/2.7.2.4.2/package/alerts/alert_upgrade_finalized.py' to 
'BigInsights/4.2/services/HDFS/package/alerts/alert_upgrade_finalized.py'
24 Jul 2017 11:17:54,753  INFO [main] AmbariMetaInfo:1186 - Updating script 
path for the alert named datanode_unmounted_data_dir from 
'HDFS/2.7.2.4.2/package/alerts/alert_datanode_unmounted_data_dir.py' to 
'BigInsights/4.2/services/HDFS/package/alerts/alert_datanode_unmounted_data_dir.py'
24 Jul 2017 11:17:54,754  INFO [main] AmbariMetaInfo:1186 - Updating script 
path for the alert named namenode_last_checkpoint from 
'HDFS/2.7.2.4.2/package/alerts/alert_checkpoint_time.py' to 
'BigInsights/4.2/services/HDFS/package/alerts/alert_checkpoint_time.py'
24 Jul 2017 11:17:54,754  INFO [main] AmbariMetaInfo:1186 - Updating script 
path for the alert named flume_agent_status from 
'FLUME/1.6.0.4.2/package/alerts/alert_flume_agent_status.py' to 
'BigInsights/4.2/services/FLUME/package/alerts/alert_flume_agent_status.py'
24 Jul 2017 11:17:54,754  INFO [main] AmbariMetaInfo:1186 - Updating script 
path for the alert named ams_metrics_monitor_process from 
'AMBARI_METRICS/0.1.0.4.1/package/alerts/alert_ambari_metrics_monitor.py' to 
'BigInsights/4.2/services/AMBARI_METRICS/package/alerts/alert_ambari_metrics_monitor.py'
```

Existing unit tests:

```
$ mvn -am -pl ambari-server clean test
...
Tests run: 5003, Failures: 0, Errors: 0, Skipped: 39
...
Total run:1162
Total errors:0
Total failures:0
OK
...
[INFO] BUILD SUCCESS
```


Thanks,

Attila Doroszlai

Re: Review Request 61074: Restart due to Stale configurations leads to an internal system exception

2017-07-25 Thread Dmytro Sen



> On Июль 24, 2017, 5:35 п.п., Attila Doroszlai wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
> > Lines 1086 (patched)
> > 
> >
> > Shouldn't it `return`, too?  Otherwise what prevents the exception a 
> > few lines later?

You're right. Thank you.


- Dmytro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61074/#review181231
---


On Июль 25, 2017, 8:59 д.п., Dmytro Sen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61074/
> ---
> 
> (Updated Июль 25, 2017, 8:59 д.п.)
> 
> 
> Review request for Ambari, Aravindan Vijayan, Sumit Mohanty, Sid Wagle, and 
> Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-21551
> https://issues.apache.org/jira/browse/AMBARI-21551
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> While performing RBAC API tests, found the following:
> POST call: 
> {code}
> http://:8080/api/v1/clusters/cl1/requests
> {"RequestInfo":{"command":"RESTART","context":"Restart all required 
> services","operation_level":"host_component"},"Requests/resource_filters":[{"hosts_predicate":"HostRoles/stale_configs=true"}]}
> {code}
> Returns:
> {code}
> {
> "status" : 500,
> message" : "An internal system exception occurred: Command execution cannot 
> proceed without a resource filter."
> }
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
>  f4b650e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RequestResourceProvider.java
>  a8f7638 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RequestResourceProviderTest.java
>  d243e73 
> 
> 
> Diff: https://reviews.apache.org/r/61074/diff/2/
> 
> 
> Testing
> ---
> 
> Unit tests passed
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Re: Review Request 61074: Restart due to Stale configurations leads to an internal system exception

2017-07-25 Thread Dmytro Sen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61074/
---

(Updated Июль 25, 2017, 8:59 д.п.)


Review request for Ambari, Aravindan Vijayan, Sumit Mohanty, Sid Wagle, and 
Vitalyi Brodetskyi.


Bugs: AMBARI-21551
https://issues.apache.org/jira/browse/AMBARI-21551


Repository: ambari


Description
---

While performing RBAC API tests, found the following:
POST call: 
{code}
http://:8080/api/v1/clusters/cl1/requests
{"RequestInfo":{"command":"RESTART","context":"Restart all required 
services","operation_level":"host_component"},"Requests/resource_filters":[{"hosts_predicate":"HostRoles/stale_configs=true"}]}
{code}
Returns:
{code}
{
"status" : 500,
message" : "An internal system exception occurred: Command execution cannot 
proceed without a resource filter."
}
{code}


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
 f4b650e 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RequestResourceProvider.java
 a8f7638 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RequestResourceProviderTest.java
 d243e73 


Diff: https://reviews.apache.org/r/61074/diff/2/

Changes: https://reviews.apache.org/r/61074/diff/1-2/


Testing
---

Unit tests passed


Thanks,

Dmytro Sen

Review Request 61102: AMBARI-21568. Pig service check fails after upgrade

2017-07-25 Thread Attila Doroszlai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61102/
---

Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Sumit Mohanty, 
and Sid Wagle.


Bugs: AMBARI-21568
https://issues.apache.org/jira/browse/AMBARI-21568


Repository: ambari


Description
---

* Execute Tez-specific part in Pig service check only if Tez is present in the 
cluster.
* HDP 2.6 comes with `exectype=tez`.  Change it back to `exectype=mapreduce` 
for upgrade from BI.


Diffs
-

  
ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/params_linux.py
 5af127414288b29114212684d099301b6ba35357 
  
ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/service_check.py
 ba4082e61a073cfb92ca992e01a6464d15eb2bd1 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
 87a2aef4a612189510a9761760d8973959680511 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
 a7ddd5c1a6cadf14acc4714c88a775fceb8e2a8d 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
 6d00a90b679cef19c956ae9cb154c77574d88aec 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
 cedc90fb47ccd6824bb36a9bd62f89429a48582a 


Diff: https://reviews.apache.org/r/61102/diff/1/


Testing
---

* Pig service check on cluster upgraded from BI 4.2:

```
...
2017-07-24 19:16:05,079 [main] INFO  org.apache.pig.Main - Pig script completed 
in 34 seconds and 863 milliseconds (34863 ms)
2017-07-24 19:16:06,008 - ExecuteHadoop['fs -test -e 
/user/ambari-qa/pigsmoke.out'] {'bin_dir': 
'/usr/hdp/current/hadoop-client/bin', 'user': 'ambari-qa', 'conf_dir': 
'/usr/hdp/current/hadoop-client/conf'}
2017-07-24 19:16:06,012 - Execute['hadoop --config 
/usr/hdp/current/hadoop-client/conf fs -test -e /user/ambari-qa/pigsmoke.out'] 
{'logoutput': None, 'try_sleep': 0, 'environment': {}, 'tries': 1, 'user': 
'ambari-qa', 'path': ['/usr/hdp/current/hadoop-client/bin']}

Command completed successfully!
```

* Pig service check on cluster upgraded from BI 4.2 with Tez installed after 
the upgrade:

```
...
2017-07-25 07:13:45,352 [pool-1-thread-1] INFO  org.apache.tez.client.TezClient 
- Shutting down Tez Session, sessionName=PigLatin:pigSmoke.sh, 
applicationId=application_1500966646772_0002
2017-07-25 07:13:46,049 - ExecuteHadoop['fs -test -e 
/user/ambari-qa/pigsmoke.out'] {'bin_dir': 
'/usr/hdp/current/hadoop-client/bin', 'user': 'ambari-qa', 'conf_dir': 
'/usr/hdp/current/hadoop-client/conf'}
2017-07-25 07:13:46,052 - Execute['hadoop --config 
/usr/hdp/current/hadoop-client/conf fs -test -e /user/ambari-qa/pigsmoke.out'] 
{'logoutput': None, 'try_sleep': 0, 'environment': {}, 'tries': 1, 'user': 
'ambari-qa', 'path': ['/usr/hdp/current/hadoop-client/bin']}

Command completed successfully!
```

* Pig service check on cluster upgraded from BI 4.2.5:

```
2017-07-25 08:16:17,511 [main] INFO  org.apache.pig.Main - Pig script completed 
in 34 seconds and 276 milliseconds (34276 ms)
2017-07-25 08:16:18,209 - ExecuteHadoop['fs -test -e 
/user/ambari-qa/pigsmoke.out'] {'bin_dir': 
'/usr/hdp/current/hadoop-client/bin', 'user': 'ambari-qa', 'conf_dir': 
'/usr/hdp/current/hadoop-client/conf'}
2017-07-25 08:16:18,212 - Execute['hadoop --config 
/usr/hdp/current/hadoop-client/conf fs -test -e /user/ambari-qa/pigsmoke.out'] 
{'logoutput': None, 'try_sleep': 0, 'environment': {}, 'tries': 1, 'user': 
'ambari-qa', 'path': ['/usr/hdp/current/hadoop-client/bin']}

Command completed successfully!
```

* Pig service check on fresh HDP 2.6 install.


Thanks,

Attila Doroszlai

Re: Review Request 61083: AMBARI-21563 : Copy Hadoop core-site.xml to Atlas conf when hadoop is configured for LDAP/AD.

2017-07-25 Thread Vishal Suvagia via Review Board



> On July 24, 2017, 4:22 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
> > Lines 210 (patched)
> > 
> >
> > Why do we need to delete?

Hi Jayush, 
The core-site.xml file is present in the HBase conf directory for previous 
stacks, so if we create core-site for previous stacks then this may lead to 
duplicate configurations for core-site.xml i.e one core-site.xml in HBase conf 
directory and one in Atlas conf directory, hence the deletion.


- Vishal


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61083/#review181221
---


On July 24, 2017, 2:51 p.m., Vishal Suvagia wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61083/
> ---
> 
> (Updated July 24, 2017, 2:51 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jayush Luniya, 
> Madhan Neethiraj, Mugdha Varadkar, Nixon Rodrigues, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-21563
> https://issues.apache.org/jira/browse/AMBARI-21563
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Atlas use user groups synced in hadoopUGI for authentication and 
> authorization process for LDAP/Kerberos type.
> As core-site.xml was available in HBase conf which is in Atlas classpath and 
> so core.site.xml configs were available. But now as core-site.xml is not 
> available in Hbase conf, to fix this issue hadoop core-site.xml need to be 
> copied to Atlas conf directory by Ambari.
> 
> PS: Attached patch is applicable for trunk branch, patch for branch-2.5 is 
> attached on Apache JIRA.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/constants.py
>  c74e029 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
>  d424f5b 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
>  3ed469a 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json
>  31cf0c8 
>   ambari-server/src/main/resources/stacks/HDP/2.6/services/ATLAS/metainfo.xml 
> 6e1a52e 
>   ambari-server/src/test/python/stacks/2.3/ATLAS/test_metadata_server.py 
> d1d8b7f 
>   ambari-server/src/test/python/stacks/2.5/ATLAS/test_atlas_server.py 539bef5 
> 
> 
> Diff: https://reviews.apache.org/r/61083/diff/1/
> 
> 
> Testing
> ---
> 
> Verified with installation on CentOS 6.
> 
> 
> Thanks,
> 
> Vishal Suvagia
> 
>

Re: Review Request 61129: AMBARI-21573. Kafka service failed to start during regenerate keytab after upgrade from Biginsights 4.2.5, 4.2.0 to HDP 2.6.2

Re: Review Request 61129: AMBARI-21573. Kafka service failed to start during regenerate keytab after upgrade from Biginsights 4.2.5, 4.2.0 to HDP 2.6.2

Re: Review Request 61129: AMBARI-21573. Kafka service failed to start during regenerate keytab after upgrade from Biginsights 4.2.5, 4.2.0 to HDP 2.6.2

Review Request 61129: AMBARI-21573. Kafka service failed to start during regenerate keytab after upgrade from Biginsights 4.2.5, 4.2.0 to HDP 2.6.2

Re: Review Request 61107: Migrate custom extension support

Re: Review Request 61107: Migrate custom extension support

Re: Review Request 61107: Migrate custom extension support

Re: Review Request 61102: AMBARI-21568. Pig service check fails after upgrade

Re: Review Request 61102: AMBARI-21568. Pig service check fails after upgrade

Re: Review Request 61102: AMBARI-21568. Pig service check fails after upgrade

Re: Review Request 61083: AMBARI-21563 : Copy Hadoop core-site.xml to Atlas conf when hadoop is configured for LDAP/AD.

Re: Review Request 61105: Auto populate zookeeper.connect in Ranger Kafka plugin properties

Re: Review Request 61083: AMBARI-21563 : Copy Hadoop core-site.xml to Atlas conf when hadoop is configured for LDAP/AD.

Re: Review Request 61107: Migrate custom extension support

Re: Review Request 60403: Add Kerberos HTTP SPNEGO authentication support to Ambari Metrics Monitor

Re: Review Request 61102: AMBARI-21568. Pig service check fails after upgrade

Re: Review Request 61107: Migrate custom extension support

Re: Review Request 61107: Migrate custom extension support

Review Request 61107: Migrate custom extension support

Re: Review Request 61079: AMBARI-21559. Alert definition paths should be updated on upgrade

Review Request 61105: Auto populate zookeeper.connect in Ranger Kafka plugin properties

Re: Review Request 61079: AMBARI-21559. Alert definition paths should be updated on upgrade

Re: Review Request 61074: Restart due to Stale configurations leads to an internal system exception

Re: Review Request 61074: Restart due to Stale configurations leads to an internal system exception

Review Request 61102: AMBARI-21568. Pig service check fails after upgrade

Re: Review Request 61083: AMBARI-21563 : Copy Hadoop core-site.xml to Atlas conf when hadoop is configured for LDAP/AD.

26 matches

Site Navigation

Mail list logo

Footer information