date:20160727

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50539/
---

Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, Jonathan
Hurley, Nate Cole, Sumit Mohanty, and Tim Thorpe.

Bugs: AMBARI-17935
https://issues.apache.org/jira/browse/AMBARI-17935

Repository: ambari

Description
---

STR
1. Deploy Ambari 2.2.2 with HDP 2.4 and Sqoop
2. Kerberize the cluster
3. Upgrade Ambari to 2.4.0
4. Install bits for HDP 2.5 and attempt to perform Express Upgrade

Observed error during Express Upgrade
```
Failed on: Updating configuration sqoop-atlas-application.properties

Server action failed
```

This happens because the sqoop-atlas-application.properties config in HDP 2.5
hasn't been created yet since it doesn't actually have any configs. To fix it,
add some configs so the merging process creates the new config type.

Diffs
-

ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/ConfigureAction.java
f7de8a9

ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
133db26

ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml
d648638

ambari-server/src/main/resources/stacks/HDP/2.5/services/SQOOP/configuration/sqoop-atlas-application.properties.xml
PRE-CREATION

Diff: https://reviews.apache.org/r/50539/diff/

Testing
---

Verified during EU from HDP 2.4 to 2.5 with Sqoop.

Python unit tests passed,

--
Total run:1034
Total errors:0
Total failures:0
OK

Thanks,

Alejandro Fernandez

Re: Review Request 50512: Spark and Spark2 should use different keytab files to avoid ACL issues


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50512/
---

(Updated July 27, 2016, 6:10 p.m.)


Review request for Ambari, bikassaha, Saisai Shao, and Sumit Mohanty.


Bugs: AMBARI-17921
https://issues.apache.org/jira/browse/AMBARI-17921


Repository: ambari


Description
---

If both Spark and Spark2 is installed and each run as a different user, then 
the ACLs on the _shared_ keytab files may block access by components in either 
service to needed keytab files. 

For example if Spark is set to run as the user with username `spark` and Spark2 
is set to run as the user with username `spark2`:
```
spark-env/spark_user = spark
spark2-env/spark_user = spark2
```

Then the keytab file for the shared headless principal - spark.headless.keytab 
- will have an ACL set that either the spark or the spark2 user can read it 
(depending on the order the keytab file is written). 

In this case, the following error will be encountered 

```
Traceback (most recent call last):
  File 
"/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/spark_thrift_server.py",
 line 87, in 
SparkThriftServer().execute()
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
 line 280, in execute
method(env)
  File 
"/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/spark_thrift_server.py",
 line 54, in start
spark_service('sparkthriftserver', upgrade_type=upgrade_type, 
action='start')
  File 
"/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/spark_service.py",
 line 57, in spark_service
Execute(spark_kinit_cmd, user=params.spark_user)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", 
line 155, in __init__
self.env.run()
  File 
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
line 160, in run
self.run_action(resource, action)
  File 
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
line 124, in run_action
provider_action()
  File 
"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
 line 273, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
line 71, in inner
result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
line 93, in checked_call
tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
line 141, in _call_wrapper
result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
line 294, in _call
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of '/usr/bin/kinit -kt 
/etc/security/keytabs/spark.headless.keytab 
spark2rndygi0zfoo3ftqildwn5...@hwqe.hortonworks.com; ' returned 1.  
Hortonworks #
This is MOTD message, added for testing in qe infra
kinit: Generic preauthentication failure while getting initial credentials
```

"kinit: Generic preauthentication failure while getting initial credentials" 
indicates, in this case, the the user running the Spark service does not have 
access to the specified keytab file.

To ensure this does not happen, keytab files for both services should have 
different file names.


Diffs (updated)
-

  ambari-server/src/main/resources/common-services/SPARK2/2.0.0/kerberos.json 
967adb0 

Diff: https://reviews.apache.org/r/50512/diff/


Testing
---

Manualy tested


Thanks,

Robert Levas

Re: Review Request 50531: AMBARI-17931. Enable log4j compression for HiveMetastore, HiveServer2 and Hive Server Interactive.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50531/#review143824
---


Ship it!




Ship It!

- Sumit Mohanty


On July 27, 2016, 8:27 p.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50531/
> ---
> 
> (Updated July 27, 2016, 8:27 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Sumit Mohanty, and Sushanth 
> Sowmyan.
> 
> 
> Bugs: AMBARI-17931
> https://issues.apache.org/jira/browse/AMBARI-17931
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Enable log4j compression for HiveMetastore, HiveServer2 and Hive Server 
> Interactive.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-log4j.xml
>  c3575fb 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-log4j2.xml
>  1337ee6 
> 
> Diff: https://reviews.apache.org/r/50531/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Swapan Shridhar
> 
>

Review Request 50532: [Preview] Disable INFO logs from HadoopMetrics2Reporter for hive on upgrade to 2.5

2016-07-27 Thread Aravindan Vijayan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50532/
---

Review request for Ambari, Alejandro Fernandez, Dmytro Grinenko, Jonathan 
Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.


Repository: ambari


Description
---

Added support for "Appending" a value to a config through Upgrade, for tackling 
the following config change.


In hive-log4j content, the following needs to be added.

# Silence HadoopMetrics2Reporter INFO logs
log4j.logger.com.github.joshelser.dropwizard.metrics.hadoop=WARN,DRFA


This specific config change is valid for only HDP-2.5 . Hence the task has been 
added for HDP 2.3, 2.4 upgrade packs.

A similar problem is being discussed in https://reviews.apache.org/r/50508.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/ConfigureAction.java
 f7de8a9 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ConfigUpgradeChangeDefinition.java
 54431eb 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ConfigureTask.java
 343c484 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
c49e18e 
  
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
 133db26 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
ff5d4d9 
  ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml 
473d8a0 
  
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml
 d648638 
  ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml 
e67aebb 
  
ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/ConfigureActionTest.java
 be0aeef 
  
ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java
 55cb23b 
  ambari-server/src/test/resources/stacks/HDP/2.1.1/upgrades/config-upgrade.xml 
44f9e02 

Diff: https://reviews.apache.org/r/50532/diff/


Testing
---

Added unit tests.

Manual Testing pending.


Thanks,

Aravindan Vijayan

Re: Review Request 50531: AMBARI-17931. Enable log4j compression for HiveMetastore, HiveServer2 and Hive Server Interactive.

2016-07-27 Thread Sushanth Sowmyan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50531/#review143806
---


Ship it!




Looks good to me, +1

- Sushanth Sowmyan


On July 27, 2016, 8:27 p.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50531/
> ---
> 
> (Updated July 27, 2016, 8:27 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Sumit Mohanty, and Sushanth 
> Sowmyan.
> 
> 
> Bugs: AMBARI-17931
> https://issues.apache.org/jira/browse/AMBARI-17931
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Enable log4j compression for HiveMetastore, HiveServer2 and Hive Server 
> Interactive.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-log4j.xml
>  c3575fb 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-log4j2.xml
>  1337ee6 
> 
> Diff: https://reviews.apache.org/r/50531/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Swapan Shridhar
> 
>

Review Request 50531: AMBARI-17931. Enable log4j compression for HiveMetastore, HiveServer2 and Hive Server Interactive.

2016-07-27 Thread Swapan Shridhar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50531/
---

Review request for Ambari, Alejandro Fernandez, Sumit Mohanty, and Sushanth 
Sowmyan.


Bugs: AMBARI-17931
https://issues.apache.org/jira/browse/AMBARI-17931


Repository: ambari


Description
---

Enable log4j compression for HiveMetastore, HiveServer2 and Hive Server 
Interactive.


Diffs
-

  
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-log4j.xml
 c3575fb 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-log4j2.xml
 1337ee6 

Diff: https://reviews.apache.org/r/50531/diff/


Testing
---


Thanks,

Swapan Shridhar

Re: Review Request 50512: Spark and Spark2 should use different keytab files to avoid ACL issues

2016-07-27 Thread bikas


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50512/#review143801
---




ambari-server/src/main/resources/common-services/SPARK2/2.0.0/kerberos.json 
(line 15)


Should this also be spark2-env/spark_user like its in the next section. 
What if Spark is not installed so there is no spark-env


- bikassaha


On July 27, 2016, 10:10 a.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50512/
> ---
> 
> (Updated July 27, 2016, 10:10 a.m.)
> 
> 
> Review request for Ambari, bikassaha, Saisai Shao, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17921
> https://issues.apache.org/jira/browse/AMBARI-17921
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> If both Spark and Spark2 is installed and each run as a different user, then 
> the ACLs on the _shared_ keytab files may block access by components in 
> either service to needed keytab files. 
> 
> For example if Spark is set to run as the user with username `spark` and 
> Spark2 is set to run as the user with username `spark2`:
> ```
> spark-env/spark_user = spark
> spark2-env/spark_user = spark2
> ```
> 
> Then the keytab file for the shared headless principal - 
> spark.headless.keytab - will have an ACL set that either the spark or the 
> spark2 user can read it (depending on the order the keytab file is written). 
> 
> In this case, the following error will be encountered 
> 
> ```
> Traceback (most recent call last):
>   File 
> "/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/spark_thrift_server.py",
>  line 87, in 
> SparkThriftServer().execute()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 280, in execute
> method(env)
>   File 
> "/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/spark_thrift_server.py",
>  line 54, in start
> spark_service('sparkthriftserver', upgrade_type=upgrade_type, 
> action='start')
>   File 
> "/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/spark_service.py",
>  line 57, in spark_service
> Execute(spark_kinit_cmd, user=params.spark_user)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", 
> line 155, in __init__
> self.env.run()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 160, in run
> self.run_action(resource, action)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 124, in run_action
> provider_action()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
>  line 273, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
> line 71, in inner
> result = function(command, **kwargs)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
> line 93, in checked_call
> tries=tries, try_sleep=try_sleep)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
> line 141, in _call_wrapper
> result = _call(command, **kwargs_copy)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
> line 294, in _call
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of '/usr/bin/kinit -kt 
> /etc/security/keytabs/spark.headless.keytab 
> spark2rndygi0zfoo3ftqildwn5...@hwqe.hortonworks.com; ' returned 1.  
> Hortonworks #
> This is MOTD message, added for testing in qe infra
> kinit: Generic preauthentication failure while getting initial credentials
> ```
> 
> "kinit: Generic preauthentication failure while getting initial credentials" 
> indicates, in this case, the the user running the Spark service does not have 
> access to the specified keytab file.
> 
> To ensure this does not happen, keytab files for both services should have 
> different file names.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/common-services/SPARK2/2.0.0/kerberos.json 
> 967adb0 
> 
> Diff: https://reviews.apache.org/r/50512/diff/
> 
> 
> Testing
> ---
> 
> Manualy tested
> 
> 
> Thanks,
> 
> Robert Levas
> 
>

Re: Review Request 50529: Kafka brokers went down after Ambari upgrade due to IllegalArgumentException

2016-07-27 Thread Sriharsha Chintalapani


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50529/#review143796
---


Ship it!




Ship It!

- Sriharsha Chintalapani


On July 27, 2016, 7:40 p.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50529/
> ---
> 
> (Updated July 27, 2016, 7:40 p.m.)
> 
> 
> Review request for Ambari, Robert Levas, Sumit Mohanty, and Sriharsha 
> Chintalapani.
> 
> 
> Bugs: AMBARI-17929
> https://issues.apache.org/jira/browse/AMBARI-17929
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> *Steps*
> # Deploy HDP-2.4.2 cluster with Ambari 2.2.2.0
> # Upgrade Ambari to 2.4.0.0
> # Observe the status of Kafka brokers
> 
> *Result*
> All brokers report down
> Logs indicate below:
> {code}
> [2016-07-27 05:48:26,535] INFO Initializing Kafka Timeline Metrics Sink 
> (org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter)
> [2016-07-27 05:48:26,571] INFO Started Kafka Timeline metrics reporter with 
> polling period 10 seconds 
> (org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter)
> [2016-07-27 05:48:26,716] INFO KafkaConfig values:
>   request.timeout.ms = 3
>   log.roll.hours = 168
>   inter.broker.protocol.version = 0.9.0.X
>   log.preallocate = false
>   security.inter.broker.protocol = PLAINTEXTSASL
>   controller.socket.timeout.ms = 3
>   broker.id.generation.enable = true
>   ssl.keymanager.algorithm = SunX509
>   ssl.key.password = [hidden]
>   log.cleaner.enable = true
>   ssl.provider = null
>   num.recovery.threads.per.data.dir = 1
>   background.threads = 10
>   unclean.leader.election.enable = true
>   sasl.kerberos.kinit.cmd = /usr/bin/kinit
>   replica.lag.time.max.ms = 1
>   ssl.endpoint.identification.algorithm = null
>   auto.create.topics.enable = true
>   zookeeper.sync.time.ms = 2000
>   ssl.client.auth = none
>   ssl.keystore.password = [hidden]
>   log.cleaner.io.buffer.load.factor = 0.9
>   offsets.topic.compression.codec = 0
>   log.retention.hours = 168
>   log.dirs = /kafka-logs
>   ssl.protocol = TLS
>   log.index.size.max.bytes = 10485760
>   sasl.kerberos.min.time.before.relogin = 6
>   log.retention.minutes = null
>   connections.max.idle.ms = 60
>   ssl.trustmanager.algorithm = PKIX
>   offsets.retention.minutes = 8640
>   max.connections.per.ip = 2147483647
>   replica.fetch.wait.max.ms = 500
>   metrics.num.samples = 2
>   port = 6667
>   offsets.retention.check.interval.ms = 60
>   log.cleaner.dedupe.buffer.size = 134217728
>   log.segment.bytes = 1073741824
>   group.min.session.timeout.ms = 6000
>   producer.purgatory.purge.interval.requests = 1
>   min.insync.replicas = 1
>   ssl.truststore.password = [hidden]
>   log.flush.scheduler.interval.ms = 9223372036854775807
>   socket.receive.buffer.bytes = 102400
>   leader.imbalance.per.broker.percentage = 10
>   num.io.threads = 8
>   zookeeper.connect = 
> nats11-36-alzs-dgm10toeriedwngdha-s11-3.openstacklocal:2181,nats11-36-alzs-dgm10toeriedwngdha-s11-4.openstacklocal:2181,nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:2181
>   queued.max.requests = 500
>   offsets.topic.replication.factor = 3
>   replica.socket.timeout.ms = 3
>   offsets.topic.segment.bytes = 104857600
>   replica.high.watermark.checkpoint.interval.ms = 5000
>   broker.id = -1
>   ssl.keystore.location = /etc/security/serverKeys/keystore.jks
>   listeners = 
> PLAINTEXT://nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:6667,SSL://nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:
>   log.flush.interval.messages = 9223372036854775807
>   principal.builder.class = class 
> org.apache.kafka.common.security.auth.DefaultPrincipalBuilder
>   log.retention.ms = null
>   offsets.commit.required.acks = -1
>   sasl.kerberos.principal.to.local.rules = [DEFAULT]
>   group.max.session.timeout.ms = 3
>   num.replica.fetchers = 1
>   advertised.listeners = 
> PLAINTEXT://nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:6667,SSL://nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:
>   replica.socket.receive.buffer.bytes = 65536
>   delete.topic.enable = false
>   log.index.interval.bytes = 4096
>   metric.reporters = []
>   compression.type = producer
>   log.cleanup.policy = delete
>   controlled.shutdown.max.retries = 3
>   log.cleaner.threads = 1
>   quota.window.size.seconds = 1
>   zookeeper.connection.timeout.ms = 25000
>

Re: Review Request 50529: Kafka brokers went down after Ambari upgrade due to IllegalArgumentException


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50529/#review143794
---


Ship it!




Ship It!


ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
 (line 1906)


kafkaBrocker --> kafkaBroker  (small nit)


- Robert Levas


On July 27, 2016, 3:40 p.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50529/
> ---
> 
> (Updated July 27, 2016, 3:40 p.m.)
> 
> 
> Review request for Ambari, Robert Levas, Sumit Mohanty, and Sriharsha 
> Chintalapani.
> 
> 
> Bugs: AMBARI-17929
> https://issues.apache.org/jira/browse/AMBARI-17929
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> *Steps*
> # Deploy HDP-2.4.2 cluster with Ambari 2.2.2.0
> # Upgrade Ambari to 2.4.0.0
> # Observe the status of Kafka brokers
> 
> *Result*
> All brokers report down
> Logs indicate below:
> {code}
> [2016-07-27 05:48:26,535] INFO Initializing Kafka Timeline Metrics Sink 
> (org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter)
> [2016-07-27 05:48:26,571] INFO Started Kafka Timeline metrics reporter with 
> polling period 10 seconds 
> (org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter)
> [2016-07-27 05:48:26,716] INFO KafkaConfig values:
>   request.timeout.ms = 3
>   log.roll.hours = 168
>   inter.broker.protocol.version = 0.9.0.X
>   log.preallocate = false
>   security.inter.broker.protocol = PLAINTEXTSASL
>   controller.socket.timeout.ms = 3
>   broker.id.generation.enable = true
>   ssl.keymanager.algorithm = SunX509
>   ssl.key.password = [hidden]
>   log.cleaner.enable = true
>   ssl.provider = null
>   num.recovery.threads.per.data.dir = 1
>   background.threads = 10
>   unclean.leader.election.enable = true
>   sasl.kerberos.kinit.cmd = /usr/bin/kinit
>   replica.lag.time.max.ms = 1
>   ssl.endpoint.identification.algorithm = null
>   auto.create.topics.enable = true
>   zookeeper.sync.time.ms = 2000
>   ssl.client.auth = none
>   ssl.keystore.password = [hidden]
>   log.cleaner.io.buffer.load.factor = 0.9
>   offsets.topic.compression.codec = 0
>   log.retention.hours = 168
>   log.dirs = /kafka-logs
>   ssl.protocol = TLS
>   log.index.size.max.bytes = 10485760
>   sasl.kerberos.min.time.before.relogin = 6
>   log.retention.minutes = null
>   connections.max.idle.ms = 60
>   ssl.trustmanager.algorithm = PKIX
>   offsets.retention.minutes = 8640
>   max.connections.per.ip = 2147483647
>   replica.fetch.wait.max.ms = 500
>   metrics.num.samples = 2
>   port = 6667
>   offsets.retention.check.interval.ms = 60
>   log.cleaner.dedupe.buffer.size = 134217728
>   log.segment.bytes = 1073741824
>   group.min.session.timeout.ms = 6000
>   producer.purgatory.purge.interval.requests = 1
>   min.insync.replicas = 1
>   ssl.truststore.password = [hidden]
>   log.flush.scheduler.interval.ms = 9223372036854775807
>   socket.receive.buffer.bytes = 102400
>   leader.imbalance.per.broker.percentage = 10
>   num.io.threads = 8
>   zookeeper.connect = 
> nats11-36-alzs-dgm10toeriedwngdha-s11-3.openstacklocal:2181,nats11-36-alzs-dgm10toeriedwngdha-s11-4.openstacklocal:2181,nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:2181
>   queued.max.requests = 500
>   offsets.topic.replication.factor = 3
>   replica.socket.timeout.ms = 3
>   offsets.topic.segment.bytes = 104857600
>   replica.high.watermark.checkpoint.interval.ms = 5000
>   broker.id = -1
>   ssl.keystore.location = /etc/security/serverKeys/keystore.jks
>   listeners = 
> PLAINTEXT://nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:6667,SSL://nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:
>   log.flush.interval.messages = 9223372036854775807
>   principal.builder.class = class 
> org.apache.kafka.common.security.auth.DefaultPrincipalBuilder
>   log.retention.ms = null
>   offsets.commit.required.acks = -1
>   sasl.kerberos.principal.to.local.rules = [DEFAULT]
>   group.max.session.timeout.ms = 3
>   num.replica.fetchers = 1
>   advertised.listeners = 
> PLAINTEXT://nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:6667,SSL://nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:
>   replica.socket.receive.buffer.bytes = 65536
>   delete.topic.enable = false
>   log.index.interval.bytes = 4096
>   metric.reporters = []
>   compression.type = producer
>

Re: Review Request 50526: NameNode High Availability Health Alert Issue

2016-07-27 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50526/#review143789
---




ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/alerts/alert_ha_namenode_health.py
 (line 195)


Looks like now, alert results will come from active, inactive and unknown 
nodes at the same time and not only from active? 

Not sure, if jxm query could be executed fine from all hosts which means 
that result wcould be different from each host. How they would be aggregated 
then?


- Dmytro Grinenko


On July 27, 2016, 7:06 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50526/
> ---
> 
> (Updated July 27, 2016, 7:06 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmitro Lisnichenko, Nate 
> Cole, and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-17928
> https://issues.apache.org/jira/browse/AMBARI-17928
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Name Node High Availability Health alert "response" message is mentioned as 
> "Another host will report this alert".
> 
> STR:
> - Setup NameNode HA
> - Shutdown the standby NameNode
> 
> The text will flip between:
> {{Active['c6402.ambari.apache.org:50070'], Standby[], 
> Unknown['c6401.ambari.apache.org:50070']}}
> and
> {{Another host will report this alert}}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/alerts/alert_ha_namenode_health.py
>  00d1421 
> 
> Diff: https://reviews.apache.org/r/50526/diff/
> 
> 
> Testing
> ---
> 
> Live testing on a cluster.
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 50500: Request for HBase metrics making a call to Namenode and Nimbus for that metric

2016-07-27 Thread Sid Wagle


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50500/#review143788
---


Ship it!




Ship It!

- Sid Wagle


On July 27, 2016, 11:44 a.m., Dmytro Sen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50500/
> ---
> 
> (Updated July 27, 2016, 11:44 a.m.)
> 
> 
> Review request for Ambari, Aravindan Vijayan, Myroslav Papirkovskyy, and Sid 
> Wagle.
> 
> 
> Bugs: AMBARI-17914
> https://issues.apache.org/jira/browse/AMBARI-17914
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> While investigating a slow API call on perf cluster found that we are 
> requesting wrong metrics from AMS which wastes a network round trip and will 
> not have any data.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/metrics/timeline/AMSPropertyProvider.java
>  7ad1192 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/metrics/timeline/AMSPropertyProviderTest.java
>  291512a 
> 
> Diff: https://reviews.apache.org/r/50500/diff/
> 
> 
> Testing
> ---
> 
> Unit tests passed
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Review Request 50529: Kafka brokers went down after Ambari upgrade due to IllegalArgumentException

2016-07-27 Thread Vitalyi Brodetskyi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50529/
---

Review request for Ambari, Robert Levas and Sumit Mohanty.


Bugs: AMBARI-17929
https://issues.apache.org/jira/browse/AMBARI-17929


Repository: ambari


Description
---

*Steps*
# Deploy HDP-2.4.2 cluster with Ambari 2.2.2.0
# Upgrade Ambari to 2.4.0.0
# Observe the status of Kafka brokers

*Result*
All brokers report down
Logs indicate below:
{code}
[2016-07-27 05:48:26,535] INFO Initializing Kafka Timeline Metrics Sink 
(org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter)
[2016-07-27 05:48:26,571] INFO Started Kafka Timeline metrics reporter with 
polling period 10 seconds 
(org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter)
[2016-07-27 05:48:26,716] INFO KafkaConfig values:
request.timeout.ms = 3
log.roll.hours = 168
inter.broker.protocol.version = 0.9.0.X
log.preallocate = false
security.inter.broker.protocol = PLAINTEXTSASL
controller.socket.timeout.ms = 3
broker.id.generation.enable = true
ssl.keymanager.algorithm = SunX509
ssl.key.password = [hidden]
log.cleaner.enable = true
ssl.provider = null
num.recovery.threads.per.data.dir = 1
background.threads = 10
unclean.leader.election.enable = true
sasl.kerberos.kinit.cmd = /usr/bin/kinit
replica.lag.time.max.ms = 1
ssl.endpoint.identification.algorithm = null
auto.create.topics.enable = true
zookeeper.sync.time.ms = 2000
ssl.client.auth = none
ssl.keystore.password = [hidden]
log.cleaner.io.buffer.load.factor = 0.9
offsets.topic.compression.codec = 0
log.retention.hours = 168
log.dirs = /kafka-logs
ssl.protocol = TLS
log.index.size.max.bytes = 10485760
sasl.kerberos.min.time.before.relogin = 6
log.retention.minutes = null
connections.max.idle.ms = 60
ssl.trustmanager.algorithm = PKIX
offsets.retention.minutes = 8640
max.connections.per.ip = 2147483647
replica.fetch.wait.max.ms = 500
metrics.num.samples = 2
port = 6667
offsets.retention.check.interval.ms = 60
log.cleaner.dedupe.buffer.size = 134217728
log.segment.bytes = 1073741824
group.min.session.timeout.ms = 6000
producer.purgatory.purge.interval.requests = 1
min.insync.replicas = 1
ssl.truststore.password = [hidden]
log.flush.scheduler.interval.ms = 9223372036854775807
socket.receive.buffer.bytes = 102400
leader.imbalance.per.broker.percentage = 10
num.io.threads = 8
zookeeper.connect = 
nats11-36-alzs-dgm10toeriedwngdha-s11-3.openstacklocal:2181,nats11-36-alzs-dgm10toeriedwngdha-s11-4.openstacklocal:2181,nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:2181
queued.max.requests = 500
offsets.topic.replication.factor = 3
replica.socket.timeout.ms = 3
offsets.topic.segment.bytes = 104857600
replica.high.watermark.checkpoint.interval.ms = 5000
broker.id = -1
ssl.keystore.location = /etc/security/serverKeys/keystore.jks
listeners = 
PLAINTEXT://nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:6667,SSL://nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:
log.flush.interval.messages = 9223372036854775807
principal.builder.class = class 
org.apache.kafka.common.security.auth.DefaultPrincipalBuilder
log.retention.ms = null
offsets.commit.required.acks = -1
sasl.kerberos.principal.to.local.rules = [DEFAULT]
group.max.session.timeout.ms = 3
num.replica.fetchers = 1
advertised.listeners = 
PLAINTEXT://nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:6667,SSL://nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:
replica.socket.receive.buffer.bytes = 65536
delete.topic.enable = false
log.index.interval.bytes = 4096
metric.reporters = []
compression.type = producer
log.cleanup.policy = delete
controlled.shutdown.max.retries = 3
log.cleaner.threads = 1
quota.window.size.seconds = 1
zookeeper.connection.timeout.ms = 25000
offsets.load.buffer.size = 5242880
zookeeper.session.timeout.ms = 3
ssl.cipher.suites = null
authorizer.class.name = 
org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.service.name = null
controlled.shutdown.enable = true
offsets.topic.num.partitions = 50
quota.window.num = 11
message.max.bytes = 100

Re: Review Request 50526: NameNode High Availability Health Alert Issue

2016-07-27 Thread Nate Cole


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50526/#review143785
---


Ship it!




Ship It!

- Nate Cole


On July 27, 2016, 3:06 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50526/
> ---
> 
> (Updated July 27, 2016, 3:06 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmitro Lisnichenko, Nate 
> Cole, and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-17928
> https://issues.apache.org/jira/browse/AMBARI-17928
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Name Node High Availability Health alert "response" message is mentioned as 
> "Another host will report this alert".
> 
> STR:
> - Setup NameNode HA
> - Shutdown the standby NameNode
> 
> The text will flip between:
> {{Active['c6402.ambari.apache.org:50070'], Standby[], 
> Unknown['c6401.ambari.apache.org:50070']}}
> and
> {{Another host will report this alert}}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/alerts/alert_ha_namenode_health.py
>  00d1421 
> 
> Diff: https://reviews.apache.org/r/50526/diff/
> 
> 
> Testing
> ---
> 
> Live testing on a cluster.
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 50476: UI does not allow to delete service is some host components are in INSTALLED_FAILED state

2016-07-27 Thread Jaimin Jetly


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50476/
---

(Updated July 27, 2016, 7:07 p.m.)


Review request for Ambari, Zhe (Joe) Wang, Sumit Mohanty, and Yusaku Sako.


Bugs: AMBARI-17841
https://issues.apache.org/jira/browse/AMBARI-17841


Repository: ambari


Description
---

*STR:*
# Deploy a service ad abort installation in the middle in a way such that some 
of its slave host components remains in INSTALLED_FAILED state but no all
# Try deleting service

*Expected Result:* UI should allow service deletion as API allows it in this 
scenario
*Actual Result:* UI shows a warning popup in green color and does not let user 
delete a service. screenshot attached


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/ServiceComponentResponse.java
 cb84b89 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ComponentResourceProvider.java
 cc13e39 
  
ambari-server/src/main/java/org/apache/ambari/server/state/ServiceComponentImpl.java
 9283e38 
  ambari-server/src/main/resources/properties.json f471628 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ComponentResourceProviderTest.java
 cbf3044 
  
ambari-server/src/test/java/org/apache/ambari/server/state/ServiceComponentTest.java
 22cf302 
  ambari-web/app/controllers/global/update_controller.js e179a2a 
  ambari-web/app/controllers/main/service/item.js ce0b240 
  ambari-web/app/mappers/components_state_mapper.js 3ad8e49 
  ambari-web/app/models/client_component.js 3902227 

Diff: https://reviews.apache.org/r/50476/diff/


Testing (updated)
---

Tested the patch on deployed cluster.
Verified that ambari-web unit tests passes with the patch

  29224 tests complete (35 seconds)
  154 tests pending

Jenkins QA Job is stuck. So ran ambari-server unit tests and verified that the 
patch does not break any unit tests


Thanks,

Jaimin Jetly

Re: Review Request 50526: NameNode High Availability Health Alert Issue

2016-07-27 Thread Vitalyi Brodetskyi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50526/#review143781
---


Ship it!




Ship It!

- Vitalyi Brodetskyi


On Липень 27, 2016, 6:59 після полудня, Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50526/
> ---
> 
> (Updated Липень 27, 2016, 6:59 після полудня)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmitro Lisnichenko, and Nate 
> Cole.
> 
> 
> Bugs: AMBARI-17928
> https://issues.apache.org/jira/browse/AMBARI-17928
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Name Node High Availability Health alert "response" message is mentioned as 
> "Another host will report this alert".
> 
> STR:
> - Setup NameNode HA
> - Shutdown the standby NameNode
> 
> The text will flip between:
> {{Active['c6402.ambari.apache.org:50070'], Standby[], 
> Unknown['c6401.ambari.apache.org:50070']}}
> and
> {{Another host will report this alert}}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/alerts/alert_ha_namenode_health.py
>  00d1421 
> 
> Diff: https://reviews.apache.org/r/50526/diff/
> 
> 
> Testing
> ---
> 
> Live testing on a cluster.
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Review Request 50526: NameNode High Availability Health Alert Issue

2016-07-27 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50526/
---

Review request for Ambari, Alejandro Fernandez, Dmitro Lisnichenko, and Nate 
Cole.


Bugs: AMBARI-17928
https://issues.apache.org/jira/browse/AMBARI-17928


Repository: ambari


Description
---

Name Node High Availability Health alert "response" message is mentioned as 
"Another host will report this alert".

STR:
- Setup NameNode HA
- Shutdown the standby NameNode

The text will flip between:
{{Active['c6402.ambari.apache.org:50070'], Standby[], 
Unknown['c6401.ambari.apache.org:50070']}}
and
{{Another host will report this alert}}


Diffs
-

  
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/alerts/alert_ha_namenode_health.py
 00d1421 

Diff: https://reviews.apache.org/r/50526/diff/


Testing
---

Live testing on a cluster.


Thanks,

Jonathan Hurley

Re: Review Request 50512: Spark and Spark2 should use different keytab files to avoid ACL issues



> On July 27, 2016, 2:35 p.m., bikassaha wrote:
> > I think the Spark and Spark2 design assumes the same keytab and user for 
> > Spark and Spark2 so that the transition between the two and concurrent use 
> > of both is seamless. So HDFS files, e.g for ATS data from both apps should 
> > be owned by the same user.

Even though the keytab files are different, the principal embedded in them are 
the same. Therefore the authenticated user is the same and this both behave as 
the same user.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50512/#review143771
---


On July 27, 2016, 1:10 p.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50512/
> ---
> 
> (Updated July 27, 2016, 1:10 p.m.)
> 
> 
> Review request for Ambari, bikassaha, Saisai Shao, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17921
> https://issues.apache.org/jira/browse/AMBARI-17921
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> If both Spark and Spark2 is installed and each run as a different user, then 
> the ACLs on the _shared_ keytab files may block access by components in 
> either service to needed keytab files. 
> 
> For example if Spark is set to run as the user with username `spark` and 
> Spark2 is set to run as the user with username `spark2`:
> ```
> spark-env/spark_user = spark
> spark2-env/spark_user = spark2
> ```
> 
> Then the keytab file for the shared headless principal - 
> spark.headless.keytab - will have an ACL set that either the spark or the 
> spark2 user can read it (depending on the order the keytab file is written). 
> 
> In this case, the following error will be encountered 
> 
> ```
> Traceback (most recent call last):
>   File 
> "/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/spark_thrift_server.py",
>  line 87, in 
> SparkThriftServer().execute()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 280, in execute
> method(env)
>   File 
> "/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/spark_thrift_server.py",
>  line 54, in start
> spark_service('sparkthriftserver', upgrade_type=upgrade_type, 
> action='start')
>   File 
> "/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/spark_service.py",
>  line 57, in spark_service
> Execute(spark_kinit_cmd, user=params.spark_user)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", 
> line 155, in __init__
> self.env.run()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 160, in run
> self.run_action(resource, action)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 124, in run_action
> provider_action()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
>  line 273, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
> line 71, in inner
> result = function(command, **kwargs)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
> line 93, in checked_call
> tries=tries, try_sleep=try_sleep)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
> line 141, in _call_wrapper
> result = _call(command, **kwargs_copy)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
> line 294, in _call
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of '/usr/bin/kinit -kt 
> /etc/security/keytabs/spark.headless.keytab 
> spark2rndygi0zfoo3ftqildwn5...@hwqe.hortonworks.com; ' returned 1.  
> Hortonworks #
> This is MOTD message, added for testing in qe infra
> kinit: Generic preauthentication failure while getting initial credentials
> ```
> 
> "kinit: Generic preauthentication failure while getting initial credentials" 
> indicates, in this case, the the user running the Spark service does not have 
> access to the specified keytab file.
> 
> To ensure this does not happen, keytab files for both services should have 
> different file names.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/common-services/SPARK2/2.0.0/kerberos.json 
> 967adb0 
> 
> Diff: https://reviews.apache.org/r/50512/diff/
> 
> 
> Testing
> ---
> 
> Manualy tested
> 
> 
> Thanks,
> 
> Robert Levas
> 
>

Re: Review Request 50445: AMBARI-17902 Config changes to support external solr and internal solr for Ranger


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50445/#review143779
---


Ship it!




Ship It!

- Robert Levas


On July 27, 2016, 5:06 a.m., Mugdha Varadkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50445/
> ---
> 
> (Updated July 27, 2016, 5:06 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jaimin Jetly, 
> Jayush Luniya, Robert Levas, Srimanth Gunturi, and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-17902
> https://issues.apache.org/jira/browse/AMBARI-17902
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Ranger Service needs to support the following two scenarios in case when 
> audit to solr is enabled and solrCloud is used as destination.
> 
> External Solr
> If Audit to Solr is Enabled and Solr Cloud == true and Kerberos is Enabled
>  If is_external_solr == true
>   If is_external_solr_kerberized == true
>Then recommend ranger.is.solr.kerberised as true.
> 
> 
> Ambari Internal Solr
> If Audit to Solr is Enabled and Solr Cloud == true
>   If is_external_solr == false and Kerberos is Enabled
>Then directly recommend ranger.is.solr.kerberised as true.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/kerberos.json 
> 3d6e25c 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py
>  3ec4b53 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py
>  1670d69 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/templates/ranger_solr_jaas_conf.j2
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/templates/ranger_solr_jass_conf.j2
>  a456688 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.6.0/configuration/ranger-admin-site.xml
>  eacf541 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.6.0/configuration/ranger-env.xml
>  2cf3539 
>   ambari-server/src/main/resources/common-services/RANGER/0.6.0/kerberos.json 
> ffebb11 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.6.0/themes/theme_version_3.json
>  e65c9b2 
>   
> ambari-server/src/main/resources/common-services/STORM/1.0.1/configuration/ranger-storm-audit.xml
>  1c869ed 
>   ambari-server/src/main/resources/common-services/STORM/1.0.1/kerberos.json 
> f9fa30d 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> c49e18e 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  133db26 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> ff5d4d9 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml 
> 473d8a0 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml
>  d648638 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml 
> e67aebb 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/ranger-atlas-audit.xml
>  ac22729 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HBASE/configuration/ranger-hbase-audit.xml
>  cc9f0d2 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HBASE/kerberos.json 
> ada02ad 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/configuration/ranger-hdfs-audit.xml
>  0a04953 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/kerberos.json 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/ranger-hive-audit.xml
>  671c08e 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/kerberos.json 
> f9a0caf 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/KAFKA/configuration/ranger-kafka-audit.xml
>  6aca7e7 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/KAFKA/kerberos.json 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/KNOX/configuration/ranger-knox-audit.xml
>  bdd1994 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/KNOX/kerberos.json 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/RANGER_KMS/configuration/ranger-kms-audit.xml
>  8c8278a 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/RANGER_KMS/kerberos.json
>  bfd142a 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/ranger-yarn-audit.xml
>  da24576 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/kerberos.json 
> 38896f5 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> c46a168 
>

Re: Review Request 50512: Spark and Spark2 should use different keytab files to avoid ACL issues

2016-07-27 Thread bikas


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50512/#review143771
---



I think the Spark and Spark2 design assumes the same keytab and user for Spark 
and Spark2 so that the transition between the two and concurrent use of both is 
seamless. So HDFS files, e.g for ATS data from both apps should be owned by the 
same user.

- bikassaha


On July 27, 2016, 10:10 a.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50512/
> ---
> 
> (Updated July 27, 2016, 10:10 a.m.)
> 
> 
> Review request for Ambari, bikassaha, Saisai Shao, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17921
> https://issues.apache.org/jira/browse/AMBARI-17921
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> If both Spark and Spark2 is installed and each run as a different user, then 
> the ACLs on the _shared_ keytab files may block access by components in 
> either service to needed keytab files. 
> 
> For example if Spark is set to run as the user with username `spark` and 
> Spark2 is set to run as the user with username `spark2`:
> ```
> spark-env/spark_user = spark
> spark2-env/spark_user = spark2
> ```
> 
> Then the keytab file for the shared headless principal - 
> spark.headless.keytab - will have an ACL set that either the spark or the 
> spark2 user can read it (depending on the order the keytab file is written). 
> 
> In this case, the following error will be encountered 
> 
> ```
> Traceback (most recent call last):
>   File 
> "/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/spark_thrift_server.py",
>  line 87, in 
> SparkThriftServer().execute()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 280, in execute
> method(env)
>   File 
> "/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/spark_thrift_server.py",
>  line 54, in start
> spark_service('sparkthriftserver', upgrade_type=upgrade_type, 
> action='start')
>   File 
> "/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/spark_service.py",
>  line 57, in spark_service
> Execute(spark_kinit_cmd, user=params.spark_user)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", 
> line 155, in __init__
> self.env.run()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 160, in run
> self.run_action(resource, action)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 124, in run_action
> provider_action()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
>  line 273, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
> line 71, in inner
> result = function(command, **kwargs)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
> line 93, in checked_call
> tries=tries, try_sleep=try_sleep)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
> line 141, in _call_wrapper
> result = _call(command, **kwargs_copy)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
> line 294, in _call
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of '/usr/bin/kinit -kt 
> /etc/security/keytabs/spark.headless.keytab 
> spark2rndygi0zfoo3ftqildwn5...@hwqe.hortonworks.com; ' returned 1.  
> Hortonworks #
> This is MOTD message, added for testing in qe infra
> kinit: Generic preauthentication failure while getting initial credentials
> ```
> 
> "kinit: Generic preauthentication failure while getting initial credentials" 
> indicates, in this case, the the user running the Spark service does not have 
> access to the specified keytab file.
> 
> To ensure this does not happen, keytab files for both services should have 
> different file names.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/common-services/SPARK2/2.0.0/kerberos.json 
> 967adb0 
> 
> Diff: https://reviews.apache.org/r/50512/diff/
> 
> 
> Testing
> ---
> 
> Manualy tested
> 
> 
> Thanks,
> 
> Robert Levas
> 
>

Re: Review Request 50431: AMBARI-17896: Fix falcon hook classname and classpath for atlas



> On July 27, 2016, 6:37 p.m., Alejandro Fernandez wrote:
> > Ship It!

Pushed to trunk, commit f428a6b97a52f6dee2e1cf7c78327770fd119ae6
branch-2.4, commit c91d18cac42bacab5d5a3c7745a6339172e2


- Alejandro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50431/#review143774
---


On July 27, 2016, 1:07 a.m., Venkat Ranganathan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50431/
> ---
> 
> (Updated July 27, 2016, 1:07 a.m.)
> 
> 
> Review request for Ambari and Alejandro Fernandez.
> 
> 
> Bugs: AMBARI-17896
> https://issues.apache.org/jira/browse/AMBARI-17896
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fixed stack advisor for 2.5 and also the param_linux for setting the right 
> conf path
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py
>  ebe3634 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 8d5cdc9 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> f8ba3e9 
> 
> Diff: https://reviews.apache.org/r/50431/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Venkat Ranganathan
> 
>

Re: Review Request 50431: AMBARI-17896: Fix falcon hook classname and classpath for atlas


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50431/#review143772
---




ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py (line 
742)


This is hasattr instead of self.hasattr.
I'll fix it and commit it for you.


- Alejandro Fernandez


On July 27, 2016, 1:07 a.m., Venkat Ranganathan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50431/
> ---
> 
> (Updated July 27, 2016, 1:07 a.m.)
> 
> 
> Review request for Ambari and Alejandro Fernandez.
> 
> 
> Bugs: AMBARI-17896
> https://issues.apache.org/jira/browse/AMBARI-17896
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fixed stack advisor for 2.5 and also the param_linux for setting the right 
> conf path
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py
>  ebe3634 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 8d5cdc9 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> f8ba3e9 
> 
> Diff: https://reviews.apache.org/r/50431/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Venkat Ranganathan
> 
>

Re: Review Request 50431: AMBARI-17896: Fix falcon hook classname and classpath for atlas


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50431/#review143774
---


Ship it!




Ship It!

- Alejandro Fernandez


On July 27, 2016, 1:07 a.m., Venkat Ranganathan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50431/
> ---
> 
> (Updated July 27, 2016, 1:07 a.m.)
> 
> 
> Review request for Ambari and Alejandro Fernandez.
> 
> 
> Bugs: AMBARI-17896
> https://issues.apache.org/jira/browse/AMBARI-17896
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Fixed stack advisor for 2.5 and also the param_linux for setting the right 
> conf path
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py
>  ebe3634 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 8d5cdc9 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> f8ba3e9 
> 
> Diff: https://reviews.apache.org/r/50431/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Venkat Ranganathan
> 
>

Re: Review Request 50479: AMBARI-17909 AMS Storm Sink: apply change of Storm metrics improvement - worker level aggregation

2016-07-27 Thread Sriharsha Chintalapani


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50479/#review143768
---


Ship it!




Ship It!

- Sriharsha Chintalapani


On July 27, 2016, 2:52 a.m., Jungtaek Lim wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50479/
> ---
> 
> (Updated July 27, 2016, 2:52 a.m.)
> 
> 
> Review request for Ambari, Aravindan Vijayan, Sriharsha Chintalapani, and Sid 
> Wagle.
> 
> 
> Bugs: AMBARI-17909
> https://issues.apache.org/jira/browse/AMBARI-17909
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Abstraction:
> This issue is for following up changes on STORM-2006: Storm metrics feature 
> improvement: support per-worker level metrics aggregation.
> 
> Details:
> With recent patches for AMS and AMS Storm sink, AMS stores task level metrics 
> from Storm, and relevant dashboards can be configured via Grafana.
> But we found that it incurs too many kinds of metrics and also too many data 
> points published to AMS because even a topology can have lots of tasks.
> In order to reduce this, I addressed STORM-2006, but it also needs AMS Storm 
> sink and relevant Storm configurations to be changed.
> 
> Note:
> We're moving some properties so STORM-2006 must be go on together in order to 
> make metrics filter working properly. But AMS Storm sink itself is backward 
> compatible.
> I don't modify legacy AMS Storm sink since STORM-2006 will not be ported back 
> to 0.10.x or lower.
> 
> 
> Diffs
> -
> 
>   
> ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java
>  3a4289b 
>   
> ambari-metrics/ambari-metrics-storm-sink/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java
>  fadb00c 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/storm-site.xml
>  280fc42 
> 
> Diff: https://reviews.apache.org/r/50479/diff/
> 
> 
> Testing
> ---
> 
> - Build succeed (build passed with 2.4 branch without Ignore annotation)
> - Tested manually with STORM-2006 pre-applied cluster
> 
> 
> Thanks,
> 
> Jungtaek Lim
> 
>

Re: Review Request 50512: Spark and Spark2 should use different keytab files to avoid ACL issues


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50512/#review143765
---


Ship it!




Ship It!

- Jayush Luniya


On July 27, 2016, 5:10 p.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50512/
> ---
> 
> (Updated July 27, 2016, 5:10 p.m.)
> 
> 
> Review request for Ambari, bikassaha, Saisai Shao, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17921
> https://issues.apache.org/jira/browse/AMBARI-17921
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> If both Spark and Spark2 is installed and each run as a different user, then 
> the ACLs on the _shared_ keytab files may block access by components in 
> either service to needed keytab files. 
> 
> For example if Spark is set to run as the user with username `spark` and 
> Spark2 is set to run as the user with username `spark2`:
> ```
> spark-env/spark_user = spark
> spark2-env/spark_user = spark2
> ```
> 
> Then the keytab file for the shared headless principal - 
> spark.headless.keytab - will have an ACL set that either the spark or the 
> spark2 user can read it (depending on the order the keytab file is written). 
> 
> In this case, the following error will be encountered 
> 
> ```
> Traceback (most recent call last):
>   File 
> "/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/spark_thrift_server.py",
>  line 87, in 
> SparkThriftServer().execute()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 280, in execute
> method(env)
>   File 
> "/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/spark_thrift_server.py",
>  line 54, in start
> spark_service('sparkthriftserver', upgrade_type=upgrade_type, 
> action='start')
>   File 
> "/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/spark_service.py",
>  line 57, in spark_service
> Execute(spark_kinit_cmd, user=params.spark_user)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", 
> line 155, in __init__
> self.env.run()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 160, in run
> self.run_action(resource, action)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 124, in run_action
> provider_action()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
>  line 273, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
> line 71, in inner
> result = function(command, **kwargs)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
> line 93, in checked_call
> tries=tries, try_sleep=try_sleep)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
> line 141, in _call_wrapper
> result = _call(command, **kwargs_copy)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
> line 294, in _call
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of '/usr/bin/kinit -kt 
> /etc/security/keytabs/spark.headless.keytab 
> spark2rndygi0zfoo3ftqildwn5...@hwqe.hortonworks.com; ' returned 1.  
> Hortonworks #
> This is MOTD message, added for testing in qe infra
> kinit: Generic preauthentication failure while getting initial credentials
> ```
> 
> "kinit: Generic preauthentication failure while getting initial credentials" 
> indicates, in this case, the the user running the Spark service does not have 
> access to the specified keytab file.
> 
> To ensure this does not happen, keytab files for both services should have 
> different file names.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/common-services/SPARK2/2.0.0/kerberos.json 
> 967adb0 
> 
> Diff: https://reviews.apache.org/r/50512/diff/
> 
> 
> Testing
> ---
> 
> Manualy tested
> 
> 
> Thanks,
> 
> Robert Levas
> 
>

Re: Review Request 50512: Spark and Spark2 should use different keytab files to avoid ACL issues


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50512/#review143758
---


Ship it!




Ship It!

- Sumit Mohanty


On July 27, 2016, 5:10 p.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50512/
> ---
> 
> (Updated July 27, 2016, 5:10 p.m.)
> 
> 
> Review request for Ambari, bikassaha, Saisai Shao, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17921
> https://issues.apache.org/jira/browse/AMBARI-17921
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> If both Spark and Spark2 is installed and each run as a different user, then 
> the ACLs on the _shared_ keytab files may block access by components in 
> either service to needed keytab files. 
> 
> For example if Spark is set to run as the user with username `spark` and 
> Spark2 is set to run as the user with username `spark2`:
> ```
> spark-env/spark_user = spark
> spark2-env/spark_user = spark2
> ```
> 
> Then the keytab file for the shared headless principal - 
> spark.headless.keytab - will have an ACL set that either the spark or the 
> spark2 user can read it (depending on the order the keytab file is written). 
> 
> In this case, the following error will be encountered 
> 
> ```
> Traceback (most recent call last):
>   File 
> "/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/spark_thrift_server.py",
>  line 87, in 
> SparkThriftServer().execute()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 280, in execute
> method(env)
>   File 
> "/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/spark_thrift_server.py",
>  line 54, in start
> spark_service('sparkthriftserver', upgrade_type=upgrade_type, 
> action='start')
>   File 
> "/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/spark_service.py",
>  line 57, in spark_service
> Execute(spark_kinit_cmd, user=params.spark_user)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", 
> line 155, in __init__
> self.env.run()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 160, in run
> self.run_action(resource, action)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 124, in run_action
> provider_action()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
>  line 273, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
> line 71, in inner
> result = function(command, **kwargs)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
> line 93, in checked_call
> tries=tries, try_sleep=try_sleep)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
> line 141, in _call_wrapper
> result = _call(command, **kwargs_copy)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
> line 294, in _call
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of '/usr/bin/kinit -kt 
> /etc/security/keytabs/spark.headless.keytab 
> spark2rndygi0zfoo3ftqildwn5...@hwqe.hortonworks.com; ' returned 1.  
> Hortonworks #
> This is MOTD message, added for testing in qe infra
> kinit: Generic preauthentication failure while getting initial credentials
> ```
> 
> "kinit: Generic preauthentication failure while getting initial credentials" 
> indicates, in this case, the the user running the Spark service does not have 
> access to the specified keytab file.
> 
> To ensure this does not happen, keytab files for both services should have 
> different file names.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/common-services/SPARK2/2.0.0/kerberos.json 
> 967adb0 
> 
> Diff: https://reviews.apache.org/r/50512/diff/
> 
> 
> Testing
> ---
> 
> Manualy tested
> 
> 
> Thanks,
> 
> Robert Levas
> 
>

Review Request 50512: Spark and Spark2 should use different keytab files to avoid ACL issues


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50512/
---

Review request for Ambari, bikassaha, Saisai Shao, and Sumit Mohanty.


Bugs: AMBARI-17921
https://issues.apache.org/jira/browse/AMBARI-17921


Repository: ambari


Description
---

If both Spark and Spark2 is installed and each run as a different user, then 
the ACLs on the _shared_ keytab files may block access by components in either 
service to needed keytab files. 

For example if Spark is set to run as the user with username `spark` and Spark2 
is set to run as the user with username `spark2`:
```
spark-env/spark_user = spark
spark2-env/spark_user = spark2
```

Then the keytab file for the shared headless principal - spark.headless.keytab 
- will have an ACL set that either the spark or the spark2 user can read it 
(depending on the order the keytab file is written). 

In this case, the following error will be encountered 

```
Traceback (most recent call last):
  File 
"/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/spark_thrift_server.py",
 line 87, in 
SparkThriftServer().execute()
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
 line 280, in execute
method(env)
  File 
"/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/spark_thrift_server.py",
 line 54, in start
spark_service('sparkthriftserver', upgrade_type=upgrade_type, 
action='start')
  File 
"/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/spark_service.py",
 line 57, in spark_service
Execute(spark_kinit_cmd, user=params.spark_user)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", 
line 155, in __init__
self.env.run()
  File 
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
line 160, in run
self.run_action(resource, action)
  File 
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
line 124, in run_action
provider_action()
  File 
"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
 line 273, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
line 71, in inner
result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
line 93, in checked_call
tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
line 141, in _call_wrapper
result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", 
line 294, in _call
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of '/usr/bin/kinit -kt 
/etc/security/keytabs/spark.headless.keytab 
spark2rndygi0zfoo3ftqildwn5...@hwqe.hortonworks.com; ' returned 1.  
Hortonworks #
This is MOTD message, added for testing in qe infra
kinit: Generic preauthentication failure while getting initial credentials
```

"kinit: Generic preauthentication failure while getting initial credentials" 
indicates, in this case, the the user running the Spark service does not have 
access to the specified keytab file.

To ensure this does not happen, keytab files for both services should have 
different file names.


Diffs
-

  ambari-server/src/main/resources/common-services/SPARK2/2.0.0/kerberos.json 
967adb0 

Diff: https://reviews.apache.org/r/50512/diff/


Testing
---

Manualy tested


Thanks,

Robert Levas

Re: Review Request 50445: AMBARI-17902 Config changes to support external solr and internal solr for Ranger



> On July 26, 2016, 3:35 p.m., Robert Levas wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/kerberos.json,
> >  line 1
> > 
> >
> > Is this change specific to HDP 2.5 or could some other stack benefit 
> > from the changes.  For example PHD X.Y?
> > 
> > If so, then maybe a new common service version of HDFS should be 
> > created and referenced from HDP/2.5/HDFS.
> 
> Mugdha Varadkar wrote:
> Till now this is specific to HDP only. @Jayush can you please take the 
> decision on the same.

We can leave HDFS as is. There are plans to refactor all services after Ambari 
2.4.0.0


> On July 26, 2016, 3:35 p.m., Robert Levas wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.5/services/KAFKA/kerberos.json,
> >  line 1
> > 
> >
> > Is this change specific to HDP 2.5 or could some other stack benefit 
> > from the changes.  For example PHD X.Y?
> > 
> > If so, then maybe a new common service version of KAFKA should be 
> > created and referenced from HDP/2.5/KAFKA.
> 
> Mugdha Varadkar wrote:
> I am not sure whether we have privilege to take that decision on adding 
> new common version for KAFKA. @Jayush can you please let use know if we can 
> add common service version of KAFKA

We actually need to create a common-services/KAFKA/0.10.0 and have 
HDP/2.5/services/KAFKA inherit from it. I also see 
ambari/ambari-server/src/main/resources/stacks/HDP/2.5/services/KAFKA/configuration/ranger-kafka-audit.xml
 which needs to be moved. Go ahead with your changes and file a separate bug 
for this, I will do the refactoring changes in it.

FYI, KAFKA, STORM, RANGER, ZOOKEEPER services have been refactored and 
versioned in common-services and HDP should simply inherit from the 
common-services version and only override specific properties.


> On July 26, 2016, 3:35 p.m., Robert Levas wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.5/services/KNOX/kerberos.json,
> >  line 1
> > 
> >
> > Is this change specific to HDP 2.5 or could some other stack benefit 
> > from the changes.  For example PHD X.Y?
> > 
> > If so, then maybe a new common service version of KNOX should be 
> > created and referenced from HDP/2.5/KNOX.
> 
> Mugdha Varadkar wrote:
> Till now this is specific to HDP only. @Jayush can you please take the 
> decision on the same.

This is ok for now


- Jayush


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50445/#review143535
---


On July 27, 2016, 9:06 a.m., Mugdha Varadkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50445/
> ---
> 
> (Updated July 27, 2016, 9:06 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jaimin Jetly, 
> Jayush Luniya, Robert Levas, Srimanth Gunturi, and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-17902
> https://issues.apache.org/jira/browse/AMBARI-17902
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Ranger Service needs to support the following two scenarios in case when 
> audit to solr is enabled and solrCloud is used as destination.
> 
> External Solr
> If Audit to Solr is Enabled and Solr Cloud == true and Kerberos is Enabled
>  If is_external_solr == true
>   If is_external_solr_kerberized == true
>Then recommend ranger.is.solr.kerberised as true.
> 
> 
> Ambari Internal Solr
> If Audit to Solr is Enabled and Solr Cloud == true
>   If is_external_solr == false and Kerberos is Enabled
>Then directly recommend ranger.is.solr.kerberised as true.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/kerberos.json 
> 3d6e25c 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py
>  3ec4b53 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py
>  1670d69 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/templates/ranger_solr_jaas_conf.j2
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/templates/ranger_solr_jass_conf.j2
>  a456688 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.6.0/configuration/ranger-admin-site.xml
>  eacf541 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.6.0/configuration/ranger-env.xml
>  2cf3539 
>   ambari-server/src/main/resources/common-services/RANGER/0.6.0/kerberos.json 
> ffebb11 
>   
>

Re: Review Request 50047: Kafka listeners property does not show SASL_PLAINTEXT protocol when Kerberos is enabled

2016-07-27 Thread Jonathan Hurley



> On July 26, 2016, 4:41 p.m., Robert Levas wrote:
> > Committed to trunk
> > ```
> > commit 124f48ef899fddb6bdb96ebea9aa3a6a1a6adbca
> > Author: Anita Jebaraj 
> > Date:   Tue Jul 26 16:25:30 2016 -0400
> > ```
> > 
> > Committed to branch-2.4
> > ```
> > commit eb4f14b6bcb0208cc3d3764bbd31e2125687c517
> > Author: Anita Jebaraj 
> > Date:   Tue Jul 26 16:38:55 2016 -0400
> > ```
> 
> Anita Jebaraj wrote:
> Thank you Robert

Please close out this review, Anita.


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50047/#review143608
---


On July 26, 2016, 2:15 p.m., Anita Jebaraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50047/
> ---
> 
> (Updated July 26, 2016, 2:15 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Di Li, Jonathan Hurley, Nate 
> Cole, and Robert Levas.
> 
> 
> Bugs: AMBARI-17694
> https://issues.apache.org/jira/browse/AMBARI-17694
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When kerberos is enabled, the protocol for listeners in 
> /etc/kafka/conf/server.properties is updated from PLAINTEXT to PLAINTEXTSASL, 
> even though the Ambari UI shows otherwise
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/VariableReplacementHelper.java
>  66be3bf 
>   
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/kafka.py
>  ac7b0ae 
>   ambari-server/src/main/resources/common-services/KAFKA/0.9.0/kerberos.json 
> 2b1c01b 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/kerberos/VariableReplacementHelperTest.java
>  ee2a671 
> 
> Diff: https://reviews.apache.org/r/50047/diff/
> 
> 
> Testing
> ---
> 
> Added 1 new test case,
>  Ran mvn test
> 
> 
> Thanks,
> 
> Anita Jebaraj
> 
>

Re: Review Request 50445: AMBARI-17902 Config changes to support external solr and internal solr for Ranger



> On July 26, 2016, 11:35 a.m., Robert Levas wrote:
> > ambari-server/src/main/resources/common-services/RANGER/0.6.0/kerberos.json,
> >  lines 18-22
> > 
> >
> > it seems like these should be set in a Jinja2 template rather than make 
> > the properties here.
> > 
> > Is it expected that a user might change them?
> 
> Mugdha Varadkar wrote:
> Ranger Code doesn't support jaas.conf file. That is why can't use jinja2 
> template.

This data seems to be rather static and we run the risk of users changing it 
via the UI or API.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50445/#review143535
---


On July 27, 2016, 5:06 a.m., Mugdha Varadkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50445/
> ---
> 
> (Updated July 27, 2016, 5:06 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jaimin Jetly, 
> Jayush Luniya, Robert Levas, Srimanth Gunturi, and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-17902
> https://issues.apache.org/jira/browse/AMBARI-17902
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Ranger Service needs to support the following two scenarios in case when 
> audit to solr is enabled and solrCloud is used as destination.
> 
> External Solr
> If Audit to Solr is Enabled and Solr Cloud == true and Kerberos is Enabled
>  If is_external_solr == true
>   If is_external_solr_kerberized == true
>Then recommend ranger.is.solr.kerberised as true.
> 
> 
> Ambari Internal Solr
> If Audit to Solr is Enabled and Solr Cloud == true
>   If is_external_solr == false and Kerberos is Enabled
>Then directly recommend ranger.is.solr.kerberised as true.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/kerberos.json 
> 3d6e25c 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py
>  3ec4b53 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py
>  1670d69 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/templates/ranger_solr_jaas_conf.j2
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/templates/ranger_solr_jass_conf.j2
>  a456688 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.6.0/configuration/ranger-admin-site.xml
>  eacf541 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.6.0/configuration/ranger-env.xml
>  2cf3539 
>   ambari-server/src/main/resources/common-services/RANGER/0.6.0/kerberos.json 
> ffebb11 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.6.0/themes/theme_version_3.json
>  e65c9b2 
>   
> ambari-server/src/main/resources/common-services/STORM/1.0.1/configuration/ranger-storm-audit.xml
>  1c869ed 
>   ambari-server/src/main/resources/common-services/STORM/1.0.1/kerberos.json 
> f9fa30d 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> c49e18e 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  133db26 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> ff5d4d9 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml 
> 473d8a0 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml
>  d648638 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml 
> e67aebb 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/ranger-atlas-audit.xml
>  ac22729 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HBASE/configuration/ranger-hbase-audit.xml
>  cc9f0d2 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HBASE/kerberos.json 
> ada02ad 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/configuration/ranger-hdfs-audit.xml
>  0a04953 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/kerberos.json 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/ranger-hive-audit.xml
>  671c08e 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/kerberos.json 
> f9a0caf 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/KAFKA/configuration/ranger-kafka-audit.xml
>  6aca7e7 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/KAFKA/kerberos.json 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/KNOX/configuration/ranger-knox-audit.xml
>  bdd1994 
>

Re: Review Request 50445: AMBARI-17902 Config changes to support external solr and internal solr for Ranger


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50445/#review143746
---


Ship it!




Ship It!

- Jayush Luniya


On July 27, 2016, 9:06 a.m., Mugdha Varadkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50445/
> ---
> 
> (Updated July 27, 2016, 9:06 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jaimin Jetly, 
> Jayush Luniya, Robert Levas, Srimanth Gunturi, and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-17902
> https://issues.apache.org/jira/browse/AMBARI-17902
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Ranger Service needs to support the following two scenarios in case when 
> audit to solr is enabled and solrCloud is used as destination.
> 
> External Solr
> If Audit to Solr is Enabled and Solr Cloud == true and Kerberos is Enabled
>  If is_external_solr == true
>   If is_external_solr_kerberized == true
>Then recommend ranger.is.solr.kerberised as true.
> 
> 
> Ambari Internal Solr
> If Audit to Solr is Enabled and Solr Cloud == true
>   If is_external_solr == false and Kerberos is Enabled
>Then directly recommend ranger.is.solr.kerberised as true.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/kerberos.json 
> 3d6e25c 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py
>  3ec4b53 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py
>  1670d69 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/templates/ranger_solr_jaas_conf.j2
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/templates/ranger_solr_jass_conf.j2
>  a456688 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.6.0/configuration/ranger-admin-site.xml
>  eacf541 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.6.0/configuration/ranger-env.xml
>  2cf3539 
>   ambari-server/src/main/resources/common-services/RANGER/0.6.0/kerberos.json 
> ffebb11 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.6.0/themes/theme_version_3.json
>  e65c9b2 
>   
> ambari-server/src/main/resources/common-services/STORM/1.0.1/configuration/ranger-storm-audit.xml
>  1c869ed 
>   ambari-server/src/main/resources/common-services/STORM/1.0.1/kerberos.json 
> f9fa30d 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> c49e18e 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  133db26 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> ff5d4d9 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml 
> 473d8a0 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml
>  d648638 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml 
> e67aebb 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/ranger-atlas-audit.xml
>  ac22729 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HBASE/configuration/ranger-hbase-audit.xml
>  cc9f0d2 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HBASE/kerberos.json 
> ada02ad 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/configuration/ranger-hdfs-audit.xml
>  0a04953 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/kerberos.json 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/ranger-hive-audit.xml
>  671c08e 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/kerberos.json 
> f9a0caf 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/KAFKA/configuration/ranger-kafka-audit.xml
>  6aca7e7 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/KAFKA/kerberos.json 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/KNOX/configuration/ranger-knox-audit.xml
>  bdd1994 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/KNOX/kerberos.json 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/RANGER_KMS/configuration/ranger-kms-audit.xml
>  8c8278a 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/RANGER_KMS/kerberos.json
>  bfd142a 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/ranger-yarn-audit.xml
>  da24576 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/kerberos.json 
> 38896f5 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> c46a168 
>

Re: Review Request 50445: AMBARI-17902 Config changes to support external solr and internal solr for Ranger


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50445/#review143745
---




ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py
 (line 325)


Merge the 2 if-blocks


- Jayush Luniya


On July 27, 2016, 9:06 a.m., Mugdha Varadkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50445/
> ---
> 
> (Updated July 27, 2016, 9:06 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jaimin Jetly, 
> Jayush Luniya, Robert Levas, Srimanth Gunturi, and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-17902
> https://issues.apache.org/jira/browse/AMBARI-17902
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Ranger Service needs to support the following two scenarios in case when 
> audit to solr is enabled and solrCloud is used as destination.
> 
> External Solr
> If Audit to Solr is Enabled and Solr Cloud == true and Kerberos is Enabled
>  If is_external_solr == true
>   If is_external_solr_kerberized == true
>Then recommend ranger.is.solr.kerberised as true.
> 
> 
> Ambari Internal Solr
> If Audit to Solr is Enabled and Solr Cloud == true
>   If is_external_solr == false and Kerberos is Enabled
>Then directly recommend ranger.is.solr.kerberised as true.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/kerberos.json 
> 3d6e25c 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py
>  3ec4b53 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py
>  1670d69 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/templates/ranger_solr_jaas_conf.j2
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/templates/ranger_solr_jass_conf.j2
>  a456688 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.6.0/configuration/ranger-admin-site.xml
>  eacf541 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.6.0/configuration/ranger-env.xml
>  2cf3539 
>   ambari-server/src/main/resources/common-services/RANGER/0.6.0/kerberos.json 
> ffebb11 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.6.0/themes/theme_version_3.json
>  e65c9b2 
>   
> ambari-server/src/main/resources/common-services/STORM/1.0.1/configuration/ranger-storm-audit.xml
>  1c869ed 
>   ambari-server/src/main/resources/common-services/STORM/1.0.1/kerberos.json 
> f9fa30d 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> c49e18e 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  133db26 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> ff5d4d9 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml 
> 473d8a0 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml
>  d648638 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml 
> e67aebb 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/ranger-atlas-audit.xml
>  ac22729 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HBASE/configuration/ranger-hbase-audit.xml
>  cc9f0d2 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HBASE/kerberos.json 
> ada02ad 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/configuration/ranger-hdfs-audit.xml
>  0a04953 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/kerberos.json 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/ranger-hive-audit.xml
>  671c08e 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/kerberos.json 
> f9a0caf 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/KAFKA/configuration/ranger-kafka-audit.xml
>  6aca7e7 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/KAFKA/kerberos.json 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/KNOX/configuration/ranger-knox-audit.xml
>  bdd1994 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/KNOX/kerberos.json 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/RANGER_KMS/configuration/ranger-kms-audit.xml
>  8c8278a 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/RANGER_KMS/kerberos.json
>  bfd142a 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/ranger-yarn-audit.xml
>  da24576 
>

Re: Review Request 50506: HDFS Service check failure on wireencrypted clusters:Ambari2400+HDP-2.4.2.0


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50506/#review143743
---


Ship it!




Ship It!

- Sumit Mohanty


On July 27, 2016, 3:15 p.m., Dmytro Sen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50506/
> ---
> 
> (Updated July 27, 2016, 3:15 p.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko, Sumit Mohanty, and Vitalyi 
> Brodetskyi.
> 
> 
> Bugs: AMBARI-17923
> https://issues.apache.org/jira/browse/AMBARI-17923
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Deployed a secure+ HA enabled cluster with Ambari2400 and HDP2420.
> After enabling wireencryption seems like HDFS service check is failing.
> hadoop_2_5_0_0_1081-client.x86_64 should be installed on every NN.
> Now this package is installed only as a dependency for
> 
>  datafu_2_4_2_0_258 falcon_2_4_2_0_258 hadoop_2_4_2_0_258-client 
> hive_2_4_2_0_258 hive_2_4_2_0_258-hcatalog hive_2_4_2_0_258-jdbc 
> hive_2_4_2_0_258-webhcat mahout mahout_2_4_2_0_258 oozie_2_4_2_0_258 
> pig_2_4_2_0_258 spark_2_4_2_0_258 spark_2_4_2_0_258-python sqoop_2_4_2_0_258
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 
> a6d216c 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/metainfo.xml 
> e0e4e5c 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/metainfo.xml 
> d4ff7b7 
> 
> Diff: https://reviews.apache.org/r/50506/diff/
> 
> 
> Testing
> ---
> 
> Unit tests passed
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Re: Review Request 50507: AMBARI-17886 : Disable INFO logs from HadoopMetrics2Reporter for hive


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50507/#review143741
---


Ship it!




Ship It!

- Sumit Mohanty


On July 27, 2016, 3:11 p.m., Aravindan Vijayan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50507/
> ---
> 
> (Updated July 27, 2016, 3:11 p.m.)
> 
> 
> Review request for Ambari, Dmytro Sen, Sumit Mohanty, Sushanth Sowmyan, and 
> Sid Wagle.
> 
> 
> Bugs: AMBARI-17886
> https://issues.apache.org/jira/browse/AMBARI-17886
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Out of disk space issues are seen due to the sheer amount of logs being 
> produced by hive after addition of metrics. For this, a single change needs 
> to be done to the log4j properties offered by Ambari to fix this.
> 
> The following needs to be added to the HDP 2.5 hive-log4j config.
> 
> # Silence HadoopMetrics2Reporter INFO logs
> log4j.logger.com.github.joshelser.dropwizard.metrics.hadoop=WARN,DRFA
> 
> Similar changes need to be made to the hive-log4j2 as well.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/hive-log4j.xml
>  c3575fb 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-log4j.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-log4j2.xml
>  edec332 
> 
> Diff: https://reviews.apache.org/r/50507/diff/
> 
> 
> Testing
> ---
> 
> Manually tested.
> 
> 
> Thanks,
> 
> Aravindan Vijayan
> 
>

Re: Review Request 50506: HDFS Service check failure on wireencrypted clusters:Ambari2400+HDP-2.4.2.0


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50506/
---

(Updated Июль 27, 2016, 3:15 п.п.)


Review request for Ambari, Dmitro Lisnichenko, Sumit Mohanty, and Vitalyi 
Brodetskyi.


Bugs: AMBARI-17923
https://issues.apache.org/jira/browse/AMBARI-17923


Repository: ambari


Description
---

Deployed a secure+ HA enabled cluster with Ambari2400 and HDP2420.
After enabling wireencryption seems like HDFS service check is failing.
hadoop_2_5_0_0_1081-client.x86_64 should be installed on every NN.
Now this package is installed only as a dependency for

 datafu_2_4_2_0_258 falcon_2_4_2_0_258 hadoop_2_4_2_0_258-client 
hive_2_4_2_0_258 hive_2_4_2_0_258-hcatalog hive_2_4_2_0_258-jdbc 
hive_2_4_2_0_258-webhcat mahout mahout_2_4_2_0_258 oozie_2_4_2_0_258 
pig_2_4_2_0_258 spark_2_4_2_0_258 spark_2_4_2_0_258-python sqoop_2_4_2_0_258


Diffs (updated)
-

  ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 
a6d216c 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/metainfo.xml 
e0e4e5c 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/metainfo.xml 
d4ff7b7 

Diff: https://reviews.apache.org/r/50506/diff/


Testing
---

Unit tests passed


Thanks,

Dmytro Sen

Re: Review Request 50506: HDFS Service check failure on wireencrypted clusters:Ambari2400+HDP-2.4.2.0



> On Июль 27, 2016, 3:10 п.п., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml,
> >  line 246
> > 
> >
> > Can we fix all occurrences? Two more.

oops. uploaded wrong patch here


- Dmytro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50506/#review143735
---


On Июль 27, 2016, 3:08 п.п., Dmytro Sen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50506/
> ---
> 
> (Updated Июль 27, 2016, 3:08 п.п.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko, Sumit Mohanty, and Vitalyi 
> Brodetskyi.
> 
> 
> Bugs: AMBARI-17923
> https://issues.apache.org/jira/browse/AMBARI-17923
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Deployed a secure+ HA enabled cluster with Ambari2400 and HDP2420.
> After enabling wireencryption seems like HDFS service check is failing.
> hadoop_2_5_0_0_1081-client.x86_64 should be installed on every NN.
> Now this package is installed only as a dependency for
> 
>  datafu_2_4_2_0_258 falcon_2_4_2_0_258 hadoop_2_4_2_0_258-client 
> hive_2_4_2_0_258 hive_2_4_2_0_258-hcatalog hive_2_4_2_0_258-jdbc 
> hive_2_4_2_0_258-webhcat mahout mahout_2_4_2_0_258 oozie_2_4_2_0_258 
> pig_2_4_2_0_258 spark_2_4_2_0_258 spark_2_4_2_0_258-python sqoop_2_4_2_0_258
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 
> a6d216c 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/metainfo.xml 
> e0e4e5c 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/metainfo.xml 
> d4ff7b7 
> 
> Diff: https://reviews.apache.org/r/50506/diff/
> 
> 
> Testing
> ---
> 
> Unit tests passed
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Re: Review Request 50506: HDFS Service check failure on wireencrypted clusters:Ambari2400+HDP-2.4.2.0


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50506/#review143735
---




ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 
(line 246)


Can we fix all occurrences? Two more.


- Sumit Mohanty


On July 27, 2016, 3:08 p.m., Dmytro Sen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50506/
> ---
> 
> (Updated July 27, 2016, 3:08 p.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko, Sumit Mohanty, and Vitalyi 
> Brodetskyi.
> 
> 
> Bugs: AMBARI-17923
> https://issues.apache.org/jira/browse/AMBARI-17923
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Deployed a secure+ HA enabled cluster with Ambari2400 and HDP2420.
> After enabling wireencryption seems like HDFS service check is failing.
> hadoop_2_5_0_0_1081-client.x86_64 should be installed on every NN.
> Now this package is installed only as a dependency for
> 
>  datafu_2_4_2_0_258 falcon_2_4_2_0_258 hadoop_2_4_2_0_258-client 
> hive_2_4_2_0_258 hive_2_4_2_0_258-hcatalog hive_2_4_2_0_258-jdbc 
> hive_2_4_2_0_258-webhcat mahout mahout_2_4_2_0_258 oozie_2_4_2_0_258 
> pig_2_4_2_0_258 spark_2_4_2_0_258 spark_2_4_2_0_258-python sqoop_2_4_2_0_258
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 
> a6d216c 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/metainfo.xml 
> e0e4e5c 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/metainfo.xml 
> d4ff7b7 
> 
> Diff: https://reviews.apache.org/r/50506/diff/
> 
> 
> Testing
> ---
> 
> Unit tests passed
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Re: Review Request 50506: HDFS Service check failure on wireencrypted clusters:Ambari2400+HDP-2.4.2.0


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50506/
---

(Updated Июль 27, 2016, 3:08 п.п.)


Review request for Ambari, Dmitro Lisnichenko, Sumit Mohanty, and Vitalyi 
Brodetskyi.


Bugs: AMBARI-17923
https://issues.apache.org/jira/browse/AMBARI-17923


Repository: ambari


Description
---

Deployed a secure+ HA enabled cluster with Ambari2400 and HDP2420.
After enabling wireencryption seems like HDFS service check is failing.
hadoop_2_5_0_0_1081-client.x86_64 should be installed on every NN.
Now this package is installed only as a dependency for

 datafu_2_4_2_0_258 falcon_2_4_2_0_258 hadoop_2_4_2_0_258-client 
hive_2_4_2_0_258 hive_2_4_2_0_258-hcatalog hive_2_4_2_0_258-jdbc 
hive_2_4_2_0_258-webhcat mahout mahout_2_4_2_0_258 oozie_2_4_2_0_258 
pig_2_4_2_0_258 spark_2_4_2_0_258 spark_2_4_2_0_258-python sqoop_2_4_2_0_258


Diffs (updated)
-

  ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 
a6d216c 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/metainfo.xml 
e0e4e5c 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/metainfo.xml 
d4ff7b7 

Diff: https://reviews.apache.org/r/50506/diff/


Testing
---

Unit tests passed


Thanks,

Dmytro Sen

Re: Review Request 50506: HDFS Service check failure on wireencrypted clusters:Ambari2400+HDP-2.4.2.0



> On Июль 27, 2016, 3:03 п.п., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml,
> >  line 218
> > 
> >
> > The common service definition should not have package specification 
> > with ${stack_version} in them as it still gets used for stacks without 
> > versioned packaging.

Thank you. Fixed.


- Dmytro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50506/#review143732
---


On Июль 27, 2016, 3:08 п.п., Dmytro Sen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50506/
> ---
> 
> (Updated Июль 27, 2016, 3:08 п.п.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko, Sumit Mohanty, and Vitalyi 
> Brodetskyi.
> 
> 
> Bugs: AMBARI-17923
> https://issues.apache.org/jira/browse/AMBARI-17923
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Deployed a secure+ HA enabled cluster with Ambari2400 and HDP2420.
> After enabling wireencryption seems like HDFS service check is failing.
> hadoop_2_5_0_0_1081-client.x86_64 should be installed on every NN.
> Now this package is installed only as a dependency for
> 
>  datafu_2_4_2_0_258 falcon_2_4_2_0_258 hadoop_2_4_2_0_258-client 
> hive_2_4_2_0_258 hive_2_4_2_0_258-hcatalog hive_2_4_2_0_258-jdbc 
> hive_2_4_2_0_258-webhcat mahout mahout_2_4_2_0_258 oozie_2_4_2_0_258 
> pig_2_4_2_0_258 spark_2_4_2_0_258 spark_2_4_2_0_258-python sqoop_2_4_2_0_258
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 
> a6d216c 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/metainfo.xml 
> e0e4e5c 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/metainfo.xml 
> d4ff7b7 
> 
> Diff: https://reviews.apache.org/r/50506/diff/
> 
> 
> Testing
> ---
> 
> Unit tests passed
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Re: Review Request 50506: HDFS Service check failure on wireencrypted clusters:Ambari2400+HDP-2.4.2.0


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50506/#review143732
---




ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 
(line 218)


The common service definition should not have package specification with 
${stack_version} in them as it still gets used for stacks without versioned 
packaging.


- Sumit Mohanty


On July 27, 2016, 3:01 p.m., Dmytro Sen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50506/
> ---
> 
> (Updated July 27, 2016, 3:01 p.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko, Sumit Mohanty, and Vitalyi 
> Brodetskyi.
> 
> 
> Bugs: AMBARI-17923
> https://issues.apache.org/jira/browse/AMBARI-17923
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Deployed a secure+ HA enabled cluster with Ambari2400 and HDP2420.
> After enabling wireencryption seems like HDFS service check is failing.
> hadoop_2_5_0_0_1081-client.x86_64 should be installed on every NN.
> Now this package is installed only as a dependency for
> 
>  datafu_2_4_2_0_258 falcon_2_4_2_0_258 hadoop_2_4_2_0_258-client 
> hive_2_4_2_0_258 hive_2_4_2_0_258-hcatalog hive_2_4_2_0_258-jdbc 
> hive_2_4_2_0_258-webhcat mahout mahout_2_4_2_0_258 oozie_2_4_2_0_258 
> pig_2_4_2_0_258 spark_2_4_2_0_258 spark_2_4_2_0_258-python sqoop_2_4_2_0_258
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 
> a6d216c 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/metainfo.xml 
> e0e4e5c 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/metainfo.xml 
> d4ff7b7 
> 
> Diff: https://reviews.apache.org/r/50506/diff/
> 
> 
> Testing
> ---
> 
> Unit tests passed
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Review Request 50506: HDFS Service check failure on wireencrypted clusters:Ambari2400+HDP-2.4.2.0


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50506/
---

Review request for Ambari, Dmitro Lisnichenko, Sumit Mohanty, and Vitalyi 
Brodetskyi.


Bugs: AMBARI-17923
https://issues.apache.org/jira/browse/AMBARI-17923


Repository: ambari


Description
---

Deployed a secure+ HA enabled cluster with Ambari2400 and HDP2420.
After enabling wireencryption seems like HDFS service check is failing.
hadoop_2_5_0_0_1081-client.x86_64 should be installed on every NN.
Now this package is installed only as a dependency for

 datafu_2_4_2_0_258 falcon_2_4_2_0_258 hadoop_2_4_2_0_258-client 
hive_2_4_2_0_258 hive_2_4_2_0_258-hcatalog hive_2_4_2_0_258-jdbc 
hive_2_4_2_0_258-webhcat mahout mahout_2_4_2_0_258 oozie_2_4_2_0_258 
pig_2_4_2_0_258 spark_2_4_2_0_258 spark_2_4_2_0_258-python sqoop_2_4_2_0_258


Diffs
-

  ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 
a6d216c 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/metainfo.xml 
e0e4e5c 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/metainfo.xml 
d4ff7b7 

Diff: https://reviews.apache.org/r/50506/diff/


Testing
---

Unit tests passed


Thanks,

Dmytro Sen

Review Request 50505: Coverity Scan Security Vulnerability - Resource Leak defects

2016-07-27 Thread Vitalyi Brodetskyi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50505/
---

Review request for Ambari, Dmitro Lisnichenko, Dmytro Sen, Robert Levas, and 
Sumit Mohanty.


Bugs: AMBARI-17922
https://issues.apache.org/jira/browse/AMBARI-17922


Repository: ambari


Description
---

Fix "Resource Leak" issues that were found by coverity scan.


Diffs
-

  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeeder.java
 8697f54 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeederUtil.java
 9881b55 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputFile.java
 b4d2bbb 
  
ambari-logsearch/ambari-logsearch-portal/src/main/java/org/apache/ambari/logsearch/LogSearch.java
 735a83a 
  
ambari-logsearch/ambari-logsearch-portal/src/main/java/org/apache/ambari/logsearch/manager/AuditMgr.java
 3dd8146 
  
ambari-logsearch/ambari-logsearch-portal/src/main/java/org/apache/ambari/logsearch/util/FileUtil.java
 ab52b06 
  
ambari-logsearch/ambari-logsearch-portal/src/main/java/org/apache/ambari/logsearch/util/PropertiesUtil.java
 f32152d 
  
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/AggregatorUtils.java
 9e41c87 
  
ambari-server/src/main/java/org/apache/ambari/server/checks/DatabaseConsistencyCheckHelper.java
 3035de9 
  
ambari-server/src/main/java/org/apache/ambari/server/view/ViewDirectoryWatcher.java
 c3d443a 
  ambari-server/src/main/java/org/apache/ambari/server/view/ViewRegistry.java 
83a3761 

Diff: https://reviews.apache.org/r/50505/diff/


Testing
---

In progress


Thanks,

Vitalyi Brodetskyi

Re: Review Request 50479: AMBARI-17909 AMS Storm Sink: apply change of Storm metrics improvement - worker level aggregation

2016-07-27 Thread Aravindan Vijayan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50479/#review143730
---


Ship it!




Ship It!

- Aravindan Vijayan


On July 27, 2016, 2:52 a.m., Jungtaek Lim wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50479/
> ---
> 
> (Updated July 27, 2016, 2:52 a.m.)
> 
> 
> Review request for Ambari, Aravindan Vijayan, Sriharsha Chintalapani, and Sid 
> Wagle.
> 
> 
> Bugs: AMBARI-17909
> https://issues.apache.org/jira/browse/AMBARI-17909
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Abstraction:
> This issue is for following up changes on STORM-2006: Storm metrics feature 
> improvement: support per-worker level metrics aggregation.
> 
> Details:
> With recent patches for AMS and AMS Storm sink, AMS stores task level metrics 
> from Storm, and relevant dashboards can be configured via Grafana.
> But we found that it incurs too many kinds of metrics and also too many data 
> points published to AMS because even a topology can have lots of tasks.
> In order to reduce this, I addressed STORM-2006, but it also needs AMS Storm 
> sink and relevant Storm configurations to be changed.
> 
> Note:
> We're moving some properties so STORM-2006 must be go on together in order to 
> make metrics filter working properly. But AMS Storm sink itself is backward 
> compatible.
> I don't modify legacy AMS Storm sink since STORM-2006 will not be ported back 
> to 0.10.x or lower.
> 
> 
> Diffs
> -
> 
>   
> ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java
>  3a4289b 
>   
> ambari-metrics/ambari-metrics-storm-sink/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java
>  fadb00c 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/storm-site.xml
>  280fc42 
> 
> Diff: https://reviews.apache.org/r/50479/diff/
> 
> 
> Testing
> ---
> 
> - Build succeed (build passed with 2.4 branch without Ignore annotation)
> - Tested manually with STORM-2006 pre-applied cluster
> 
> 
> Thanks,
> 
> Jungtaek Lim
> 
>

Re: Review Request 50489: Zeppelin install fails for non-root users due to wrong permissions

2016-07-27 Thread Rohit Choudhary


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50489/#review143728
---


Ship it!




Ship It!

- Rohit Choudhary


On July 27, 2016, 10:41 a.m., Renjith Kamath wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50489/
> ---
> 
> (Updated July 27, 2016, 10:41 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, DIPAYAN BHOWMICK, Gaurav 
> Nagar, Nitiraj Rathore, Pallav Kulshreshtha, Prabhjyot Singh, Rohit 
> Choudhary, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17913
> https://issues.apache.org/jira/browse/AMBARI-17913
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> - fix "Permission denied: '/etc/zeppelin/conf/interpreter.json'"
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/master.py
>  2de99b1 
> 
> Diff: https://reviews.apache.org/r/50489/diff/
> 
> 
> Testing
> ---
> 
> manually tested on ubuntu with non-root user
> 
> 
> Thanks,
> 
> Renjith Kamath
> 
>

Re: Review Request 50475: AMBARI-17778 Add usage information in ambari-server script

2016-07-27 Thread Nate Cole


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50475/#review143727
---


Ship it!




Ship It!

- Nate Cole


On July 26, 2016, 8:30 p.m., Masahiro Tanaka wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50475/
> ---
> 
> (Updated July 26, 2016, 8:30 p.m.)
> 
> 
> Review request for Ambari, Nate Cole, Sebastian Toader, Sid Wagle, and Yusaku 
> Sako.
> 
> 
> Bugs: AMBARI-17778
> https://issues.apache.org/jira/browse/AMBARI-17778
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> In ambari-server script, there are some lacks of usage information (e.g. 
> install-mpack, upgrade-mpack.)
> 
> They should be included in usage information.
> 
> 
> Diffs
> -
> 
>   ambari-server/sbin/ambari-server 81cf53b 
> 
> Diff: https://reviews.apache.org/r/50475/diff/
> 
> 
> Testing
> ---
> 
> ```
> --1 overall. Here are the results of testing the latest attachment
> http://issues.apache.org/jira/secure/attachment/12818681/AMBARI-17778.patch
> against trunk revision .
> 
> +1 @author. The patch does not contain any @author tags.
> 
> -1 tests included. The patch doesn't appear to include any new or modified 
> tests.
> Please justify why no new tests are needed for this patch.
> Also please list what manual steps were performed to verify this patch.
> 
> +1 javac. The applied patch does not increase the total number of javac 
> compiler warnings.
> 
> +1 release audit. The applied patch does not increase the total number of 
> release audit warnings.
> 
> +1 core tests. The patch passed unit tests in ambari-server.
> 
> Test results: 
> https://builds.apache.org/job/Ambari-trunk-test-patch/7913//testReport/
> Console output: 
> https://builds.apache.org/job/Ambari-trunk-test-patch/7913//console
> 
> This message is automatically generated.
> ```
> & manual test
> 
> 
> Thanks,
> 
> Masahiro Tanaka
> 
>

Re: Review Request 50489: Zeppelin install fails for non-root users due to wrong permissions

2016-07-27 Thread Nitiraj Rathore


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50489/#review143720
---


Ship it!




Ship It!

- Nitiraj Rathore


On July 27, 2016, 10:41 a.m., Renjith Kamath wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50489/
> ---
> 
> (Updated July 27, 2016, 10:41 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, DIPAYAN BHOWMICK, Gaurav 
> Nagar, Nitiraj Rathore, Pallav Kulshreshtha, Prabhjyot Singh, Rohit 
> Choudhary, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17913
> https://issues.apache.org/jira/browse/AMBARI-17913
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> - fix "Permission denied: '/etc/zeppelin/conf/interpreter.json'"
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/master.py
>  2de99b1 
> 
> Diff: https://reviews.apache.org/r/50489/diff/
> 
> 
> Testing
> ---
> 
> manually tested on ubuntu with non-root user
> 
> 
> Thanks,
> 
> Renjith Kamath
> 
>

Re: Review Request 50489: Zeppelin install fails for non-root users due to wrong permissions

2016-07-27 Thread Gaurav Nagar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50489/#review143719
---


Ship it!




Ship It!

- Gaurav Nagar


On July 27, 2016, 10:41 a.m., Renjith Kamath wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50489/
> ---
> 
> (Updated July 27, 2016, 10:41 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, DIPAYAN BHOWMICK, Gaurav 
> Nagar, Nitiraj Rathore, Pallav Kulshreshtha, Prabhjyot Singh, Rohit 
> Choudhary, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17913
> https://issues.apache.org/jira/browse/AMBARI-17913
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> - fix "Permission denied: '/etc/zeppelin/conf/interpreter.json'"
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/master.py
>  2de99b1 
> 
> Diff: https://reviews.apache.org/r/50489/diff/
> 
> 
> Testing
> ---
> 
> manually tested on ubuntu with non-root user
> 
> 
> Thanks,
> 
> Renjith Kamath
> 
>

Review Request 50500: Request for HBase metrics making a call to Namenode and Nimbus for that metric


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50500/
---

Review request for Ambari, Aravindan Vijayan, Myroslav Papirkovskyy, and Sid 
Wagle.


Bugs: AMBARI-17914
https://issues.apache.org/jira/browse/AMBARI-17914


Repository: ambari


Description
---

While investigating a slow API call on perf cluster found that we are 
requesting wrong metrics from AMS which wastes a network round trip and will 
not have any data.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/metrics/timeline/AMSPropertyProvider.java
 7ad1192 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/metrics/timeline/AMSPropertyProviderTest.java
 291512a 

Diff: https://reviews.apache.org/r/50500/diff/


Testing
---

Unit tests passed


Thanks,

Dmytro Sen

Re: Review Request 50489: Zeppelin install fails for non-root users due to wrong permissions

2016-07-27 Thread Prabhjyot Singh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50489/#review143715
---


Ship it!




Ship It!

- Prabhjyot Singh


On July 27, 2016, 10:41 a.m., Renjith Kamath wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50489/
> ---
> 
> (Updated July 27, 2016, 10:41 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, DIPAYAN BHOWMICK, Gaurav 
> Nagar, Nitiraj Rathore, Pallav Kulshreshtha, Prabhjyot Singh, Rohit 
> Choudhary, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17913
> https://issues.apache.org/jira/browse/AMBARI-17913
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> - fix "Permission denied: '/etc/zeppelin/conf/interpreter.json'"
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/master.py
>  2de99b1 
> 
> Diff: https://reviews.apache.org/r/50489/diff/
> 
> 
> Testing
> ---
> 
> manually tested on ubuntu with non-root user
> 
> 
> Thanks,
> 
> Renjith Kamath
> 
>

Review Request 50489: Zeppelin install fails for non-root users due to wrong permissions

2016-07-27 Thread Renjith Kamath


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50489/
---

Review request for Ambari, Alejandro Fernandez, DIPAYAN BHOWMICK, Gaurav Nagar, 
Nitiraj Rathore, Pallav Kulshreshtha, Prabhjyot Singh, Rohit Choudhary, and 
Sumit Mohanty.


Bugs: AMBARI-17913
https://issues.apache.org/jira/browse/AMBARI-17913


Repository: ambari


Description
---

- fix "Permission denied: '/etc/zeppelin/conf/interpreter.json'"


Diffs
-

  
ambari-server/src/main/resources/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/master.py
 2de99b1 

Diff: https://reviews.apache.org/r/50489/diff/


Testing
---

manually tested on ubuntu with non-root user


Thanks,

Renjith Kamath

Re: Review Request 50470: AMBARI-17907 On AMS collector move the webapp address does not change

2016-07-27 Thread Aleksandr Kovalenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50470/#review143711
---


Ship it!




Ship It!

- Aleksandr Kovalenko


On Июль 26, 2016, 10:49 п.п., Zhe (Joe) Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50470/
> ---
> 
> (Updated Июль 26, 2016, 10:49 п.п.)
> 
> 
> Review request for Ambari, Aleksandr Kovalenko, Jaimin Jetly, Richard Zang, 
> Vivek Ratnavel Subramanian, Xi Wang, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-17907
> https://issues.apache.org/jira/browse/AMBARI-17907
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Move collector to a different host. Collector start fails because following 
> property is not modified:
> 
> {quote}timeline.metrics.service.webapp.address{quote}
> 
> 
> Diffs
> -
> 
>   ambari-web/app/controllers/main/service/reassign/step4_controller.js 
> d8f0394 
>   ambari-web/test/controllers/main/service/reassign/step4_controller_test.js 
> d6ef9dc 
> 
> Diff: https://reviews.apache.org/r/50470/diff/
> 
> 
> Testing
> ---
> 
> Modified unit test.
> Local ambari-web test passed.
> 29371 tests complete (25 seconds)
> 154 tests pending
> Manual testing done.
> 
> 
> Thanks,
> 
> Zhe (Joe) Wang
> 
>

Re: Review Request 50445: AMBARI-17902 Config changes to support external solr and internal solr for Ranger


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50445/
---

(Updated July 27, 2016, 9:06 a.m.)


Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jaimin Jetly, 
Jayush Luniya, Robert Levas, Srimanth Gunturi, and Velmurugan Periasamy.


Bugs: AMBARI-17902
https://issues.apache.org/jira/browse/AMBARI-17902


Repository: ambari


Description
---

Ranger Service needs to support the following two scenarios in case when audit 
to solr is enabled and solrCloud is used as destination.

External Solr
If Audit to Solr is Enabled and Solr Cloud == true and Kerberos is Enabled
 If is_external_solr == true
  If is_external_solr_kerberized == true
   Then recommend ranger.is.solr.kerberised as true.


Ambari Internal Solr
If Audit to Solr is Enabled and Solr Cloud == true
  If is_external_solr == false and Kerberos is Enabled
   Then directly recommend ranger.is.solr.kerberised as true.


Diffs (updated)
-

  ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/kerberos.json 
3d6e25c 
  
ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py
 3ec4b53 
  
ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py
 1670d69 
  
ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/templates/ranger_solr_jaas_conf.j2
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/templates/ranger_solr_jass_conf.j2
 a456688 
  
ambari-server/src/main/resources/common-services/RANGER/0.6.0/configuration/ranger-admin-site.xml
 eacf541 
  
ambari-server/src/main/resources/common-services/RANGER/0.6.0/configuration/ranger-env.xml
 2cf3539 
  ambari-server/src/main/resources/common-services/RANGER/0.6.0/kerberos.json 
ffebb11 
  
ambari-server/src/main/resources/common-services/RANGER/0.6.0/themes/theme_version_3.json
 e65c9b2 
  
ambari-server/src/main/resources/common-services/STORM/1.0.1/configuration/ranger-storm-audit.xml
 1c869ed 
  ambari-server/src/main/resources/common-services/STORM/1.0.1/kerberos.json 
f9fa30d 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
c49e18e 
  
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
 133db26 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
ff5d4d9 
  ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml 
473d8a0 
  
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml
 d648638 
  ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml 
e67aebb 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/ranger-atlas-audit.xml
 ac22729 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/HBASE/configuration/ranger-hbase-audit.xml
 cc9f0d2 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/HBASE/kerberos.json 
ada02ad 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/configuration/ranger-hdfs-audit.xml
 0a04953 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/kerberos.json 
PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/ranger-hive-audit.xml
 671c08e 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/kerberos.json 
f9a0caf 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/KAFKA/configuration/ranger-kafka-audit.xml
 6aca7e7 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/KAFKA/kerberos.json 
PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/KNOX/configuration/ranger-knox-audit.xml
 bdd1994 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/KNOX/kerberos.json 
PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/RANGER_KMS/configuration/ranger-kms-audit.xml
 8c8278a 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/RANGER_KMS/kerberos.json
 bfd142a 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/ranger-yarn-audit.xml
 da24576 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/kerberos.json 
38896f5 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
c46a168 
  ambari-server/src/test/python/stacks/2.5/RANGER/test_ranger_admin.py 2345b8e 

Diff: https://reviews.apache.org/r/50445/diff/


Testing
---

1) External solr cloud on simple env (i.e. external solr cloud)
- audit to solr ON
- solr cloud ON
- external solr cloud ON
- external solr kerberos OFF

2) Internal solr cloud on simple env (i.e. log search)
- audit to solr ON
- solr cloud ON
- external solr cloud OFF
- external solr kerberos OFF

3) Internal solr cloud on secure env (i.e. log search + kerberos)
- audit to solr ON
- solr cloud ON
- external solr cloud OFF
- external solr

Re: Review Request 50445: AMBARI-17902 Config changes to support external solr and internal solr for Ranger



> On July 27, 2016, 4:15 a.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/kerberos.json,
> >  line 189
> > 
> >
> > How did this typo not cause problems in the past?

This typo is for branch-2.4 and trunk. The commit link: 
https://github.com/apache/ambari/commit/592609bb077e157f094da7c968ca629dc5095ae5


- Mugdha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50445/#review143669
---


On July 26, 2016, 2:49 p.m., Mugdha Varadkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50445/
> ---
> 
> (Updated July 26, 2016, 2:49 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jaimin Jetly, 
> Jayush Luniya, Robert Levas, Srimanth Gunturi, and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-17902
> https://issues.apache.org/jira/browse/AMBARI-17902
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Ranger Service needs to support the following two scenarios in case when 
> audit to solr is enabled and solrCloud is used as destination.
> 
> External Solr
> If Audit to Solr is Enabled and Solr Cloud == true and Kerberos is Enabled
>  If is_external_solr == true
>   If is_external_solr_kerberized == true
>Then recommend ranger.is.solr.kerberised as true.
> 
> 
> Ambari Internal Solr
> If Audit to Solr is Enabled and Solr Cloud == true
>   If is_external_solr == false and Kerberos is Enabled
>Then directly recommend ranger.is.solr.kerberised as true.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/kerberos.json 
> 3d6e25c 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py
>  3ec4b53 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py
>  1670d69 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/templates/ranger_solr_jaas_conf.j2
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/templates/ranger_solr_jass_conf.j2
>  a456688 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.6.0/configuration/ranger-admin-site.xml
>  eacf541 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.6.0/configuration/ranger-env.xml
>  2cf3539 
>   ambari-server/src/main/resources/common-services/RANGER/0.6.0/kerberos.json 
> ffebb11 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.6.0/themes/theme_version_3.json
>  e65c9b2 
>   
> ambari-server/src/main/resources/common-services/STORM/1.0.1/configuration/ranger-storm-audit.xml
>  1c869ed 
>   ambari-server/src/main/resources/common-services/STORM/1.0.1/kerberos.json 
> f9fa30d 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> bfdb3d3 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  a1b93e3 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> 96b1400 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml 
> b2cc1c4 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml
>  86e0964 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml 
> 2099958 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/ranger-atlas-audit.xml
>  ac22729 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HBASE/configuration/ranger-hbase-audit.xml
>  cc9f0d2 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HBASE/kerberos.json 
> ada02ad 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/configuration/ranger-hdfs-audit.xml
>  0a04953 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/kerberos.json 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/ranger-hive-audit.xml
>  671c08e 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/kerberos.json 
> f9a0caf 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/KAFKA/configuration/ranger-kafka-audit.xml
>  6aca7e7 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/KAFKA/kerberos.json 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/KNOX/configuration/ranger-knox-audit.xml
>  bdd1994 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/KNOX/kerberos.json 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/RANGER_KMS/configuration/ranger-kms-audit.xml
>  8c8278a 
>   
>

Re: Review Request 50445: AMBARI-17902 Config changes to support external solr and internal solr for Ranger



> On July 27, 2016, 4:12 a.m., Jayush Luniya wrote:
> > Is Kerberized External Solr + Unkerberized Cluster a valid scenario?

AFAIK, even if solr is kerberized all scripts will make calls using simple 
method.


- Mugdha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50445/#review143667
---


On July 26, 2016, 2:49 p.m., Mugdha Varadkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50445/
> ---
> 
> (Updated July 26, 2016, 2:49 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jaimin Jetly, 
> Jayush Luniya, Robert Levas, Srimanth Gunturi, and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-17902
> https://issues.apache.org/jira/browse/AMBARI-17902
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Ranger Service needs to support the following two scenarios in case when 
> audit to solr is enabled and solrCloud is used as destination.
> 
> External Solr
> If Audit to Solr is Enabled and Solr Cloud == true and Kerberos is Enabled
>  If is_external_solr == true
>   If is_external_solr_kerberized == true
>Then recommend ranger.is.solr.kerberised as true.
> 
> 
> Ambari Internal Solr
> If Audit to Solr is Enabled and Solr Cloud == true
>   If is_external_solr == false and Kerberos is Enabled
>Then directly recommend ranger.is.solr.kerberised as true.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/kerberos.json 
> 3d6e25c 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py
>  3ec4b53 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py
>  1670d69 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/templates/ranger_solr_jaas_conf.j2
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/templates/ranger_solr_jass_conf.j2
>  a456688 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.6.0/configuration/ranger-admin-site.xml
>  eacf541 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.6.0/configuration/ranger-env.xml
>  2cf3539 
>   ambari-server/src/main/resources/common-services/RANGER/0.6.0/kerberos.json 
> ffebb11 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.6.0/themes/theme_version_3.json
>  e65c9b2 
>   
> ambari-server/src/main/resources/common-services/STORM/1.0.1/configuration/ranger-storm-audit.xml
>  1c869ed 
>   ambari-server/src/main/resources/common-services/STORM/1.0.1/kerberos.json 
> f9fa30d 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> bfdb3d3 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  a1b93e3 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> 96b1400 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml 
> b2cc1c4 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml
>  86e0964 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml 
> 2099958 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/ranger-atlas-audit.xml
>  ac22729 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HBASE/configuration/ranger-hbase-audit.xml
>  cc9f0d2 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HBASE/kerberos.json 
> ada02ad 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/configuration/ranger-hdfs-audit.xml
>  0a04953 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/kerberos.json 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/ranger-hive-audit.xml
>  671c08e 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/kerberos.json 
> f9a0caf 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/KAFKA/configuration/ranger-kafka-audit.xml
>  6aca7e7 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/KAFKA/kerberos.json 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/KNOX/configuration/ranger-knox-audit.xml
>  bdd1994 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/KNOX/kerberos.json 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/RANGER_KMS/configuration/ranger-kms-audit.xml
>  8c8278a 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/RANGER_KMS/kerberos.json
>  bfd142a 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/ranger-yarn-audit.xml
>  da24576 
>

Re: Review Request 50445: AMBARI-17902 Config changes to support external solr and internal solr for Ranger



> On July 26, 2016, 6:07 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/common-services/RANGER/0.6.0/configuration/ranger-admin-site.xml,
> >  line 330
> > 
> >
> > I thought this property will be completely removed. Why do we need this 
> > property anymore?

This property is required as it is internally used in ranger code base.


- Mugdha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50445/#review143568
---


On July 26, 2016, 2:49 p.m., Mugdha Varadkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50445/
> ---
> 
> (Updated July 26, 2016, 2:49 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jaimin Jetly, 
> Jayush Luniya, Robert Levas, Srimanth Gunturi, and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-17902
> https://issues.apache.org/jira/browse/AMBARI-17902
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Ranger Service needs to support the following two scenarios in case when 
> audit to solr is enabled and solrCloud is used as destination.
> 
> External Solr
> If Audit to Solr is Enabled and Solr Cloud == true and Kerberos is Enabled
>  If is_external_solr == true
>   If is_external_solr_kerberized == true
>Then recommend ranger.is.solr.kerberised as true.
> 
> 
> Ambari Internal Solr
> If Audit to Solr is Enabled and Solr Cloud == true
>   If is_external_solr == false and Kerberos is Enabled
>Then directly recommend ranger.is.solr.kerberised as true.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/kerberos.json 
> 3d6e25c 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py
>  3ec4b53 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py
>  1670d69 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/templates/ranger_solr_jaas_conf.j2
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/templates/ranger_solr_jass_conf.j2
>  a456688 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.6.0/configuration/ranger-admin-site.xml
>  eacf541 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.6.0/configuration/ranger-env.xml
>  2cf3539 
>   ambari-server/src/main/resources/common-services/RANGER/0.6.0/kerberos.json 
> ffebb11 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.6.0/themes/theme_version_3.json
>  e65c9b2 
>   
> ambari-server/src/main/resources/common-services/STORM/1.0.1/configuration/ranger-storm-audit.xml
>  1c869ed 
>   ambari-server/src/main/resources/common-services/STORM/1.0.1/kerberos.json 
> f9fa30d 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
> bfdb3d3 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
>  a1b93e3 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> 96b1400 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml 
> b2cc1c4 
>   
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml
>  86e0964 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml 
> 2099958 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/ranger-atlas-audit.xml
>  ac22729 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HBASE/configuration/ranger-hbase-audit.xml
>  cc9f0d2 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HBASE/kerberos.json 
> ada02ad 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/configuration/ranger-hdfs-audit.xml
>  0a04953 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/kerberos.json 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/ranger-hive-audit.xml
>  671c08e 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/kerberos.json 
> f9a0caf 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/KAFKA/configuration/ranger-kafka-audit.xml
>  6aca7e7 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/KAFKA/kerberos.json 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/KNOX/configuration/ranger-knox-audit.xml
>  bdd1994 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/KNOX/kerberos.json 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/RANGER_KMS/configuration/ranger-kms-audit.xml
>  8c8278a 
>   
>

Re: Review Request 50445: AMBARI-17902 Config changes to support external solr and internal solr for Ranger